I recently dealt with a support case exhibiting the below issue.
My customer was publishing an Exchange CAS server with Forefront TMG, and Kerberos Constrained Delegation (KCD) was used in the publishing rule as the authentication delegation method to the web server.
The web publishing was working just fine (including KCD) but surprisingly the “test rule” button was returning the error “Failed to get domain controller name for this published server” when validating the rule.
Here is a screenshot of the test rule button results:
Based on this error details, I led some investigations using both Repro mode data collected with the useful TMG Data Packager (that comes with the TMG BPA) and source code analysis in order to understand the root cause of this.
I found out that one of the checks done by the “test rule” code for the KCD scenario is to check if the published web server is member of a domain and which domain it belongs to.
To do this, the code makes a call the public Windows API DsRoleGetPrimaryDomainInformation which is described in MSDN at http://msdn.microsoft.com/en-us/library/ms676042.aspx).
I found that the call to DsRoleGetPrimaryDomainInformation was failing with an error code of 0x800706ba (which translates to RPC_S_SERVER_UNAVAILABLE).
Reviewing the Network Monitor capture taken on the internal network interface of TMG I spotted that the RPC calls, initiated by DsRoleGetPrimaryDomainInformation, to the published server were failing.
The capture was showing unsuccessful TCP connection attempts from the TMG machine to the published on destination ports 445 and 139:
After further discussion with the customer, I discover that a network filtering device (firewall) located in between TMG and the published server was dropping this connection attempts, explaining the error.
After opening these ports on this filtering device, the “test rule” button returned no error!
Escalation Engineer - Microsoft CSS Forefront Security Edge Team