Application Proxy Blog

This blog provides additional information about the Application Proxy service of Windows Server and Azure Active Directory.

Understanding and fixing Proxy Trust CTL Issues with AD FS 2012 R2 and Web Application Proxy

Understanding and fixing Proxy Trust CTL Issues with AD FS 2012 R2 and Web Application Proxy

  • Comments 20
  • Likes

Firstly, thanks to Meir for his opening intro blogs.

As our first deep technical blog we thought we’d throw you in at the deep-end and the below article intentionally has quite a lot of technical depth to help give you some insight into some of the architectural workings of Web Application Proxy.

Over the coming months we’ll be bringing a good mix of content to you at differing technical levels but this one is definitely towards the deeper end of the technical spectrum. We’ll look forward to hearing if this is too much detail or if you’d like to see more of the same.

Introduction

As people are rolling out Web Application Proxy deployments we’ve seen some limited issues where the Proxy Trust relationship between Web Application Proxy and AD FS 2012 R2 is unable to establish correctly or subsequently fails often due to issues with the Certificate Trust List being sent by the AD FS server. We wanted to take the time to share our findings and explain some of the underlying architectural principles involved.

Issue Symptoms

The symptoms that we have been seeing are that the Proxy Trust relationship between a Web Application Proxy server and an ADFS 2012 R2 server either cannot be correctly established or starts to fail at some point in time. The symptoms of this can be found in the AD FS event logs on both the Web Application Proxy server and AD FS 2012 R2 server.

In the AD FS event log on the Web Application Proxy server:- 

Log Name:      AD FS/Admin
Source:        AD FS
Date:          3/24/2014 2:09:22 PM
Event ID:      422
Task Category: None
Level:         Error
Keywords:      AD FS
User:          S-1-5-21-2461873363-3247490994-1217893606-500
Computer:      wap.contoso.com
Description:
Unable to retrieve proxy configuration data from the Federation Service.
 
Additional Data
 
Trust Certificate Thumbprint:
48C0E6811E4631C1D115CD28B02BFA4214D4A179
 
Status Code:
Unauthorized

Exception details:
System.Net.WebException: The remote server returned an error: (401) Unauthorized.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.IdentityServer.Management.Proxy.StsConfigurationProvider.GetStsProxyConfiguration()

  

In the AD FS event logs on the AD FS server:-

Error 3/24/2014 2:49:52 PM AD FS 276 None
Log Name:      AD FS/Admin
Source:        AD FS
Date:          3/24/2014 2:49:52 PM
Event ID:      276
Task Category: None
Level:         Error
Keywords:      AD FS
User:          S-1-5-21-2461873363-3247490994-1217893606-1157
Computer:      adfs.contoso.com
Description:
The federation server proxy was not able to authenticate to the Federation Service.
 
User Action
Ensure that the proxy is trusted by the Federation Service. To do this, log on to the proxy computer

with the host name establish trust between the proxy and the Federation Service using the

Install-WebApplicationProxy cmdlet.
 
Additional Data
 
Certificate details:
 
Subject Name:
<null>
 
Thumbprint:
<null>
 
NotBefore Time:
<null>
 
NotAfter Time:
<null>

These events show that the Web Application Proxy was presenting a client authentication certificate when trying to retrieve the proxy configuration data but that the AD FS server is not seeing the certificate for some reason.

 


 

Background Information – how does the Proxy Trust work?

The proxy trust relationship between a Web Application Proxy server and the AD FS 2012 R2 server is client certificate based. When the Web Application Proxy post-install wizard is run a self-signed Client Certificate is generated and inserted into the AD FS configuration store using the credentials specified in the wizard. AD FS also propagates this to the AdfsTrustedDevices certificate store on the AD FS server.

The Proxy Trust certificate is then used by the Web Application Proxy server to authenticate to the AD FS server. This also means that the Proxy Trust is independent of domain membership and that the Web Application Proxy does not need to be domain joined. The Common Name on the certificate is “ADFS Proxy Trust – machinename”

When the client certificate authentication is negotiated, the AD FS server sends a Certificate Trust List (CTL) based on the contents of the AdfsTrustedDevices store. A CTL allows a client to filter valid client certificates from all the client certificates that may be present. If the CTL list of filtered certificates returns only a single certificate which is presented without the user needing to manually select a certificate.


 

If Device Registration has been enabled the AdfsTrustedDevices store also contains the MS-Organization-Access certificate which is the certificate used to issue Workplace Join client certificates.

This is important from an AD FS perspective primarily from a Workplace Join certificate as the CTL includes filtering for Workplace Join certificates so that a client doesn’t get unexpected and potentially confusing client certificate authentication prompts when accessing Workplace Join secured resources.

 

How Does this Relate to the Problem?

In a number of cases, we have seen that the CTL list sent by AD FS to the Web Application Proxy server was incorrect.

This was causing the Web Application Proxy server Proxy Trust cert to be filtered out as a valid certificate leading to no client certificate being presented to AD FS and the Proxy Trust failing with the event logs errors detailed above.

 

Where Does the AD FS Certificate Trust List come from?

So, we now know that the issue is due to an incorrect CTL being presented we now need to understand how the CTL is generated.

As many of you will know, AD FS 2012 R2 does not run on IIS as previous versions of AD FS did but instead runs directly as a service on top of http.sys. We need to understand a little bit about how AD FS uses http.sys in order to understand the root cause of the proxy trust issues.

From an http.sys perspective, AD FS uses hostname:port bindings as opposed to IP:port bindings. This is beneficial as we can host multiple disparate SSL certificates on the same IP/port instead of having to have a separate IP address for each certificate we want to bind to port 443. This is mainly leveraged in Web Application Proxy’s web application publishing but the same principles are used in AD FS 2012 R2.

When a client sends an SSL Client Hello it sends a Server Name header which details the FQDN that the client is connecting to. The server can then use this header to decide which certificate to serve. This is known as Server Name Indication.

Note - we will be following this blog up with a subsequent blog on SNI as there are some interesting discussions to be had about how to support clients who don’t send an SNI header.

Let’s take a look at what an AD FS certificate binding looks like. We can do this using the http.sys netsh sslcert command from the command prompt:-

C:\Users\administrator.CONTOSO>netsh http show sslcert

SSL Certificate bindings:
-------------------------
    Hostname:port                : adfs.contoso.com:443
    Certificate Hash             : 3638de9b03a488341dfe32fc3ae5c480ee687793
    Application ID               : {5d89a20c-beab-4389-9447-324788eb944a}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : AdfsTrustedDevices
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

 

In the above netsh output you will see a number of different certificate bindings, the above excerpt shows the main AD FS port 443 binding. As you can see this is a hostname:port binding using the AD FS service FQDN. The sharp-eyed among you will also have spotted that the Ctl Store Name is set to AdfsTrustedDevices. This is the certificate store that http.sys will use to generate the CTL list and is set by AD FS when it creates the SSL Certificate binding.

If the CTL Store name is not specified then Windows 2012 R2 will fall back to use a one of number of different certificate stores as detailed in the “Management of trusted issuers for client authentication” section of the following article:-

http://technet.microsoft.com/en-us/library/hh831771.aspx

In practice, on an AD FS server this will mean the Trusted Root Certification Authority store is used.

Also, as the above article also mentions, CTL generation from a CTL store can also be affected by having a CA Issued certificate in a store where you would typically only have self-signed certificates such as the Trusted Root CA store as noted in http://support.microsoft.com/kb/2802568. Root Certification Authority certificates are by definition self-signed.

 

http.sys SSL Certificate bindings and order of precedence

An equally important piece of information we need to understand in relation to the issues we are looking at here is the http.sys order of precedence for SSL Certificate bindings. HTTP.sys uses the following order of precedence for its SSL bindings:-

  • Specific IP:port binding

    If you have an IP:port binding for the IP address the inbound traffic is sent to then this binding will be used e.g. 1.2.3.4:443

  • Hostname:port

    If there is no specific IP:port binding in place then http.sys will try to match a certificate using the SNI header in the SSL Client Hello and the Hostname:port bindings e.g. adfs.contoso.com:443

  • Wildcard IP:port binding

    http.sys also supports a wildcard:port binding for 0.0.0.0:443 which will be used as a final fallback if we have not match on a specific IP or hostname binding.

As we will see later on, this becomes very important and conflicting or incorrect bindings can lead to CTL issues. The above is actually a slightly simplified version of the full http.sys certificate precedence. The full list for Windows Server 2012 R2 is as follows:-

Priority Name Parameters Description
1 IP IP, Port Exact IP and Port match
2 SNI Hostname, Port Exact hostname match (connection must specify SNI)
3 CCS Port Invoke Central Certificate Store
4 IPv6 Wildcard Port IPv6 wildcard match (connection must be IPv6)
5 IP Wildcard Port IP wildcard match (connection can be IPv4 or IPv6)

 

 

 

 

 

 

 

 


 

Known Causes of Proxy Trust failures

Keeping of all the above background information in mind let’s look at a number of different root causes we have seen for these Proxy Trust issues over the last few weeks along with resolutions for these.

Root Cause 1 – a specific IP:port certificate binding on the AD FS IP address

As we know, the Certificate Trust List is generated based on the CTL Store Name defined on the http.sys certificate binding. We can use “netsh http show sslcert” output to examine the certificate bindings from a problem AD FS server. In a number of customer cases we’ve found a binding similar to the following:- 

C:\Users\administrator.CONTOSO>netsh http show sslcert

SSL Certificate bindings:
    IP:port                      : 1.2.3.4:443
    Certificate Hash             : 3638de9b03a488341dfe32fc3ae5c480ee687793
    Application ID               : { 4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : (null)
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

There are 2 important things to note here:-

  1. This is an IP:port binding where 1.2.3.4 is the IP address of the ADFS server. This is also the IP address that the ADFS FQDN resolves to.
  2. The Ctl Store Name is null so the OS will fallback to use the Trusted Root CA store to build the Certificate Trust List

 

In this case as there is a specific IP:port binding in place this binding was being used ahead of the ADFS Hostname:port binding. Despite the certificate being used was exactly the same as the IP:port binding does not have a Ctl Store specified the Trusted Root CA store was being used to build the CTL list causing the Proxy Trust certificate on Web Application Proxy to be filtered out.

The only missing piece of the puzzle is why this binding was in place and, in the above example, the Application ID helped us understand this.

                Application ID               : { 4dc3e181-e14b-4a21-b022-59fc669b0914}

This happens to be the SSL Certificate Owner Application ID for w3svc (IIS) so it looked as if the customer had configured this binding in IIS and this proved to be the case. We have seen a number of customers install IIS on the AD FS server either due to believing it is required for AD FS 2012 R2 (it isn't) or as a way to generate a Certificate request.

Note – when you configure http.sys certificate bindings using netsh you can configure whatever Application ID you want so a binding may have been created manually and appear to have been configured by an application.

The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can be run on an AD FS 2012 R2 server to determine if this issue is present.

 

Root Cause 1 - Resolution Options

When looking at a resolution the first thing to determine is why the binding is in place.

This might have been created by mistake or as a leftover from an install of a different product or Windows component but it may equally be there for a valid reason. As mentioned above it’s also possible that a binding may have been manually created possibly to support non-SNI clients.

Once you have determined the reason for the binding being in place, you can decide which of the following resolutions is more appropriate:-

1. Remove the specific IP:port binding

Be sure to check that the binding does not come back as if there is an application configured with such a binding it may recreate this automatically or on next service start-up.

2. Use an additional IP address for AD FS traffic

In our example above if the 1.2.3.4 SSL Binding is expected and required, then using a 2nd IP such as 1.2.3.5 for ADFS and resolving the ADFS service FQDN to this IP address would mean that the Hostname:port bindings would then be used (assuming that there isn’t an IP:port binding for 1.2.3.5:443 of course).

3. Configure the AdfsTrustedDevices store as the Ctl Store for the specific IP:port binding

This will again have some dependence on why the specific IP:port binding is there and if this relies on the default CTL Store for client certificate authentication but an option would be to set the Ctl Store on the IP:port binding to be the AdfsTrustedDevices store

At this point in time, our recommendation is to avoid having IP specific bindings and use option 1 as the preferred resolution. 

 


 

Root Cause 2 – the AD FS Hostname:port binding does not have a Ctl Store defined

This was a bit more of a strange one as when we checked the http.sys cert bindings everything looked correct except that the AD FS Hostname:port binding did not have a Ctl Store defined:-

C:\Users\administrator.CONTOSO>netsh http show sslcert

SSL Certificate bindings:
-------------------------
    Hostname:port                : adfs.contoso.com:443
    Certificate Hash             : 3638de9b03a488341dfe32fc3ae5c480ee687793
    Application ID               : {5d89a20c-beab-4389-9447-324788eb944a}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

Of interest here is that the Application ID is the AD FS 2012 R2 SSL Certificate Owner Application ID so on the surface does look to have been created by AD FS. We did not manage to identify why the CTL Store was not correctly set in this case. The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can be run on an AD FS 2012 R2 server to determine if this issue is present.

Root Cause 2 - Resolution

In this case we can use the following PowerShell cmdlet to regenerate the ADFS Certificate bindings on the AD FS server:-

Set-AdfsSslCertificate -Thumbprint 3638de9b03a488341dfe32fc3ae5c480ee687793

The Thumbprint here is the Thumbprint for the ADFS Server SSL certificate and needs to be set accordingly. This cmdlet should be run on each AD FS server in a farm. For further details on this cmdlet please see the following article - http://technet.microsoft.com/en-us/library/dn479374.aspx 

 


 

Root Cause 3 – CA issued server certificate placed in the AdfsTrustedDevices store

The AdfsTrustedDevices store is an AD FS specific store and should not need to have any certificates manually placed in here.

We found in one case that the ADFS service SSL certificate was incorrectly placed in the AdfsTrustedDevices store as well as the correct Local Machine, Personal store. The AdfsTrustedDevices store should only contain the MS-Organization-Access certificate which is the self-signed cert used for issuing Workplace Join certificates, and the Proxy Trust certificates for each of the Web Application Proxy servers.

As detailed earlier, having a CA Issued certificate in a store where only Self-Signed certs would normally exist affects the CTL generated from this store and the CTL will then only contain the CA Issued certificate. The presence of a CA issued certificate in the AdfsTrustedDevices store changed the CTL being returned and led to the proxy trust failure.

The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can be run on an AD FS 2012 R2 server to determine if this issue is present.

Root Cause 3 – Resolution

Deleting the non-self signed SSL Server Certificate from the AdfsTrustedDevices store resolved the issue and meant that the CTL being generated from the AdfsTrustedDevices store was now correct.

 


 

Root Cause 4 – Proxy Trust certificate propagation issues across an AD FS 2012 R2 farm

When a Proxy Trust is established with an AD FS server, the Proxy Trust certificate is written to the AD FS configuration database and added to the AdfsTrustedDevices store on the AD FS server that handled the Proxy Trust set-up. In an AD FS farm a regular task runs on each AD FS server in the farm which polls the AD FS configuration store for the current list of Proxy Trust certificates and updates the local AdfsTrustedDevices store.

We have recently discovered an issue where this task only successfully runs if Device Registration has been initialized.

If Device Registration has not been initialized then Proxy Trust certificate propagation across the AD FS farm may fail and a Proxy Trust will only work against the AD FS server the trust was established with. As the CTL is based on the AdfsTrustedDevices store, if this has not been populated with the correct Proxy Trust certificates and a Web Application Proxy server talks to an AD FS server other than the one it established the proxy trust with, then the proxy trust may fail due to the CTL being incorrect.

You can check if Device Registration has been initialized by running the following PowerShell cmdlet on one of the AD FS servers:-

                Get-AdfsDeviceRegistration

This requires Enterprise Administrator credentials to run. If Device Registration has been initialized you would see output similar to the following:-

:-PS C:\users\administrator.CONTOSO\Desktop> Get-AdfsDeviceRegistration


DrsObjectDN          : CN=DeviceRegistrationService,CN=Device Registration Services,CN=Device Registration
                       Configuration,CN=Services,CN=Configuration,DC=contoso,DC=com
DevicesPerUser       : 10
MaximumInactiveDays  : 90
IsEnabledOnPremises  : True
IsEnabledInCloud     : False
DeviceObjectLocation : CN=RegisteredDevices,DC=contoso,DC=com

The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can also be run on an AD FS 2012 R2 server to determine if this issue is present.

Note – if your load balancing for your AD FS server farm uses client IP based affinity we would generally expect a Web Application Proxy server to use the same AD FS server other than in a failover scenario.

 

Root Cause 4 - Resolution Options

 

Update 1st July 2014 - A fix is now available for this specific issue in the June 2014 Windows update rollup - http://support.microsoft.com/kb/2964735

 

1. Initialize Device Registration in Active Directory

The current recommended resolution for this is to initialize Device Registration using the following PowerShell script on one of the AD FS 2012 R2 servers which creates the DRS containers in Active Directory.

Initialize-ADDeviceRegistration

Note – this needs to be run with Enterprise Administrator credentials and requires that the Windows 2012 R2 AD schema update has been deployed. For further details on this cmdlet please see the following article - http://technet.microsoft.com/en-us/library/dn479332.aspx

This cmdlet does *not* enable the device authentication nor the device registration service in the ADFS servers. Enabling Device Registration is a second step in the DRS set-up process.

The AD FS Product Group are currently investigating a fix for this issue but at the time of writing we do not have further details on this or any expected timelines. We will provide an update when we have further information on this.

2. Manually Sync Proxy Trust Certificates using PowerShell

The PowerShell script provided in the ‘Issue Detection PowerShell Script’ section below can be run with the - syncproxytrustcerts switch to sync down missing  Proxy Trust certificates from the AD FS configuration to the AdfsTrustedDevices certificate store. This script should be run on each of the AD FS servers in the farm.

Note – this is not a permanent solution as the Proxy Trust certificates will be renewed on a regular basis so until a fix is available Option 1 is the recommended resolution.

3. Use source IP based affinity on your AD FS 2012 R2 load balancing solution

Using source IP based affinity on the AD FS Load Balancer will mean a Web Application Proxy server is generally going to use the same AD FS server it established the initial trust with and therefore not be reliant on the proxy trust certificate being propagated to other members of the AD FS farm. If the preferred AD FS node a Web Application Proxy server changes or in a failover situation then this issue may still be exposed. 

 


 

Root Cause 5 – System time mis-match between WAP and ADFS

It is expected that the Web Application Proxy server and AD FS server are time synchronized. Where Web Application Proxy is not domain joined this may not be the case and we have seen Proxy Trust issues as a result of this e.g.

AD FS Server time = 12pm GMT timeone

Web Application Proxy Server time = 12pm PST timezone (8pm GMT)

In this scenario while the system time may seem synchronized the timezone difference means that the Web Application Proxy server is 8 hours ahead of the AD FS server time. When the Proxy Trust relationship is established, the self-signed Proxy Trust certificate generated by the Web Application Proxy is not yet valid on the AD FS server and so the Proxy Trust fails.

While this is not directly related to CTL issues it’s one to be aware of with similar symptoms.

Root Cause 5 - Resolution

Ensure that the system time of the Web Application Proxy server and AD FS 2012 R2 server are in sync including that the timezones are set appropriately. 

 


 

Root Cause 6  – SSL termination taking place on a network device between Web Application Proxy and AD FS server

We saw a recent issue where we were seeing the following in the AD FS event logs on the Web Application Proxy server and AD FS server.

In the AD FS event log on the Web Application Proxy server:- 

Log Name:      AD FS/Admin
Source:        AD FS
Date:          3/24/2014 2:09:22 PM
Event ID:      422
Task Category: None
Level:         Error
Keywords:      AD FS
User:          S-1-5-21-2461873363-3247490994-1217893606-500
Computer:      wap.contoso.com
Description:
Unable to retrieve proxy configuration data from the Federation Service.
 
Additional Data
 
Trust Certificate Thumbprint:
48C0E6811E4631C1D115CD28B02BFA4214D4A179
 
Status Code:
Unauthorized

Exception details:
System.Net.WebException: The remote server returned an error: (401) Unauthorized.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.IdentityServer.Management.Proxy.StsConfigurationProvider.GetStsProxyConfiguration()

In the AD FS event logs on the AD FS server:-

Error 3/24/2014 2:49:52 PM AD FS 276 None
Log Name:      AD FS/Admin
Source:        AD FS
Date:          3/24/2014 2:49:52 PM
Event ID:      276
Task Category: None
Level:         Error
Keywords:      AD FS
User:          S-1-5-21-2461873363-3247490994-1217893606-1157
Computer:      adfs.contoso.com
Description:
The federation server proxy was not able to authenticate to the Federation Service.
 
User Action
Ensure that the proxy is trusted by the Federation Service. To do this, log on to the proxy computer

with the host name establish trust between the proxy and the Federation Service using the

Install-WebApplicationProxy cmdlet.
 
Additional Data
 
Certificate details: 
 
 Subject Name:
<CN=adfsapps.contoso.com, OU=Contoso, O=Contoso, L=Contoso, S=Contoso, C=US>
 
Thumbprint:
<AD467089E3407BD72CD5F3D46A26B3873C5F0F4C>
 
NotBefore Time:
<2014-05-09 00:00:00>
 
NotAfter Time:
<2015-05-09 23:59:59>
 

 

There are a number of things to note here:-

  1. The AD FS server was presented a client certificate. This is different to most of the above issues where a client certificate is not seen at the AD FS server.
  2. The thumbprint seen at the AD FS server is different to the one seen in the Web Application Proxy AD FS event logs.
  3. The Subject Name seen by the AD FS proxy is not what we would expect from a proxy trust certificate which should have the form 'ADFS Proxy Trust - webappproxymachinename'

Given the difference in Thumbprint and unexpected Subject Name we investigated the network devices between the Web Application Proxy server and the AD FS server and determined that the hardware load balancer sitting in-front of the AD FS farm was carrying SSL bridging i.e. terminating the SSL connection from Web Application Proxy and establishing a new SSL connection to the AD FS server. The certificate seen by the AD FS server was being presented by the hardware load balancer.

As the Proxy Trust relationship between Web Application Proxy and AD FS server is client certificate based this was breaking the proxy trust relationship.

The SSL communications between the Web Application Server and AD FS server cannot be pre-terminated on a load balancer, Firewall, other reverse proxy device and doing so will break the proxy trust relationship. 

Root Cause 6 - Resolution

Disabling SSL Termination on the hardware load balancer to allow the SSL negotiation to happen between Web Application Proxy and AD FS server resolved this issue.

 


 

Issue Detection PowerShell Script

The following PowerShell script can be run on an AD FS server to determine if any of the first 4 issues detailed above are present. In an AD FS farm the script should be run on each farm member.

The script can also be run with the –syncproxytrustcerts switch to sync down missing Proxy Trust certificates from the AD FS configuration to the AdfsTrustedDevices certificate store as a temporary resolution for Issue 4. This sync would need to be done on each AD FS server.

 

param
(
  [switch]$syncproxytrustcerts
)


function checkhttpsyscertbindings()
{

Write-Host; Write-Host("1 - Checking http.sys certificate bindings for potential issues")

$httpsslcertoutput = netsh http show sslcert
$adfsservicefqdn = (Get-AdfsProperties).HostName
$i = 1
$certbindingissuedetected = $false

While($i -lt $httpsslcertoutput.count)
{
        $ipport = $false
        $hostnameport = $false

        if ( ( $httpsslcertoutput[$i] -match "IP:port" ) ) { $ipport = $true }
        elseif ( ( $httpsslcertoutput[$i] -match "Hostname:port" ) ) { $hostnameport = $true }

        # Check for IP specific certificate bindings
        if ( ( $ipport -eq $true ) )
        {
            $httpsslcertoutput[$i]
            $ipbindingparsed = $httpsslcertoutput[$i].split(":")

            if ( ( $ipbindingparsed[2].trim() -ne "0.0.0.0" ) -and ( $ipbindingparsed[3].trim() -eq "443") )
            {
                $warning = "There is an IP specific binding on IP " + $ipbindingparsed[2].trim() + " which may conflict with the AD FS port 443 cert binding." | Write-Warning
                $certbindingissuedetected = $true
            }
           
            $i = $i + 14
            continue
        }

        # check that CTL Store is set for ADFS service binding
        elseif ( $hostnameport -eq $true )
        {
            $httpsslcertoutput[$i]
            $ipbindingparsed = $httpsslcertoutput[$i].split(":")
            If ( ( $ipbindingparsed[2].trim() -eq $adfsservicefqdn ) -and ( $ipbindingparsed[3].trim() -eq "443") -and ( $httpsslcertoutput[$i+10].split(":")[1].trim() -ne "AdfsTrustedDevices" ) )

            {
                Write-Warning "ADFS Service binding does not have CTL Store Name set to AdfsTrustedDevices"
                $certbindingissuedetected = $true
            }
        $i = $i + 14
        continue
        }
    $i++
}

If ( $certbindingissuedetected -eq $false ) { Write-Host "Check Passed: No certificate binding issues detected" }

}

function checkadfstrusteddevicesstore()
{
# check for CA issued (non-self signed) certs in the AdfsTrustedDevices cert store
Write-Host; Write-Host "2 - Checking AdfsTrustedDevices cert store for non-self signed certificates"

$certlist = Get-Childitem cert:\LocalMachine\AdfsTrustedDevices -recurse | Where-Object {$_.Issuer -ne $_.Subject}

If ( $certlist.count -gt 0 )
{
    Write-Warning "The following non-self signed certificates are present in the AdfsTrustedDevices store and should be removed"
    $certlist | Format-List Subject
}
Else { Write-Host "Check Passed: No non-self signed certs present in AdfsTrustedDevices cert store" }

}


function checkproxytrustcerts
{
    Param ([bool]$repair=$false)

    Write-Host; Write-Host("3 - Checking AdfsTrustedDevices cert store is in sync with ADFS Proxy Trust config")

    $doc = new-object Xml
    $doc.Load("$env:windir\ADFS\Microsoft.IdentityServer.Servicehost.exe.config")

    $connString = $doc.configuration.'microsoft.identityServer.service'.policystore.connectionString

    $command = "Select ServiceSettingsData from [IdentityServerPolicy].[ServiceSettings]"

    $cli = new-object System.Data.SqlClient.SqlConnection
    $cli.ConnectionString = $connString

    $cmd = new-object System.Data.SqlClient.SqlCommand
    $cmd.CommandText = $command
    $cmd.Connection = $cli

    $cli.Open()

    $configString = $cmd.ExecuteScalar()

    $configXml = new-object XML
    $configXml.LoadXml($configString)

    $rawCerts = $configXml.ServiceSettingsData.SecurityTokenService.ProxyTrustConfiguration._subjectNameIndex.KeyValueOfstringArrayOfX509Certificate29zVOn6VQ.Value.X509Certificate2


    #$ctl = dir cert:\LocalMachine\ADFSTrustedDevices

    $store = new-object System.Security.Cryptography.X509Certificates.X509Store("ADFSTrustedDevices","LocalMachine")
    $store.open("MaxAllowed")

    $atLeastOneMismatch = $false
    $badCerts = @()

    foreach($rawCert in $rawCerts)
    {   

        $rawCertBytes = [System.Convert]::FromBase64String($rawCert.RawData.'#text')
        $cert=New-Object System.Security.Cryptography.X509Certificates.X509Certificate2(,$rawCertBytes)

        $now = Get-Date

        if ( ($cert.NotBefore -lt $now) -and ($cert.NotAfter -gt $now))
        {
            $certThumbprint = $cert.Thumbprint
         $certSubject = $cert.Subject
         $ctlMatch = dir cert:\localmachine\ADFSTrustedDevices\$certThumbprint -ErrorAction SilentlyContinue
         if ($ctlMatch -eq $null)
         {
       $atLeastOneMismatch = $true
          Write-Warning "This cert is NOT in the CTL: $certThumbprint - $certSubject"
   
       if ($repair -eq $true)
       {
        write-Warning "Attempting to repair"
        $store.Add($cert)
        Write-Warning "Repair successful"
       }
                else
                {
                    Write-Warning ("Please install KB.2964735 or re-run script with -syncproxytrustcerts switch to add missing Proxy Trust certs to AdfsTrustedDevices cert store")
                }
   
         }
        }

    }
    $store.Close()
 
    if ($atLeastOneMismatch -eq $false)
    {
     Write-Host("Check Passed: No mismatched certs found. CTL is in sync with DB content")
    }


}

checkhttpsyscertbindings
checkadfstrusteddevicesstore
checkproxytrustcerts($syncproxytrustcerts)
Write-Host; Write-Host("All checks completed.")

 

Summary

First of all, congratulations if you made it this far :-)

One the key things we wanted you to explain is that the Proxy Trust relationship is based on Client Certificates and can be affected both by http.sys bindings and the associated Ctl Store on the AD FS server. Having a specific IP:port binding takes precedence over the ADFS Hostname:port binding and may lead to the wrong Certificate Trust List being sent. The wrong CTL will prevent Web Application Proxy from presenting the correct Proxy Trust certificate.

The first place to look when troubleshooting a Proxy Trust issue is the http.sys certificate bindings on the ADFS server by running the following command from the command prompt:-

                netsh http show sslcert

Resolution steps will depend on what bindings are found and on why these are there. The simplest solution is to ensure that there is not a specific IP:port binding in place for the IP address that the AD FS Service FQDN resolves to.

Another item to check is the contents of the AdfsTrustedDevices certificate store for the presence of both the expected Proxy Trust certificates and for any unexpected CA issued certificates.

We also covered a number of other causes of Proxy Trust failures that we've encountered with our customers and provided a PowerShell script that can be run on an AD FS 2012 R2 server to detect a number of the issues detailed in this article.

We hope this helps some of you solve Proxy Trust issues you might come up against and we’d love to hear any other conditions you might come across that lead to similar issues.

 

Update 1st July 2014 - article updated with hotfix availability for Issue 4.

 

Ian Parramore, Senior Escalation Engineer, Web Application Proxy Support team             

 

Thanks to the following for their input on this article:-

Billy Price, Senior Escalation Engineer, Web Application Proxy Support team

Meir Mendelovich, Product Manager, Web Application Proxy Product Group

Ramiro Calderon, Software Development Engineer, AD FS Product Group

Mike Bishop, Product Manager, http.sys Product Group

Andrew Rockall, Software Development Engineer, http.sys Product Group 

Blog - Email Author
Contact - Application Proxy Blog
  • Send
Comments
  • Very nice . Great job.

  • Thanks

  • Great blog post. Thank you!

  • Excellent post, fixed one of our issues with a binding.

    We are definitely looking forward to your post on SNI as we have both Windows XP and Lync Phone Edition (CE 6.0) clients that currently cannot connect to ADFS 3.0

  • Awwesome

  • Awwesome

  • Got similar issue


    Log Name: AD FS/Admin
    Source: AD FS
    Date: 7/12/2014 8:34:20 AM
    Event ID: 422
    Task Category: None
    Level: Error
    Keywords: AD FS
    User: NETWORK SERVICE
    Computer: DCVADFSPROX1.tttdmz.rttscorp.com
    Description:
    Unable to retrieve proxy configuration data from the Federation Service.

    Additional Data

    Trust Certificate Thumbprint:

    Status Code:


    Exception details:
    System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 199.107.71.58:443
    at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
    at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
    --- End of inner exception stack trace ---
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.IdentityServer.Management.Proxy.StsConfigurationProvider.GetStsProxyConfiguration()
    Event Xml:
    http://schemas.microsoft.com/win/2004/08/events/event">


    422
    0
    2
    0
    0
    0x8000000000000001

    70595


    AD FS/Admin
    DCVADFSPROX1.bkrdmz.mbakercorp.com



    http://schemas.microsoft.com/ActiveDirectoryFederationServices/2.0/Events">

    Data>


    System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 199.107.71.58:443
    at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
    at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
    --- End of inner exception stack trace ---
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.IdentityServer.Management.Proxy.StsConfigurationProvider.GetStsProxyConfiguration()





    Issue: ADFS proxy WAP server pointing to itself when resolving STS name.
    Resolution : Pointed ADFS proxy to ADFS server when resolving STS name.

  • Understanding and fixing Proxy Trust CTL Issues with AD FS 2012 R2 and Web Application Proxy

  • Hello Ian

    When I configure any WAP server through the Server Manager Dashboard the system creates an ADFS Proxy Trust certificate which is only valid for 5 days - this certificate does not renew itself. How can I change the certificate expiration date to lets say 3 years?

    However on my primary ADFS server (secondary servers do not contain a copy these certs) under the console root (AdfsTrustedDevices tab) I can see the same certificate plus all the newly renewed ones which expiration date is only valid for 2 weeks or so.

    whats going on man?

    Please advise

    Many Thanks

    rio.sanches@cmtww.com




  • Hi Rio,

    The certs are auto-renewed on a regular basis and stored in a different cert store so you won't see these in the normal Local Machine store - this is perfectly normal and expected. These are essentially internal certs and you don't need to worry about their expiration time.
    It does sound as if your ADFS server has an issue clearing up the ADFS Trusted Devices store and I suspect you need to apply the following update - http://support.microsoft.com/kb/2964735/en-us.This resolves an issue with the Proxy Trust cert update task on the ADFS server as we should clear up older Proxy Trust certs.

    Hope this helps,

    Ian

  • Thank you Ian, I will apply the patches and keep you posted just for the record.

  • Hi Ian,
    Here's to hoping you or someone else might have insight to our problem:

    We are running 2 ADFS servers (loadbalanced) and we would like to add 2 ADFS proxies, loadbalanced.
    Everything is on 2012 R2.

    The ADFS servers are both up, and one proxy is up, the federation sign in page is reachable from internal and external network. So all good so far.

    However when configuring the 2nd proxy, the WAP wizard gives us this issue
    "Time out has expired and the operation has not been completed."

    The associated event in eventlog is the following:

    Web Application Proxy encountered an error while retrieving the configuration from configuration storage.
    Details: (0x80072f8f).
    Web Application Proxy will continue to use the existing configuration.

    Any idea what has gone wrong?

    Cheers,
    Syd

  • Hi Syd,

    It's hard to give depth support in the blog comments so you might want to open up a case with us.
    However, 0x80072f8f = Windows error code 2f8f (hex) = 12175 (dec). This is almost certainly a WinHttp error.

    If we look at http://msdn.microsoft.com/en-us/library/windows/desktop/aa383770(v=vs.85).aspx we can see this:-
    12175 - One or more errors were found in the Secure Sockets Layer (SSL) certificate sent by the server.

    First things to check would be that the ADFS certificate being used has the correct name and includes the ADFS Service FQDN. Next thing is to check that the Issuing Trusted Root CA and any Intermediate Certs are installed in the Local Machine cert store on the WAP server.

    If that doesn't help you should enable the CAPI2 debug log in event viewer while repro'ing the issue to see why CAPI doesn't like the ADFS SSL Cert.

    Hope this helps,

    Ian

  • Hi Ian,

    thank you very much for taking the time to reply. We actually got it going eventually. Not sure about "why" the following worked for us, but all we did was have a browser in the background pointing to our federation server (with a dns record in etc/hosts), then we ran the configuration wizard again, and it worked.. which was enough for us.

    Appreciate the explanation though, I'm fairly new to troubleshooting in Windows, it's great to know these error codes are publicly available.

    Cheers,
    Syd

  • Hi Syd,

    Good to hear you got up and running.

    I have seen something similar where a user initiated process downloaded the intermediate CA cert from the AIA location which wasn't working for services. This meant the intermediate cert was then available locally for services to access.

    We didn't get to the bottom of why the services AIA access was failing as manual installation of the intermediate cert resolved things for this specific customer and they didn't want to look further.

    It sounds like you may have hit something similar but hard to tell. I'd have expected CAPI2 logging wold javae shown something.

    Thanks,

    Ian

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment