ADFS Product Support Blog : Troubleshooting

Interesting problem when adding an ADFS Proxy

jimsim — Wed, 04 Jun 2008 18:10:00 GMT

I am working on a blog post (step-by-step) for the Proxy component and I ran into a problem yesterday that ran me around pretty good. We have seen this issue or variations of it on some support cases recently, so I thought the actual problem itself would make a good post.

The problem is caused by permissions to the private key on the Client Authentication Certificate needed. In my initial attempt to setup and document the Proxy component, I made a request to my Standalone CA for a client authentication certificate. After approving the request, the only option from the certificate web page was to "install this certificate". Next, when I viewed the certificate snap-in on the proxy server, I noticed that the certificate was installed to the user store and not the computer store. I simply did a copy paste operation from user to computer. This appeared to work for me because when I double clicked the certificate, it looked fine. I saw the "You have a private key" on the general tab and I assumed all was well.

When I went to test - I received a failure. The first thing I did was run the ADFS Diagnostic tool. I ran it on the FS-A, then copied the file to the FS-A Proxy. I passed all tests and the tool was not finding the failure!

Here are the Event Log and Debug Logs from my FS-A and FS-A Proxy when I attempted to access the application with the Proxy in place:

From the FS-A

Event Viewer:

Event Type: Error

Event Source: ADFS Federation Service

Event Category: None

Event ID: 664

Date: 6/3/2008

Time: 5:13:09 PM

User: N/A

Computer: ADFSACCOUNT

Description:

The Federation Service failed a privileged Web method call because Secure Sockets Layer (SSL) client authentication information was not available.

This event can occur if the client does not provide a client certificate or if Internet Information Services (IIS) rejects the client's certificate because it does not chain to a trusted root certification authority in the Federation Service.

User Action

If this is a valid call from the Federation Service Proxy, ensure that the root of the Federation Service Proxy client certificate is trusted by the Federation Service.

Debug logs:

2008-06-03T22:13:09 [INFO] Processing HTTP POST: https://adfsaccount.adatum.com/adfs/fs/FederationServerService.asmx

2008-06-03T22:13:09 [VERBOSE] Received message that is not SignIn Request or Response.

2008-06-03T22:13:09 [ERROR] MethodInvocationCheck: Client cert is not present

2008-06-03T22:13:09 [EVENTLOG] Error ProxyWebMethodAccessDeniedNoCert ()

2008-06-03T22:13:09 [ERROR] MethodInvocationCheck: Denying access

From the FS-A Proxy

Event Viewer:

Event Type: Error

Event Source: ADFS

Event Category: None

Event ID: 605

Date: 6/3/2008

Time: 5:13:09 PM

User: N/A

Computer: FSA-PROXY

Description:

The Federation Service Proxy encountered an exception when it called a Federation Service Web method.

Federation Server URL: https://adfsaccount.adatum.com/adfs/fs/FederationServerService.asmx

Web method: GetProxyTrustConfiguration

Proxy certificate thumbprint: ECF1FE79E51231DF48098E1044233FCBDABF04CC

This may cause a user request to fail.

User Action

The exception details may give an indication of the precise problem.

Check network connectivity between the Federation Service Proxy and the Federation Service.

Ensure that the Federation Service is running.

Ensure that the Federation Service Proxy client authentication certificate has been added to the list of proxy authentication certificates in the Federation Service trust policy.

Ensure that the Federation Service Proxy client authentication certificate chains to a root that is trusted by the Federation Service.

Ensure that the Federation Service Internet Information Services (IIS) Secure Sockets Layer (SSL) server certificate chains to a root that is trusted by the Federation Service Proxy.

Ensure that the Federation Service Uniform Resource Locator (URL) that is configured in the Federation Service Proxy web.config uses the name that is the subject of the Federation Service IIS SSL server certificate.

Additional Data

Exception details:

System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> Attempted to perform an unauthorized operation.

at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

at System.Web.Security.SingleSignOn.FederationServerSoapProxy.GetProxyTrustConfiguration(VersionInformation proxyVersion, VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& trustConfig)

at System.Web.Security.SingleSignOn.LSPersistentState.GetPolicy(VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& data)

Debug logs:

2008-06-03T22:13:09 [VERBOSE] Processing HTTP GET: https://adfsaccount.adatum.com/adfs/ls/?wa=wsignin1.0&wtrealm=urn:federation:treyresearch&wct=2008-06-03T22:13:09Z&wctx=https://adfsweb.treyresearch.net:8081/claimapp/\https://adfsweb.treyresearch.net:8081/claimapp/default.aspx

2008-06-03T22:13:09 [VERBOSE] Received SignIn Request.

2008-06-03T22:13:09 [ERROR] Exception from GetProxyTrustConfiguration: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> Attempted to perform an unauthorized operation.

at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

at System.Web.Security.SingleSignOn.LSPersistentState.GetPolicy(VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& data)

2008-06-03T22:13:09 [EVENTLOG] Error ExceptionFromFedServer (https://adfsaccount.adatum.com/adfs/fs/FederationServerService.asmx, GetProxyTrustConfiguration, ECF1FE79E51231DF48098E1044233FCBDABF04CC, System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> Attempted to perform an unauthorized operation.

at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

at System.Web.Security.SingleSignOn.LSPersistentState.GetPolicy(VersionInformation& fsVersion, ProxyInformation& proxyInformation, TrustConfigurationData[]& data))

As you can see, there is a problem with the client auth certificate somewhere. I did a fair amount of double checking my steps - but everything looked correct and seemed to be checking out. The doubt was starting to creep in - I started to wonder how much I knew about this stuff! Then I remembered an issue that came up a few weeks ago.

The diagnostic tool does check for the existence and proper permissions of the private key and will flag it - but it does so in the user context. ADFS is operating under the machine context. So when I look at the certificate or run some certutil commands against it - it all checks out because I'm in my user security context. If I launch a CMD prompt with AT scheduler and run the same commands or run the Diagnostic tool - I find the error. The local computer does not have permissions to the private key of the client authentication certificate.

I was able to re-issue the certificate and mark the private keys as exportable, then do an export/import operation from the user store to computer store and everything worked as expected.

Since Client Authentication certificates are commonly used for user operations vs. computer operations - it is easy to see how others could hit this very same problem. Hopefully the errors and debug log entries will make this blog post discoverable for others hitting this.

ADFS Diagnostic Tool

jimsim — Thu, 01 Nov 2007 22:44:00 GMT

A huge thanks to the ADFS test team for developing such a great tool.

Here is a quick "how to"

The tool is very simple to use and provides a graphical UI. In order to perform distributed diagnosis, i.e. diagnose failures based on the configuration of multiple machines in the scenario, it’s necessary to copy the out file generated by the tool each time it’s run and use it as an input/output file when running the tool on the next machine.

For example, to debug a scenario with an FS at the account role (FS-A), an FS at the resource role (FS-R) and a Web Server (WS), first run the tool on the FS-A selecting a new file, say adfsdiag.out. After the tool is run, this file will now contain configuration information relative to the FS-A. Copy the file to the FS-R machine and run the tool there, this time selecting the existing adfsdiag.out file. The tool will detect it already contains information relative to other roles and will execute extra configuration checks, for example, a claim flow check that verifies the outgoing claims sent by the FS-A match the incoming claims expected by the FS-R. After this second run, adfsdiag.out will contain information relative to both the FS-A and FS-R. Finally, copy the out file to the WS machine and run the tool again following the same steps. When running the tool for a role for which there’s already information present in the selected file, the old data for that role will be overwritten with the new information, making it possible to fix errors on a machine and re-run the tool without having to start the whole process all over again. There’s no “right order” to run the tool, all of them should give the same output, except for some certificate checks that will only be executed at the WS in case the information from the FS-R is available beforehand

Please give this tool a try and provide any feedback to this blog.

Enabling debug logging for Claims Aware Applications

jimsim — Fri, 10 Aug 2007 22:16:00 GMT

Place the following in your applications web.config file. Place this after the </system.net> section of the file.

<system.diagnostics>

</switches>

</listeners>

</trace>

</system.diagnostics>

IFSEXT.DLL and the dialog box that is so very WRONG

jimsim — Fri, 27 Apr 2007 03:21:00 GMT

Ifsext.dll is the ADFS ISAPI used by the Token based Web Agent...We have seen issues before where we either need to add this manually or move it to the top of the list on the application config section of IIS.

Once you go the properties of a web site, the Virtual Directory tab has a button labeled Configuration.

The bottom section of the dialog has a box that is labeled Wildcard application maps (order of implementation). This is where you may need to insert the ifsext.dll file. When you do this - the box below is presented and you can browse to the needed file.

For ADFS - this file must be at the top of the list. Using the token based applications for SharePoint 2007 - this is a common "gotcha" - the ifsext.dll is below the Framework 2.0 ISAPI. After setting everything up - you get an "access denied" error message from the site even though you have the proper group SID according to the ADFS logs and you have added that group to SharePoint permissions. Once you move the ifsext.dll to the top - everything works as expected. I used to think that was a whipping - not anymore...

The dialog that is wrong - so very very wrong, is the part that says “Verify that file exists”

What would you think this means? I can tell you that I thought it meant - verify the .dll file placed in here actually exist before you say OK.

Well – that is NOT what it means…

From the IIS Documentation:

Add/Edit Application Extension Mapping (for Wildcard Application Maps)

Executable

Type the name of the executable file (.exe or .dll). The executable file must be located on your Web server's local hard disk.

Browse

Click to locate your Web server's local hard disk for the ISAPI application.

Verify that file exists

Select Verify that file exists to instruct the Web server to verify the existence of the requested script file and to ensure that the requesting user has access permission for that script file. If the script does not exist or the user does not have permission, the appropriate warning message is returned to the browser and the script engine is not invoked. This option can be useful for scripts mapped to non-CGI executables, such as the Perl interpreter, that do not send a CGI response if the script is not accessible. Because the script must be opened twice, once by the server and once by the script engine, enabling this option can impact performance.

WOW! The fact is that if the file doesn’t exist in the location you specified – you will get an error either way.

I mentioned a future blog on SQL Reporting Services and that is still going to happen.

After just understanding this after several hours of troubleshooting by many different people - I felt I had to quickly write about it.

Having this box checked on the reportserver directory will make it so a report will never render if the toolbar is enabled. The request for /ReportServer/Reserved.ReportViewerWebControl.axd does not exist in the ReportServer directory.

MS Virtual Lab - A PKI troubleshooting exercise

jimsim — Sun, 25 Feb 2007 19:11:00 GMT

I was going through some old items and came across this link for an on-line ADFS lab. I decided to run through the lab (takes about an hour). There are problems with it. The title of this blog tells you what these problems are.

I can tell it's an old lab, the manual tells you to enter the federation server url like:

https://adfsserver.treyresearch.com/adfs/ls/clientlogon.aspx - specifying the clientlogon.aspx file was how things were done up until beta 3 of R2 if I remember correctly.

You can get it working (just look around and get creative with the certs)

If you want some practice working with certificates used in ADFS - then you should check it out.

I submitted feedback and will try to locate who controls this content - but it will most likely be up there and in this condition for a while.

The online lab is a cool idea - too bad this is the only one I can find (and it's broken)

The NT Token Cache

jimsim — Sat, 24 Feb 2007 23:20:00 GMT

The NT Token cache on the web server – Maybe you didn’t know this even existed…

Consider this scenario:

You are setting up ADFS in a federated scenario with SharePoint configured as a token based application.

The initial setup has miscellaneous configuration errors that you correct along the way. You test again and find some more configuration issues further down the line. Each time you correct something – you try to get to the web site with your client machine and your test account. Each time – you are getting closer and closer.

Sound familiar?

You finally make it to the SharePoint page and you are happy…No errors from the Federation Servers and things went as expected. Maybe not 100% the way you expected – but you just need to make some minor changes with the SharePoint permissions, then you are ready to test out some different claims and other items you have on your list.

OK – careful right here. Something happened under the hood here – and it’s important!

Let’s talk about what just happened when you finally made it to SharePoint error free. The ADFS token based web agent wrote a NT token on the web server and this user (identified by their identity claim) will find and use this same token on subsequent requests to applications on this box for the next 60 minutes.

See where you can run into trouble during initial setup and testing? Let's continue with this example…

Let's assume that when you first accessed the site (successfully) – you had a UPN identity claim and Group Claim A (which mapped to Windows Group A). The agent wrote a token with the SID of Windows Group A on the web server. You then realize you need to test Group Claim B which is associated with Windows Group B. You take all the correct steps necessary – add him to a different group on the account side, log off/log on, test again.

Hmmm – you are still getting the permissions associated with Group Claim/Windows Group A. You start checking your configuration - looking at logs – you see the Group claim B getting passed as it should from the FS-A to the FS-R – but when the user gets to the Web Server – you don’t have the permissions you associated with group claim B.

Now the doubt starts to creep in…Just when you thought you had the hang of this claim thing ;)

You check/double/triple check your configuration – maybe you configure group claim C – same thing! What changed you ask yourself? Trying a different user on the account side probably never occurred to you (I know it never does to me when I’m in this place).

Hopefully you read this (and remember about the NT Token Cache) before you spin your wheels too long with a scenario as I described here. The example I gave above is not the only way you can get in trouble here – It’s just one of the ways.

If you enable debug logging on the web server, you will see a message indicating that a cache entry has been found

When you are in the lab and are going to be making changes, testing, then more changes, and more testing – You may want to consider reducing the CacheEntryLifetime to the minimum (1 minute) from the default (60 minutes). To do this – add the following registry values to the web server at this location:

HKLM\System\CCS\Control\LSA\WebSSO\Parameters

CacheEntryLifetime – dword – 60 decimal

CacheScavengeInterval – dword – 60 decimal

Reboot the server for these changes to take effect.

Now – you can continue your testing without hitting this type of problem. Keep in mind – this is for lab environments only. I have no idea what this would do to a busy production web server from a performance standpoint.

A complete list of all the cache settings is located here

This blog certainly raises some questions (for me anyway) - when I tried to test things to verify and provide more detailed information, I got into a major rat hole…

I’ll follow up with more detailed information like:

How the debug logs look – how to verify this is what you are hitting
Shadow account existence – and the account partner “ resource account” setting

I think more detailed items on the subject are needed here. I’m going to put this out for now and build on it later.