Disclaimer: All postings are provided "AS IS" with no warranties, and confer no rights. This weblog does not represent the thoughts, intentions, plans or strategies of Microsoft. Because a weblog is intended to provide a semi-permanent point-in-time snapshot, you should not consider out of date posts to reflect current thoughts and opinions.
AD Troubleshooting
There's tons of good stuff out there on Kerberos PAC verification - but with current trends showing an increase in incoming cases related to this type of issue I though it would do me good to brush up on this and link the most relevant articles together.
In short; PAC verification is the process where a member server sends a verification request to a DC to verify the Kerberos ticket of an incoming user to confirm they are members of the groups they claim to be members of.
A simplified version of PAC verification is as follows:
I.e. the purpose of the PAC verification is to confirm that the PAC hasn't been tampered with.If the attempt to verify the PAC fails for some reason, you'll see a Kerberos Event ID 7 error logged on the server (similar to the one below):
Event Type: ErrorEvent Source: KerberosEvent Category: NoneEvent ID: 7User: N/ADescription: The Kerberos subsystem encountered a PAC verification failure. This indicates that the PAC from the client <computer name> in realm <AD DNS domain name> had a PAC which failed to verify or was modified. Contact your system administrator.Data: 0000: c000005e
Note the additional data error code above may be different depending on whether you're looking at this from a member server or a domain controller:
5e 00 00 c0
C000005e
-1073741730
STATUS_NO_LOGON_SERVERS
There are currently no logon servers available to service the logon request.
6d 00 00 c0
c000006d
-1073741715
STATUS_LOGON_FAILURE
The attempted logon is invalid. This is either due to a bad username or authentication information
50 00 02 c0
c0020050
-1073610672
RPC_NT_CALL_CANCELLED
The remote procedure call was cancelled.
However, by itself it doesn't mean that there is a problem with the actual PAC in the Kerberos ticket. If the PAC verification failed it might have failed because of the following:
Of these, only the first indicates an issue with the PAC itself - the others are failing because of external factors.
There are two other important factors that come into play in PAC verification - besides network issues (typically followed by a Netlogon 5719 event which may be temporary and resolve itself or may need active troubleshooting from the networking people).
One is the default MaxConcurrentAPI setting and the other is the LsaLookupRestrictIsolatedNameLevel setting which controls how DC's treat Name2Sid requests which dont't contain a domain prefix to qualify which domain they belong to.
The default settings for both are fine for a typical AD setup. However, care should be taken to evaluate whether you need to tweak these in larger installations where you have high volume web servers (f.x. Sharepoint) that are generating a large number of authentication requests and PAC verifications.
The two settings become important when you have multiple trusts, multiple authentication attempts that don't prefix the username with an identifying domain name and massive Kerberos PAC verifications caused by busy web servers.
In this scenario, the default MaxConcurrentAPI setting effectively creates a bottleneck on both the member server and the DC side that is exacerbated further by chasing isolated names across trusts and even further if one or more of those trusts is broken.
If you couple that together with the DC's being constantly busy walking the list of trusts to locate isolated usernames then you have a scenario where the member server starts failing PAC verification because the DC eventually starts taking longer and longer to respond with the PAC verification message as the queue of incoming authentication and PAC verification requests grows - eventually exceeding the 45 second timeout limit.
In the case where there is a problem with one of the trusts then this becomes an even bigger problem - as it will cause the time the DC takes to respond to increase even further is it must wait for DC's at the other end of the trusts to respond or time out.
Note: there is a registry value that makes it possible to disable PAC verification if you want to trust the PAC without verifying it (ValidateKdcPacSignature) but this really only applies to processes that are running as a service.The IIS worker processes run as application pools that are started by the WWW service so they effectively don't run as a service.I.e. you cannot turn off Kerberos PAC verification for IIS (or Sharepoint which runs on IIS) using the registry entry (but see the first link below for a User Right which can accomplish that).
[Tons of] Links:
Understanding Microsoft Kerberos PAC Validationhttp://blogs.msdn.com/b/openspecification/archive/2009/04/24/understanding-microsoft-kerberos-pac-validation.aspx
Kerberos PAC Validation… what is it?http://blogs.msdn.com/b/spatdsg/archive/2007/03/07/pac-validation.aspx
You are intermittently prompted for credentials or experience time-outs when you connect to Authenticated Serviceshttp://support.microsoft.com/kb/975363
More Kerberos fun with PAC’s- decrypt the PAChttp://blogs.msdn.com/b/spatdsg/archive/2009/03/26/more-kerberos-fun-with-pac-s.aspx
MaxConcurrentAPI - or, how fast can you authenticate users?http://blogs.msdn.com/b/spatdsg/archive/2006/01/05/507299.aspx
Utilizing the Authorization Data in Kerberos Tickets for Access Control to Resourceshttp://msdn.microsoft.com/en-us/library/aa302203.aspx
Trusts and isolated names and logon performancehttp://blogs.technet.com/b/instan/archive/2009/07/07/trusts-and-isolated-names-and-logon-performance.aspx
You experience a delay in the user-authentication process when you run a high-volume server program on a domain member in Windows 2000 or Windows Server 2003http://support.microsoft.com/kb/906736
Updated: NTLM and MaxConcurrentApi Concernshttp://blogs.technet.com/b/ad/archive/2008/09/23/ntlm-and-maxconcurrentapi-concerns.aspx
How to configure an ISA Server computer for a very large number of authentication requestshttp://support.microsoft.com/kb/326040
Improving Web Proxy Client Authentication Performance on ISA Server 2006 http://technet.microsoft.com/en-us/library/bb984870.aspx