Potential for Kerberos Issues When Using a Cisco VPN/ASA with Win2003 or later DC’s

Potential for Kerberos Issues When Using a Cisco VPN/ASA with Win2003 or later DC’s

  • Comments 10
  • Likes

Hey everyone, Rob Greene here back after a long hiatus from blogging. I had an interesting case come through that I thought many of you IT pros would be interested in.


The customer had an issue with using Cisco VPN and Cisco ASA concentrators and authenticating the user with Kerberos authentication. After they upgraded all their Windows 2000 domain controllers to Windows Server 2003 domain controllers, users could not authenticate through the VPN.

After doing this they gave Cisco a call and they were told to start DSA.MSC and check the box under Account Options to “Do not require Kerberos preauthentication” for all user accounts that need to connect through the appliances. When they did this as a test it did allow the user to start authenticating through the devices. However the customer did not like this option since:

1. This change lowers the security of Kerberos authentication by disabling password replay attack protection.

2. Everything had worked previously with Windows 2000.

If you want to learn more about preauthentication you can click here and look under the section:

Example AS Administration.

If you search Cisco’s website you will find that they have older documentation stating that you must disable this feature on the user’s account before the VPN will start working.

I don’t have real network traces (as I can’t show you a real customer’s network!), but I can tell you what we saw in the traces that led us to a potential workaround.

VPN Appliance --> UDP Port 88 / KRB5 AS-REQ --> Windows DC

VPN Appliance <-- UDP Port 88 / KRB5KDC_ERR_REAUTH_REQUIRED <--Windows DC

VPN Appliance --> UDP Port 88 / AS-REQ (with Pre-Auth Data) --> Windows DC

VPN Appliance <-- UDP Port 88 / KRB5KDC_ERR_RESPONSE_TOO_BIG <-- Windows DC

A KRB5KDC_ERR_RESPONSE_TOO_BIG means that the requested Kerberos data was larger than the maximum default datagram size of 1465 bytes. The next step here usually would be for the client to start the Kerberos ticketing process over using TCP rather than UDP. If this was a Windows XP client instead of the Cisco VPN appliance we would have the customer implement the registry key change outlined in the below KB article.

244474 How to force Kerberos to use TCP instead of UDP in Windows

So basically the real issue is that the Cisco appliance is being asked by the KDC to use TCP instead of UDP to do the AS-REQ with pre-authentication data present. However, the Cisco appliance does not continue at this point. I spent the past few weeks working with Cisco directly (big shoutout - they have been great!), and they have published the following new articles:

Cisco VPN 3000: CSCed27444 - VPN3000 does not switch to TCP upon receiving Kerberos error code 52

Cisco ASA platform: CSCsi32224 - ASA does not switch to TCP upon receiving Kerberos error code 52

CSCtd92673 -- "Kerberos authentication fails with pre-auth enabled"

Unfortunately, the Cisco VPN 3000 concentrator has been discontinued so there is no update available for the Cisco VPN 3000 to add this functionality. Also, you will need a support contract from Cisco to logon to those two articles and read them.

Please contact Cisco directly to find out more information about the issue, and with what IOS version it will be included in; we have provided this information to make it easier to find the content.

In the Meantime

So now the question becomes this: If that is what is going on, then how did it work when a Windows 2000 domain controller was used? And what can you do about it until an IOS update is released?

There was a change in the default datagram reply packet size used by the Kerberos Key Distribution Center (KDC) between Windows 2000 and Windows Server 2003. In Windows 2000 the default size is 2000 bytes, where it is 1465 bytes in Windows Server 2003. For those interested you can review the below KB article and find the registry key of MaxDatagramReplySize.

837361 Kerberos protocol registry entries and KDC configuration keys in Windows Server 2003

So that means this customer’s Kerberos ticket size for the user accounts is less than 2000 bytes, but more than 1465 bytes. Keep in mind that if you are reading this blog, your environment may still not work with the registry key setting at 2000. This is because the Kerberos ticket size is going to vary based on the number of groups that the user belongs to. You can review the KB below to learn more:

327825 New resolution for problems with Kerberos authentication when users belong to many groups

The only workaround that we can give a customer is to modify the MaxDatagramReplySize registry value to the size of your user’s largest Kerberos ticket size. It really depends on the group membership size of the user attempting to authenticate because of the PAC data that is included in the TGT. For those customers that have used the MaxTokenSize registry value should equal that value and restart the KDC service. If none of these acronyms are making sense you might want to check out my older blog post Kerberos for the Busy Admin.

Hey wow, this sounds like a pretty easy fix right? Well, you can actually have other problems because of this change. If your networking gear (switches, routers, firewalls) cannot handle large UDP packets without fragmenting them then all Kerberos authentication could start to fail in your environment. See KB244474 under more information to learn more about this.

Please keep in mind what you are doing when you implement the MaxDatagramReplySize registry value. You are in fact telling the KDC to never request the client send KRB5KDC_ERR_RESPONSE_TOO_BIG message back to a Kerberos client, and you should consider implementing the custom group policy setting included in KB244474 on all domain-joined machines to use TCP for Kerberos authentication always. If you are running purely Vista/2008 or later though, this is on by default.

Lastly, if you have to set this value very high, and you know that your networking gear will not support the packet sizes that had to be implemented, you may have to put a domain controller on the same IP subnet as the Cisco appliance to successfully get Kerberos tickets.

- Rob ‘I mean, you know’ Greene

  • Really good information Rob and Ned.  We recently ran into this same issue where I am (federal agency).  We were not using the Cisco VPN but another major VPN solution.  Checkpoint in our case.  I'm guessing there will be other comments from other VPN implementations too.

    We did end up going with the 244474 solution on all our XP machines to force TCP and that has been the fix.  We have had no issues with that fix and it has been on for about a month now.

    ...hey Dan you were right :)



  • Interesting info as I too have been looking at a frustrating intermittent network problem invovling a Cisco ASA 5510. I read this post with interest as I was looking to see if it was going to relate to my problem which is SMB conversations specifically SMB1 Trans2 requests getting most of the way communicated in the standard fashion then dying with a RST,ACK packet from the server seemingly out of nowhere resulting in a "The specified network name is not available." error.

    My Server is Win2K8 and the problem clients are XP and 2k but not Vista. We use desktop redirection and it sucks having to explain disappearing icons to users. A larger description is here: http://social.technet.microsoft.com/Forums/en-US/winserverfiles/thread/83abfc8d-72b6-4d18-b097-cfc804a323e7



    Gary Andraza

    Systems Coordinator

    Chan Centre | University of British Columbia


  • That sounds more like pure networking and SMB there - youc an roll the dice and try our sister site here to see what they say:


    - Ned

  • Ned,

    I will surely give that a try. Thanks for the suggestion it is greatly apprieciated.


  • Found the answer to my question via a link from the NTDebugging blog....

    If you are intrested...

    Analysis of a networking problem: The case of the mysterious SMB connection resets (or “How to not design a network protocol”)



  • Hey Mike,

    Thanks for the feed back...  Keep in mind that this Blog discusses the actual authentication to the VPN device not data flow after the VPN has been established.

    However, what you stated as your fix of forcing the Kerberos client to use TCP is a valid fix for authentication problems to different services on your network once the VPN has been established...

    This is more than likely because of a black hole router issue on the network, and causing Kerberos to always use TCP is a valid workaround.  In fact as stated in the blog Vista/2008 by default use TCP instead of UDP because of all the issues we have seen out there with customer networks and UDP packet loss.

    Rob Greene

  • Hope this thread is still alive :)

    Very useful info. There is one thing, however, that I don't understand at all. It is well known that disabling pre-authentication always solves the problem with KDC reply size. (I'm working with a customer right now who had exactly this problem with Cisco ASA). So, from technical point of view, does disabling pre-authentication changes Kerberos protocol somehow? Does it make the KDC reply smaller??? Or, are MS registry settings ignored, so that the UDP reply is somehow fragmented into smaller chunks???

    BTW, the defect CSCsi32224 "ASA does not switch to TCP upon receiving Kerberos error code 52" is not yet fixed and the new defect CSCtd92673 "Kerberos authentication fails with pre-auth enabled" is opened for this issue.

  • Hey Otipisov,

    Disabling Pre-Authentication does not "change" the Kerberos protocol.  It only takes away the requirement to have Pre-Authentication data the intial AS-REQ that the client sends.  

    It will not make the overall Kerberos ticketing smaller, only the AS-REQ and AS-REP packets would be smaller in this scenario.

    Keep in mind that really the issue is that the Cisco device is ignoring / not processing the error back from the domain controller of:  KRB5KDC_ERR_RESPONSE_TOO_BIG

    So reality is if the users Kerberos ticket is large enough you could still have the same issue because really that error code means request as TCP instead of UDP since the ticket is larger than one packet size.

    The workaround basically just changes the packet size allowed before the KDC will respond with that KDC Error code.  Again keep in mind that this is really not an error, the Kerberos client on the ASA should be smart enough to respond to the error with creating a new Kerberos ticket request using TCP.

    Also, thanks for the update on the Cisco defect number.  By chance do you have the URL for that?  Did a search and could not get a URL for this.  If we can get the URL we would be glad to get the blog updated with the number.  Unfortunately Cisco will not let you see this information unless you have some kind of contract with them, and the previous bug URL was provided by the Cisco Engineer that I talked with.

  • Hi Rob,

    Just to understand this better: both AS-REQ and AS-REP contain three "PA-ENC-TIMESTAMP" fields with the total length of 212 bytes in case pre-authentication is enabled, right? So, if pre-authentication is disabled, the AS-REP becomes smaller by 212 bytes and can fit within the 1465 bytes limit. If the AS-REP is still larger than 1465 bytes the KDC will again request TCP and the entire process will fail.

    So the real issue is the CSCsi32224 -- "ASA does not switch to TCP upon receiving Kerberos error code 52". The new defect (CSCtd92673 -- "Kerberos authentication fails with pre-auth enabled") was actually opened to track customer cases when the MaxDatagramReplySize workaround doesn't work for some strange reason.

    The url is: http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtd92673

    You need to login to access this page.

    Thank you for very useful info,


    Cisco TAC :)

  • Hey Oleg,

    First thanks for the updated URL for the Cisco defect.  We have gotten this updated on the blog.

    To answer your question, even if you do not send pre-authentication data in the AS-REQ, the KDC could still respond back with KRB5KDC_ERR_RESPONSE_TOO_BIG if the users Kerberos TGT is going to be larger than one packet size.  If the ASA does not recognize that it should resend the entire authentication again (AS) then I would suspect that the Cisco appliance will again fail to authenticate the user.  

    The users Kerberos ticket size is going to be dictated by the amount of groups the user belongs to since this information is included in the ticket as the Partial Attribute Certificate (PAC).

    Since you actually work with Cisco, I would be glad to speak with you further on this.  Please send an e-mail via the page so that we can discuss further if you need more information.


    Rob Greene