June, 2009

The series:

Designing and Implementing a PKI: Part I Design and Planning

Designing and Implementing a PKI: Part II

Designing and Implementing a PKI: Part III Certificate Templates

Chris here again. For those Security Architects and PKI implementers, you may have known that since Windows Server 2008 we have an Online Certificate Status Protocol (OCSP) responder, and since Windows Vista we have an OCSP client that is integrated with the operating system. I wanted to cover the in and outs of the OCSP responder, and walk through the installation.

So, you may be asking the question “OCSP what?” First a little background. One of the capabilities of a PKI and in particular a Certificate Authority, aside from issuing certificates, is to publish revocation information.

For example, let’s say you issue a User certificate to a user for authentication. When the user leaves the company you will most likely want to make sure no one can use that certificate for authentication so you log onto the Certificate Authority and revoke that certificate. Each CA has a period specified when it publishes what are called Certificate Revocation Lists or CRLs for short. When the next CRL is published it will contain the serial number of the certificate, the date and time it was revoked, and the reason that the certificate was revoked. Depending on the configuration the CA it will publish the CRL to a repository such as an LDAP server or a web server. In some instances a task or job must be created to copy the CRL to a repository.

Aside from CRLs, there are also delta CRLs. Delta CRLs simply contain the revocation information for certificates that have been revoked since the last Base CRL was published. In order to determine revocation status an application would examine the last base CRL, and the latest delta CRL. The reason for publishing delta CRLs is to provide revocation information that has more current data. Also, it can reduce bandwidth since if the base CRL is already cached on the client, just the delta CRL can be downloaded. More on this later.

In order for applications to determine if a certificate has been revoked, the application examines the CRL Distribution Point (CDP) extension in the certificate. This extension will have information on locations where the CRL can be obtained. These locations are normally either HTTP or LDAP locations.

The application then can go to those locations to download the CRL. There are, however, some potential issues with this scenario. CRLs over time can get rather large depending on the number of certificates issued and revoked. If CRLs grow to a large size, and many clients have to download CRLs, this can have a negative impact on network performance. More importantly, by default Windows clients will timeout after 15 seconds while trying to download a CRL. Additionally, CRLs have information about every currently valid certificate that has been revoked, which is an excessive amount of data given the fact that an application may only need the revocation status for a few certificates. So, aside from downloading the CRL, the application or the OS has to parse the CRL and find a match for the serial number of the certificate that has been revoked.

With the above limitations, which mostly revolve around scalability, it is clear that there are some drawbacks to using CRLs. Hence, the introduction of Online Certificate Status Protocol (OCSP). OCSP reduces the overhead associated with CRLs. There are server/client components to OCSP: The OCSP responder, which is the server component, and the OCSP Client. The OCSP Responder accepts status requests from OCSP Clients. When the OCSP Responder receives the request from the client it then needs to determine the status of the certificate using the serial number presented by the client. First the OCSP Responder determines if it has any cached responses for the same request. If it does, it can then send that response to the client. If there is no cached response, the OCSP Responder then checks to see if it has the CRL issued by the CA cached locally on the OCSP. If it does, it can check the revocation status locally, and send a response to the client stating whether the certificate is valid or revoked. The response is signed by the OCSP Signing Certificate that is selected during installation. If the OCSP does not have the CRL cached locally, the OCSP Responder can retrieve the CRL from the CDP locations listed in the certificate. The OCSP Responder then can parse the CRL to determine the revocation status, and send the appropriate response to the client.

OCSP Components

OCSP Client

The OCSP Client is a component that generates OCSP requests based on information stored in the AIA extension of the certificate it is validating. The Windows OCSP client supports the Lightweight OCSP Profile as specified in RFC 5019.

OCSP Responder

Web Proxy Cache

Web Proxy Cache is the Web service that receives requests, sends and caches responses.

Online Responder Service

The Online Responder Service is the component that is responsible for managing the configuration of the OCSP responder, retrieving revocation information from the Revocation Providers, signing responses, and auditing changes to the configuration of the OCSP responder (if configured to do so).

The Online Responder service runs under the Network Service account. When you create the Revocation Configuration you will assign the Signing Certificate that will be used by the Online Responder Service to digitally sign the responses sent back to a requesting client. If you are utilizing the OCSP in conjunction with an Enterprise CA you can choose to enroll for the signing certificate during the Revocation Configuration setup, and you can also choose to automatically reenroll for signing certificates. This eases management because the Signing Certificates are generally set to be valid for a short period of time.

The reason for the short validity periods is that the OCSP signing certificate contains the id-pkix-ocsp-nocheck extension. This extension tells the client that the certificate is valid for its entire lifetime so the revocation status of the certificate is never checked. The reason why this extension is included is to avoid circular revocation checking. If this extension was not included, the client would contact the OCSP Responder to verify the revocation status for a certificate. The OCSP Responder would then respond with a signed request. The client would then have perform revocation checking for the certificate that signed the response, before finishing revocation checking for the original certificate. At this point if there was an OCSP location specified for the signing certificate, you would run into a loop where the OCSP client would ask for the revocation status for the signing certificate from the OCSP and get a signed response. Then the client would again have to validate the revocation status for the signing certificate. This would occur over and over again. Or alternatively, if a CDP location was specified for the signing certificate, you would then need to download the CRL, and verify the signing certificate, in effect making the OCSP pointless, since you would have to download a CRL to validate the OCSP Signing Certificate. We avoid all of this with the inclusion of the id-pkix-ocsp-nocheck extension.

So, since we are not checking revocation status for the OCSP Signing certificate you should have a short validity period for the OCSP Signing Certificate to increase security.

Regardless you will have to the give permissions to the private key of the OCSP Signing Certificate to the Network Service Account since that is the identity under which the service runs. If you are using the OCSP with a Windows Server 2008 Enterprise CA, in the Request Handling tab of a Version 3 Certificate Template there is the option to Add Read permissions to Network Service on the private key. This option is enabled by default on the OCSP Response Signing template.

If you are using a certificate issued from a Windows Server 2008 Standalone CA, a Windows Server 2003 Enterprise CA or a Windows Server 2003 Standalone CA, you will need to manually grant permissions to the private Key of the OCSP Signing Response Certificate to the Network Service account.

1. To manually give the Network Service Account access to the private key, open up the Certificates MMC targeted for the Local Computer.

2. Right click on the certificate, then select All Tasks from the context menu, and then select Manage Private Keys…

3. Click Add on the Permissions dialog box.

4. Type Network Service, and then click Check Names to resolve the name. Then click OK.

5. The Network Service only needs read permissions to the Private Key, so deselect the Allow privilege for Full Control, and verify the Allow privilege is granted for Read, and click OK.

Revocation Configuration

A Revocation Configuration contains PKI components required to respond to an OCSP request. These include items such as the CA Certificate, OCSP Signing Certificate, and information about the Revocation Provider.

You can have multiple Revocation Configurations per OCSP Responder allowing the OCSP Responder to provide revocation information for multiple CAs.

When configuring the Revocation Configuration for the OCSP Responder you will designate the following

• The certificate of the CA for which you are providing revocation status
• The Signing Certificate (If the CA is an Enterprise CA, and you are using a certificate template)
• The Revocation Provider (Limited to Base and Delta CRLs in Windows Server 2008)

Revocation Provider is the component responsible for retrieving revocation information. In Windows Server 2008 the only revocation provider supported is the CRL based Revocation Provider. In other words the Windows Server 2008 OCSP Responder can only retrieve revocation information from published CRLs.

High Availability

OCSP Responders can be configured for high availability by placing the OCSP responders in an Array. The Array itself does not provide fault tolerances, but maintains the configurations of multiple OCSP responders that are part of the Array. The configuration is maintained by the OCSP Responder that is designated as the “Array Controller”.

Once the responders are arranged in an Array you can use Network Load Balancing to provide a highly available configuration.

I will cover the process of creating a highly available OCSP configuration in a future blog article.

Conclusion

I hope you found the information in this posting helpful. I plan on continuing the series on deploying an OCSP Responder. I will be posting the following blog entries soon, stay tuned:

Implementing an OCSP responder: Part I Introducing OCSP
Implementing an OCSP responder: Part II Preparing Certificate Authorities
Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 
Implementing an OCSP Responder: Part V High Availability
Implementing an OCSP Responder: Part VI Configuring Custom OCSP URIs via Group Policy

- Chris Delay

Warren here again. The purpose of this blog post is to outline the proper steps to move from an unsupported one-way replication deployment when using DFSR from Windows 2003 R2 and Windows 2008 to a supported configuration of two-way replication.

Before we get started, here is some good news. Starting with Windows 2008 R2 we will now have Read Only replicas support in DFSR. Details of this feature can be found here:

https://blogs.technet.com/filecab/archive/2009/01/21/read-only-replicated-folders-on-windows-server-2008-r2.aspx

Recovering from One Way Replication

First let’s define some terminology so the rest of this post will make more sense.

• Upstream server – For this article “Upstream” server will refer to the DFSR server that is able to replicate changes to its partner.
• Downstream server – For this article the “Downstream server” will refer to the DFSR server that is not allowed to replicate changes to its partner.

Note: In practice any server in a multi-master replication implementation can be Upstream or Downstream. It all depends on who has the changes “Upstream” and where the change is getting replicated to “Downstream”.

• RG – Replication Group - a set of servers, known as members, that participates in the replication of one or more replicated folders
• RF – Replicated Folder - defines a set of files and folders to be kept in sync across multiple servers in a replication group

See these links for more information on RG’s and RF’s

http://msdn.microsoft.com/en-us/library/bb540026(VS.85).aspx

http://technet.microsoft.com/en-us/library/cc759803.aspx

What Not To Do

1. You do not want to simply enable replication from the Downstream server to the Upstream server. That is a BAD idea. The reasons why are listed in the section below “Why One-way Replication is not Supported”. You may end up with very old data in your Replicated Folder and some upset users wondering why they are looking at last year’s report.
2. Do not remove and re-add the Downstream server to the Replication Group again. It is a common misconception that doing this will trigger DFSR to perform an Initial Sync and Initial Replication.

What To Do

To successfully recover from a one way replication deployment we must force the downstream server to perform an Initial Synchronization and Initial Replication of the RF so that all queued changes on the Downstream server are discarded.

I will provide two methods for you to consider. These would be the ones I would use based on key considerations:

1. How divergent is the data?
2. How slow are the links?

Method 1 will generate some file replication traffic as the downstream member is most likely out of date for at least some of its data. If the data is fairly consistent then I would use this method

Method 2 will generate the least amount of file replication as you will be pre-staging a recent backup of the data taken from the Upstream server on the Downstream. I would use this method if the data is very different or the degree of divergence is unknown but suspected to be high. Also if the links are slow Method 2 is a good choice.

If you use method 2 make sure you read Ned’s blog post on the proper method of pre-staging files. If you pre-stage the files incorrectly you will generate more replication traffic than if you had just used Method 1.

http://blogs.technet.com/askds/archive/2008/02/12/get-out-and-push-getting-the-most-out-of-dfsr-pre-staging.aspx

Whatever you do, always backup the data on your Upstream and Downstream server before making any changes to your one way replication configuration. Backing up your data is a matter of best practice so you should be doing this nightly already. As with any setup of replication, the work should be done off hours or on a weekend when possible to minimize user interruption.

Patching

Make sure your DFSR servers patch levels are up to date per the KB Articles linked below before implementing any other changes. Do not skip this point.

2003 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;958802

2008 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;968429

Important Note: DFSR stores its configuration in AD. When changes are made to the DFSR configuration the update will take place on the DC that the DFSR server is connected to. Those changes are then replicated to all DC’s in the domain. DFSR will pick up the change on its own during its next poll of AD.

In the methods mentioned below all changes to the DFSR configuration will be made on the Upstream server. We will then force AD replication and finally force a poll of AD on the DFSR servers. The steps to do this is listed below and the referenced in both Methods 1 and 2.

Forcing AD Synchronization

a. To find out what DC a DFSR server is connected to, use WMIC.

WMIC /namespace:\\root\microsoftdfs path DfsrReplicationGroupConfig get LastChangeSource

b. To force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain run this command on the DC returned in step A. i.e <dc name> = the DC returned in step A.

repadmin /syncall /d /e /P < dc name> <Naming Context>

c. On the Upstream and Downstream servers run this command. “DFSRDIAG Pollad”.

Tip: If you want to remote the pollad command you can by specifying the target server with the “/mem:” switch.

Method 1 – Disabling the Downstream server’s membership in the RF to force Initial Sync and Initial Replication when the membership is re-enabled

1. Get a full backup of the Replicated Folder(s) from the Upstream server.

2. If there are changes you want to keep on the Downstream server, back them up now.

Note: Any files that are different or unique on the downstream will be moved to the ConflictAndDeleted or pre-existing directories. The data in ConflictAndDeleted will be permanently removed if the quota is reached. The quota by default is 660 MB. Make sure you backup the Downstream if there is any data there you want to keep. See http://technet.microsoft.com/en-us/library/cc782648.aspx

3. On the Upstream sever open the DFSR management snap-in and highlight the Replication Group that has the affected Replicated Folder. Make sure you are on the Memberships Tab. Right click the Downstream server and select Disable.

Figure 1 - Disabling the Downstream servers membership in the Replication Group

You will get prompted for verification that you want to disable the membership of the server. Depending on if the RF is published in a DFS Namespace or not you will get a different set of prompts.

If your RF is not published you will see only the prompt in Figure 2

Select yes.

Figure 2 - Disable Membership prompt when the Replicated Folder is not published in a DFS namespace

If your RF is published in a DFS Namespace you see the prompts in Figure 3 and 4.

Click yes and Ok.

Figure 3 First popup you will get disabling membership on an RF that is published via a DFS namespace

Figure 4 Second popup that you get disabling membership on a RF that is Published via a DFS Namespace

4. On the Upstream server use the DFS Management console to enable the connection from the Downstream server to the Upstream server. This is done on the Connections Tab. If there is no connection from Downstream to Upstream create it at this time.

Figure 5 - Enabling the connection from Downstream to Upstream

5. On the Upstream find out what DC it is connected to -Step A in Forcing AD Synchronization.

6. Force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain. Step B in Forcing AD Synchronization.

7. Force the Upstream and Downstream servers to poll AD. Step C in Forcing AD Synchronization.

Once the downstream server detects the change to its membership the downstream server will then log events 4114 and 4008. Once you confirm these events are logged you can proceed to step 8. Event 4114 informs the admin that the data in the in the RF will be seen as pre-existing. The data will be treated like any other pre-staged data during Initial Sync and Replication when its membership is re-enabled.

Figure 6 Event ID 4008 logged

 

Figure 7 Event ID 4114

8. Enable the membership of the Downstream server in the Replicated Folder using the DFS Management snap-in. This is located on the Memberships Tab.

Figure 8 - Enabling the membership of the Downstream server in the Replication Group.

Depending if your RF is published in a DFS Namespace or not you will a get set of different prompts.

If the RF is not published in DFS you will get prompted with the dialog box in figure 9. Verify the path and click OK.

Figure 9 Enabling Membership on a server where the RF is not published via DFS

If your RF is published in a DFS namespace you will get the popup in figure 10. You will not be able to click OK until you set the share permissions and share name by clicking on the Edit button. Clicking in the Edit button will bring up the popup in figure 11. Set the perms and share names as needed (defaults are fine if suitable). The extra prompts are presented when the RF is published because the member is being added again as a target for the folder in the DFS namespace.

Figure 10 First popup presented when enabling membership on an RF that is published in DFS.

Figure 11 Second popup displayed when enabling membership on an RF that is published in DFS

9. Repeat steps 5-7. After those steps are done the Downstream server will log Event 4102 when Initial Replication begins and 4104 when it is complete. You will also more than likely see 4412 events due to the servers having different versions of some files. If the data is very different on the servers you will see a large amount of these.

Figure 12 Event ID 4102

Figure 13 Event ID 4412 “Conflict Event”

Figure 14 Event 4104

10. Once you see event ID 4104 on the Downstream server for the replicated folder(s), you are finished with setting up two-way replication.

Method 2 – Total Recreation of the RG using pre-seeded data on the Downstream server.

1. On the Upstream server get a full Backup of the replicated folder.

2. Ship the Backup to the Downstream server’s site.

3. On the Upstream server delete the Replication Group.

4. On the Upstream server find out what DC is connected to. – Step A in Forcing AD Synchronization.

5. Force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain. – Step B in Forcing AD Synchronization.

6. On the Upstream and Downstream servers force them to PollAD. – Step C in Forcing AD Synchronization.

7. On both the Upstream and Downstream server s you will log events 4010 and 3006 noting that the RF and RG have been removed from the configuration:

 

Figure 15 Event ID 4010 logged when a RF is removed from a RG

Figure 16 Event ID 3006 Logged when a RG is removed

8. Pre-seed the backup on the downstream server. See Ned’s blog post on how to do this correctly: http://blogs.technet.com/askds/archive/2008/02/12/get-out-and-push-getting-the-most-out-of-dfsr-pre-staging.aspx. In my experience most pre-seeding attempts that fail are due to mismatched NTFS permissions. The entire directory tree must have matching ACLs or the files will be different and file replication will occur. See Ned’s post for options on how to get ACLS to match. (I like icacls.exe but there are more options)

9. On the Upstream server recreate the Replication Group and Replicated Folder(s), specifying that the Upstream server is the Primary server for the content. Make sure to set your staging areas to the largest size possible up to the size of the RF if possible.

10. Repeat Steps 4-6. The upstream server will log event 4112. The Downstream server will log 4102 when initial replication begins and 4104 when it is finished:

Figure 17 Event ID 4112 logged on the Primary Member of a new RF.

Figure 18 Event ID 4102 on the downstream when the RF is initialized

Figure 19 Event 4104 on the downstream when Intial Replication is finished.

11. Once you see the event ID 4104 on the downstream server Initial Replication is done and you are finished with your task.

Why One-way Replication is not Recommended or Supported

Here is a snippet from the blog post by DFSR PM Mahesh Unnikrishnan covering the reasons why it is not recommended nor supported to use one way replication in DFSR in Windows 2003 R2 and Windows 2008. The full post can be found here:

https://blogs.technet.com/filecab/archive/2007/08/16/using-one-way-connections-in-dfs-replication.aspx

“We recommend that customers avoid configuring such one way connections to the extent possible since:

a)       The DFS Replication service’s conflict resolution algorithms are severely hampered if the outbound connection from a member server is deleted (or disabled). Therefore, scenarios where the DFS Replication service is unable to over-write undesired updates occurring on the ‘read-only’ member server with the authoritative contents of the hub/datacenter server may arise.

b)       Accidental deletions on the ‘read-only’ server (in this case, site server ‘design.contoso.com’) could cause issues with the replication updates being trapped on that server. Further, as described above, updates from the authoritative server can potentially not be applied since the parent folder could have been deleted locally. Therefore, with time it is possible to see substantial divergence in the contents of the replicated folders across all replication member servers.

c)       Problems with the deployment are difficult to detect without regular and meticulous monitoring. There might be a lot of false positives in the health report and system eventlogs owing to the fact that the replication topology is being set up to do something DFSR wasn’t designed to handle. Mining through these false positives and monitoring servers can be a challenge.

d)       Administrators need to develop their own scripts to identify which files are backlogged on the ‘read-only’ member (in this case site server ‘design.contoso.com’) and replicate authoritative content back to that ‘read-only’ site server. This can be quite tricky to get right and might need a lot of very close monitoring (perhaps, at times on a per-file basis). Microsoft does not supply any tools for this purpose.

e)       There is a risk of administrators inadvertently creating the missing connection and causing backlogs to flow to and corrupt the contents of an authoritative server. With these changes getting replicated out further from the authoritative server, the contents of the replicated folder could get out of sync and corrupt on all replication member servers very quickly.

“Please note that configuring one way connections is not a configuration supported by Microsoft Product Support Services.”

Reducing Unwanted Changes

To reduce unwanted changes at certain servers you have a few items in your arsenal. These work for UNC connections.

1. If it is not necessary to share the data on some servers, don’t share it. DFSR does not require that a replicated folder be shared.

2. Set Share permissions (not NTFS permissions) to read only.

3. If you never want anyone accessing the data through the DFS Namespace, disable the referral for that target or delete the server as a folder target.

Hopefully you will never find yourself in the situation where you need to use the information in this blog post.

- Warren “Don’t Call Me Warren” Williams

Hey Rob here again, I thought that I would share with you some of the things that we see where Internet Explorer Kerberos authentication fails.

It is important to understand the default behavior of Internet Explorer and its support for Kerberos authentication so that you don’t start ripping out your hair (can’t speak to what Ned does here). I have listed three very common problems that we typically see when Kerberos authentication is failing with web-based applications.

Scenario 1: Website does not use the standard TCP/IP port of (80/443)

Web Server Configuration:

• Webserver1 has two different IIS sites running on it.
• Website1 runs with a web application pool account assigned to NetworkService.
• Website2 runs with a web application pool account assigned to a domain user account.
• The website2 is configured to listen on Port 8080.
• The following SPN’s have been defined on the website2 application pool account.
  • http/webserver1.contoso.com:8080
  • http/webserver1:8080

In this scenario you can see why a non-standard port is being used since multiple websites have been configured on the same web server. When this happens you need to specify the port to be used when you add the Service Principal Name, otherwise there is going to be a high likely hood that you will get a Kerberos ticket for the wrong web application pool account.

Client Workaround

In this scenario you need to make sure that when Internet Explorer accesses Website2 that it asks for a Service Principal name with the port number defined. However, the default behavior of IE is to not add the port number to the Kerberos ticket request. When this ticket is presented to IIS you will see a KRB_AP_ERR_MODIFIED message back.

You will need to use the below KB article change the default behavior on all IE client versions. For Internet Explorer 6 it will require the QFE Brach of Wininet.dll to be installed before the registry change will actually work.

908209 Internet Explorer 6 cannot use the Kerberos authentication protocol to connect to a Web site that uses a non-standard port in Windows XP and in Windows Server 2003 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;908209

I can tell you that there is not a version of this KB article for IE7 and above, but you do have to make the same registry change for these versions of IE also.

Server Workaround:

There really is not a good workaround to the issue other than to use host headers for one of the websites and adding a DNS HOST record for the host header in your DNS configuration. You will see shortly why we are not recommending a CNAME record in DNS.

Scenario 2: CNAME DNS RR is used for a website.

Web Server Configuration:

• Webserver1 has two different sites running on it.
• Website1 runs with a web application pool account assigned to NetworkService.
• Website2 runs with a web application pool account assigned to a domain user account.
• Website2 is configured to us a host header of app1.contoso.com.
• In DNS a CNAME record for app1.contoso.com was created and pointed to webserver1.contoso.com HOST record.
• The following SPN’s have been defined on the website2 application pool account.
  • http/app1.contoso.com
  • http/app1

In this scenario it appears that this should work just fine. When a user goes to app1.contoso.com the client machine is going to do a DNS lookup, and the DNS server is going to respond with the CNAME record and point to the webserver1.contoso.com HOST record. We can also see that the Service Principal Name configuration is properly configured on the web application pool account for website2.

The default behavior of Internet Explorer is to generate the Kerberos ticket request for the HOST record that is returned from a CNAME record, not the actual CNAME record itself. So IE specifically asks for a Kerberos ticket for http/webserver1.contoso.com which will result in a Kerberos ticket being generated encrypted with the WebServer1 computer’s password. This will in turn generate a KRB_AP_ERR_MODIFIED from IIS back to IE when the user attempts to visit the app1.contoso.com website.

Client Workaround

You will need to use the KB articles below to change the default behavior on all IE versions. For IE 6 it will require the QFE Brach of Wininet.dll to be installed before the registry key change will actually work.

For Internet Explorer 6:

911149 Error message in Internet Explorer when you try to access a Web site that requires Kerberos authentication on a Windows XP-based computer: "HTTP Error 401 - Unauthorized: Access is denied due to invalid credentials" - http://support.microsoft.com/default.aspx?scid=kb;EN-US;911149

For Internet Explorer 7 and above:

938305 Error message when you try to log on to a Web site that requires Kerberos authentication by using Internet Explorer 7: "Access is denied due to invalid credentials" - http://support.microsoft.com/default.aspx?scid=kb;EN-US;938305

Server Workaround:

The only work around is to remove the DNS CNAME RR and replace it with a HOST RR.

Scenario 3: Website works on first access but starts failing 30 minutes later

Web Server Configuration:

• Computer Webserver1 one site on it.
• Website1 runs with a web application pool account assigned to a domain user account.
• The following SPN’s have been defined on the website1 application pool account.
  • http/webserver1.contoso.com

When a user visits the website they use the NETBIOS computer name for the URL to visit. For example: http://webserver1.

In this scenario there does not seem to be much wrong here, except that there is only the FQDN version of the Service Principal Name defined. Yeah, I know all of our documentation around Kerberos always states to add FQDN as well as NETBIOS name versions of the SPN. Believe it or not, we see this all the time where the user did not register both of them, but stick with me here.

The default behavior of Internet Explorer is to add on the computer’s DNS suffix or use DNS suffix search order if defined on the machine to whatever the user types in the URL if it is not a dotted name. If your DNS configuration is correct it will resolve to webserver1.contoso.com. Once IE finds this name it stores the DNS entry in its own DNS cache. Just like most caches it times out - for IE’s cache it is 30 minutes. After 30 minutes IE again has to resolve the name, however the next time it does not try to resolve the name through DNS again, it tries just NetBIOS name resolution (hopefully there is WINS in the environment; otherwise it will just fail). Based on your configuration you could expect one of the following Kerberos errors:

• KRB_AP_ERR_MODIFIED – Expect to get this error if web site name is the same name as your web server’s computer name. That is because it is going to ask for an http/webserver1 SPN and will resolve to HOST/webserver1 which is assigned to the computer account.
• KRB_ERR_S_PRINCIPAL_UNKNOWN – Expect to get this error if the web site name is something like app1.contoso.com. That is because it is going to ask for an http/app1 SPN and will not resolve to any account in the domain.

Client Workaround:

Thankfully there is a fix that can be implemented for Internet Explorer.

899417 You may receive an "Access is denied" error message you use the WWW-Authenticate: Negotiate method of HTTP authentication to connect to a Web server - http://support.microsoft.com/default.aspx?scid=kb;EN-US;899417

I will tell you that there is not a version of this KB article for IE7 and above, but you do have to make the registry key change for these versions of IE also before the functionality is supported.

Server Workaround:

You can register the NetBIOS version of the Service Principal Name to the account, using SETSPN.EXE.

I hope that you found this post interesting. As always it is easier to spot these type of issues by reviewing network trace taken at the client side (where IE is being used) to find the root cause of the issue.

- Rob “I Speak Tampa” Greene

Ned here. I don't feel like being particularly technical today, so check out a good chuckle:

http://www.codesqueeze.com/the-ultimate-top-25-chuck-norris-the-programmer-jokes/

If you don't know who Chuck Norris is:

1. What is your problem?
2. He is the greatest actor of our generation.
3. He does not sleep. He waits...

- Ned 'The Octagon' Pyle

Mike here. PolicyMaker customers rejoice—Microsoft has a way for you to migrate from PolicyMaker 2.x to the new Group Policy preferences released with Windows Server 2008 and included in the Remote Server Administration Tools for Windows Vista Service Pack 1 or higher.

Download GPPMIG: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=35791cb6-710b-48c4-aaa1-90db170bcf2a

PolicyMaker to Preferences… how to get there

If you’ve been using PolicyMaker then you already know how to use Group Policy Preferences. It is all managed using the Group Policy Management Console included with Windows Server 2008 or, using Windows Vista Service Pack1 by installing the Remote Server Administration Tools. However, Group Policy preferences cannot process PolicyMaker data and vice versa. Therefore, you need to have a strategy to migrate from PolicyMaker to Group Policy Preferences. Hopefully, this should help. Everything discussed below is also included in the GPPMIG installer as the ‘GPPMIG Migration Guide'.

Prerequisites

I want to take a few minutes to discuss some of the prerequisites before we jump right into the migration strategy. We have two categories Management and Client.

Management

Policymaker’s management looks and feels the same as managing other Group Policy setting. The same look and feel returns using Preferences. One thing to consider is-- each instance (PolicyMaker or Group Policy Preferences) cannot edit the others data. For this reason, you may need to leave one or more Windows XP computer, with the PolicyMaker administrative tools installed, until you’ve completed your migration. If your migration follows a staged approach, then you may encounter a small period of time where you may need to manage using both Windows Vista and Windows XP. Or, you may be the weekend warrior type and have your migration complete from Friday to Monday. The choice and freedom are there, but the requirement remains—PolicyMaker administrative additions can only edit PolicyMaker items. Server 2008 and the RSAT tools can only edit Preferences. Read Microsoft Knowledgebase article 941314, Description of Windows Server 2008 Remote Server Administration Tools for Windows Vista Service Pack 1 for more information.

Client

The critical component that actual makes PolicyMaker and/or Preferences work is the client side extensions (CSEs), which you must install on the client computer. The CSEs make normal Group Policy processing PolicyMaker/Preferences aware. The same rules apply to the client portion—PolicyMaker CSEs only process PolicyMaker data and Preference CSEs only process Preference data. Also, installing the Group Policy Preference CSEs automatically removes PolicyMaker CSEs. The new Group Policy preference client side extensions installs on

• Windows Vista RTM and Service Pack 1
• Windows Server 2003 Service Pack 1
• Windows XP Service Pack 2

Both Windows Server 2003 and Windows XP require the installation of XmlLite prior to installing the CSEs. Preference CSEs are included in Windows Server 2008. Read Microsoft Knowledgebase article 943729, Information about new Group Policy preferences in Windows Server 2008 for more information.

Testing

It goes without saying—you can never test enough and this scenario is not any different. Make sure you have backups… and they actually work. If you are going to use GPMC to backup your GPOs, then remember to use the correct version. GPMC backups are not interchangeable. If you backup with pre-Server 2008 GPMC, then you must restore with the same version. Back up some of your most complex or important GPOs and then important them into isolated test GPOs in a test OU with a single user and computer. Run through your entire migration strategy—noting what works and what does not— refining the plan with each pass. All efforts spent in planning usually pay off during implementation.

Group Policy Preference Migration utility

Now that we have the planning stuff out of the way—on to the good stuff. GPPMIG is a console application developed with version 3.0 of the .NET framework. Use GPPMIG to migrate PolicyMaker items to Group Policy Preference items into the same or a different Group Policy object. GPPMIG does not migrate PolicyMaker Application or Mail Profile data as Group Policy Preferences do not included client-side extensions for these items.

What it does

Let us take a few moments to discuss how GPPMIG works. For starters, GPPMIG always uses the domain of the currently logged on user. You’ll want to remember this so you can log on with domain administrator account for the domain GPOs you want to migrate. And, you must be a domain administrator as GPPMIG write to SYSVOL and Active Directory. One last point is that GPPMIG always connects to the PDC of the user domain—for reading and writing to Active Directory and SYSVOL. So, you’ll want to run GPPMIG from a computer close (same subnet) as the PDC emulator.

With GPPMIG, you can target a single GPO to migrate or, you can choose to migrate all GPOs. GPPMIG performs a paged LDAP query to the PDC to retrieves a list of all the Group Policy objects in the user’s domain. GPPMIG then filters out any GPO in the list that is not configured for PolicyMaker items. Then, GPPMIG iterates through each GPO in the final list, looking for PolicyMaker specific client side extensions in each GPO. The entire GPO is evaluated before moving to the next. If a PolicyMaker setting is found, then GPPMIG ensures there is not an equivalent Group Policy Preference configuration, as it will not migrate PolicyMaker items into existing Group Policy Preference items. When GPPMIG completes its search for PolicyMaker items in the GPO, it then updates the Group Policy object to included Group Policy Preference client side extensions and then increases the version number for the user, computer, or both depending on what PolicyMaker items it migrated. In no way does a migration alter any PolicyMaker items for the GPO. All PolicyMaker items remain configured and available in the GPO. GPPMIG creates a migration log in the directory from which it ran.

Usage

You can use GPPMIG to migrate to Group Policy Preferences in staged approach or, you can create brand new GPOs to hold your new Group Policy Preference items and migrate to those new GPOs. The staged approach is a planned migration strategy and is the approach I’ll document here. After reading this, you should be able to alternate this strategy to best suit the needs of your environment. Generally, you’ll migrate from PolicyMaker to Group Policy Preferences in three stages (after you’ve done your testing).

• Stage 1— Identify GPOs containing PolicyMaker items and use GPMC 1.x to back up those GPOs
• Stage 2— Migrate PolicyMaker items to Group Policy Preference items in the same or a new Group Policy object. Then, deploy the Group Policy Preference CSEs to your client computers.
• Stage 3— Confirm Group Policy Preference items migrated and are successfully applying to user and computers. Use GPMC 6.0.0.1 to backup your GPOs (to a different back up location then Stage 1. Then remove PolicyMaker items from GPOs, if applicable

Download GPPMIG: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=35791cb6-710b-48c4-aaa1-90db170bcf2a

Commands

GPPMIG contains four basic commands:

• Whatif — display all the Group Policy objects that contain PolicyMaker items
• Migrate— migrates PolicyMaker items to Group Policy Preference items in the same GPO
• MigrateTo— migrates PolicyMaker items to Group Policy Preference items to a different GPO
• Remove— removes PolicyMaker items from a GPO

Stage 1 – Identify PolicyMaker GPOs

Begin your migration process by identifying GPOs containing PolicyMaker items. You can do this by using the –whatif command. Use the –all command afterwards to search all the GPOs in the user’s domain or, you can use the –name command and provided the display name of the GPO. Use GPMC to backup all of the GPOs identified to have PolicyMaker items.

Stage 2 – Migrate PolicyMaker Data to Group Policy Preferences

Next, you’ll want to migrate PolicyMaker items to Group Policy Preference items. You have a choice to migrate the setting within the same or to a different Group Policy object.

Note
The migration does not modify PolicyMaker items, regardless of the migration action you choose.

Use the –migrate command to migrate PolicyMaker items to Group Policy Preference items within the same GPO. Use the following syntax:

Gppmig –migrate –name:gpo_name

Alternatively, you replace the –name argument with –all to migrate all the GPOs in the users domain that contain PolicyMaker items.

Figure 1 GPPMIG In-place Migration

You may prefer to keep PolicyMaker GPOs separate from Group Policy Preference GPOs. You use the –migrateTo command to accomplish this task

Important
You must create the target GPO before using the -migrateTo command. GPPMIG does create the target Group Policy object.

Figure 2 - GPPMIG Source-target Migration

The –migrateTo command requires two additional arguments: -source: and –target: follows by the display name of the Group Policy object. Enclose the name of the GPO in quotes if the name contains spaces. Also, the –migrateTo command does not support the –all argument.

Deploy GPP Client

You’re now ready to deploy the Group Policy Preference client-side extensions after you’ve migrated all of your GPOs to include Group Policy Preference items. The migration does not modify any PolicyMaker items; so clients with the PolicyMaker CSE and the Group Policy preference CSEs process the same data

Note
GPPMIG does not migrate Application or Mail PolicyMaker items. Therefore, Group Policy Preference CSEs do not apply these items to users or computers. Leave the PolicyMaker CSE installed on computers that require these items and do not install the Group Policy Preference CSEs as the installation removes PolicyMaker CSEs).

You can apply Group Policy Preferences to several Microsoft operating systems. The minimum operating system requirements are:

• Windows Vista RTM or Windows Vista Service Pack 1 (32 or 64-bit)
• Windows Server 2003 Service Pack 1 or later (32 or 64-bit)
• Windows XP Service Pack 2 or later (32 or 64-bit)

Group Policy Preference client-side extensions are included in Windows Server 2008. You can use the links above to download the client-side extension installation packages. Or, you can download the extensions as an optional update from Windows Update.

Important
Remember-- installing Preference client-side extensions removes PolicyMaker Client Side Extensions.

Stage 3

The last stage in the migration process involves verifying your items migrated and apply correctly. Use GPMC to view the Group Policy object to which you migrated your items. Click the Settings tab to show the Preference items included in the GPO.

Figure 3 GPMC Settings Tab

Next, you'll want to apply the Group Policy object to your client computers. For in-place migrations, you'll want to apply the GPO to computers using PolicyMaker CSEs and computers using Preference CSEs. Also verify user PolicyMaker and Preference items apply to the appropriate user. GPOs that are targets of in-place migrations should apply items to both (PolicyMaker and Preferences). Source-target migrations migrate the PolicyMaker items to Preference items in the newly created GPO. This allows you to keep your existing PolicyMaker GPOs separate from your Preference GPOs. You apply GPOs containing Preference items to computers are users using the Group Policy Preference CSEs.

Use the Resultant Set of Policy (RSOP) management console to confirm PolicyMaker items are applying to computers or users. Use the Group Policy Results feature within GPMC to confirm Preference items are applying to computers or users.

Figure 4 GPMC Group Policy Results for testuser

The actual migration from PolicyMaker to Group Policy Preferences is complete. Computers running either Preferences or PolicyMaker should be applying their respective items. Source-target migrations contain both PolicyMaker and Preference items. After you've transitioned your client to use the Group Policy Preference CSEs, you'll want to remove the PolicyMaker data, which remains in the GPO. You can use GPPMIG with the -remove option to remove overlapping PolicyMaker and Preference items.

Figure 5 Removing PolicyMaker settings

Note
GPPMIG does not remove PolicyMaker Application and Mail items from the Group Policy object.

Source-target Migrations do not included PolicyMaker items. Therefore, once you've completed transitioning your client computers to use Preference CSEs, you can delete the source version of the GPO, which contains only PolicyMaker items.

Conclusion

You should consider backing up your Group Policy objects after you've completed your migration and cleanup of Group Policy objects. Use the Group Policy Management Console included in Windows Server 2008 and the Remote Server Administration tools to backup all of your Group Policy objects before you proceed with any further changes.

- Mike Stephens

Hey everyone, Rob Greene here back after a long hiatus from blogging. I had an interesting case come through that I thought many of you IT pros would be interested in.

Background

The customer had an issue with using Cisco VPN and Cisco ASA concentrators and authenticating the user with Kerberos authentication. After they upgraded all their Windows 2000 domain controllers to Windows Server 2003 domain controllers, users could not authenticate through the VPN.

After doing this they gave Cisco a call and they were told to start DSA.MSC and check the box under Account Options to “Do not require Kerberos preauthentication” for all user accounts that need to connect through the appliances. When they did this as a test it did allow the user to start authenticating through the devices. However the customer did not like this option since:

1. This change lowers the security of Kerberos authentication by disabling password replay attack protection.

2. Everything had worked previously with Windows 2000.

If you want to learn more about preauthentication you can click here and look under the section:

Example AS Administration.

If you search Cisco’s website you will find that they have older documentation stating that you must disable this feature on the user’s account before the VPN will start working.

I don’t have real network traces (as I can’t show you a real customer’s network!), but I can tell you what we saw in the traces that led us to a potential workaround.

VPN Appliance --> UDP Port 88 / KRB5 AS-REQ --> Windows DC

VPN Appliance <-- UDP Port 88 / KRB5KDC_ERR_REAUTH_REQUIRED <--Windows DC

VPN Appliance --> UDP Port 88 / AS-REQ (with Pre-Auth Data) --> Windows DC

VPN Appliance <-- UDP Port 88 / KRB5KDC_ERR_RESPONSE_TOO_BIG <-- Windows DC

A KRB5KDC_ERR_RESPONSE_TOO_BIG means that the requested Kerberos data was larger than the maximum default datagram size of 1465 bytes. The next step here usually would be for the client to start the Kerberos ticketing process over using TCP rather than UDP. If this was a Windows XP client instead of the Cisco VPN appliance we would have the customer implement the registry key change outlined in the below KB article.

244474 How to force Kerberos to use TCP instead of UDP in Windows
http://support.microsoft.com/default.aspx?scid=kb;EN-US;244474

So basically the real issue is that the Cisco appliance is being asked by the KDC to use TCP instead of UDP to do the AS-REQ with pre-authentication data present. However, the Cisco appliance does not continue at this point. I spent the past few weeks working with Cisco directly (big shoutout - they have been great!), and they have published the following new articles:

Cisco VPN 3000: CSCed27444 - VPN3000 does not switch to TCP upon receiving Kerberos error code 52

Cisco ASA platform: CSCsi32224 - ASA does not switch to TCP upon receiving Kerberos error code 52

CSCtd92673 -- "Kerberos authentication fails with pre-auth enabled"

Unfortunately, the Cisco VPN 3000 concentrator has been discontinued so there is no update available for the Cisco VPN 3000 to add this functionality. Also, you will need a support contract from Cisco to logon to those two articles and read them.

Please contact Cisco directly to find out more information about the issue, and with what IOS version it will be included in; we have provided this information to make it easier to find the content.

In the Meantime

So now the question becomes this: If that is what is going on, then how did it work when a Windows 2000 domain controller was used? And what can you do about it until an IOS update is released?

There was a change in the default datagram reply packet size used by the Kerberos Key Distribution Center (KDC) between Windows 2000 and Windows Server 2003. In Windows 2000 the default size is 2000 bytes, where it is 1465 bytes in Windows Server 2003. For those interested you can review the below KB article and find the registry key of MaxDatagramReplySize.

837361 Kerberos protocol registry entries and KDC configuration keys in Windows Server 2003
http://support.microsoft.com/default.aspx?scid=kb;EN-US;837361

So that means this customer’s Kerberos ticket size for the user accounts is less than 2000 bytes, but more than 1465 bytes. Keep in mind that if you are reading this blog, your environment may still not work with the registry key setting at 2000. This is because the Kerberos ticket size is going to vary based on the number of groups that the user belongs to. You can review the KB below to learn more:

327825 New resolution for problems with Kerberos authentication when users belong to many groups
http://support.microsoft.com/default.aspx?scid=kb;EN-US;327825

The only workaround that we can give a customer is to modify the MaxDatagramReplySize registry value to the size of your user’s largest Kerberos ticket size. It really depends on the group membership size of the user attempting to authenticate because of the PAC data that is included in the TGT. For those customers that have used the MaxTokenSize registry value should equal that value and restart the KDC service. If none of these acronyms are making sense you might want to check out my older blog post Kerberos for the Busy Admin.

Hey wow, this sounds like a pretty easy fix right? Well, you can actually have other problems because of this change. If your networking gear (switches, routers, firewalls) cannot handle large UDP packets without fragmenting them then all Kerberos authentication could start to fail in your environment. See KB244474 under more information to learn more about this.

Please keep in mind what you are doing when you implement the MaxDatagramReplySize registry value. You are in fact telling the KDC to never request the client send KRB5KDC_ERR_RESPONSE_TOO_BIG message back to a Kerberos client, and you should consider implementing the custom group policy setting included in KB244474 on all domain-joined machines to use TCP for Kerberos authentication always. If you are running purely Vista/2008 or later though, this is on by default.

Lastly, if you have to set this value very high, and you know that your networking gear will not support the packet sizes that had to be implemented, you may have to put a domain controller on the same IP subnet as the Cisco appliance to successfully get Kerberos tickets.

- Rob ‘I mean, you know’ Greene

Hello everyone, David Everett and Scott Goad here to discuss a recent issue that we thought you might find interesting. We were working with a customer that was trying to implement password complexity, but they were not seeing the behavior that we would normally expect.

The issue came about when trying to apply password complexity requirements, and having existing users change their password. This should force the complexity at password change, but in this customer’s case, complexity was not being enforced. Looking into the issue further, we tested with a new user account we created in DSA.MSC and noticed that password complexity worked as expected.

Lab time! We tested the password complexity requirements and started to look at one of the problem users. We were able to compare an LDIFDE dump, of the problem user with one taken from our lab that was not having the issue. At first glance, nothing looked awry, but digging deeper, we soon realized that the displayName attribute was not populated in the LDIFDE dump.

We investigated as to how the user accounts were provisioned, to better understand why this attribute was missing. The user accounts were created using SUN Identity Manager and then sync’d to Active Directory.

Why does this matter? Password complexity evaluates the sAMAccountName and displayName attributes and builds “tokens” that are used to check the password that is being presented against complexity requirements. These two attributes, in combination or separately, are used to build the tokens which are compared against the password that the user enters. In this case, having an empty displayName attribute caused password complexity to be relaxed. The policy was in place and working as expected, since it was still evaluating the sAMAccountName. Here is a sample of what you can see in the LDIFDE dump:

dn: CN=Test User 1,OU=staff,DC=top,DC=contoso,DC=com
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: Test User 1
sn: User
givenName: Test
distinguishedName: CN=Test User 1,OU=staff,DC=top,DC=contoso,DC=com
instanceType: 4
whenCreated: 20060817140027.0Z
whenChanged: 20090518142050.0Z
uSNCreated: 693960
uSNChanged: 12104216
name: Test User 1
objectGUID:: a2dz2ugl0xGQeQCAXzH4Jg==
userAccountControl: 512
badPwdCount: 0
codePage: 0
countryCode: 0
homeDrive: U:
badPasswordTime: 128856853740701244
lastLogoff: 0
lastLogon: 128856853892263744
pwdLastSet: 128871300501924190
primaryGroupID: 513
accountExpires: 9223372036854775807
logonCount: 1
sAMAccountName: t_tUser
sAMAccountType: 805306368
userPrincipalName: t_tUser@top.contoso.com
lockoutTime: 0
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=top,DC=contoso,DC=com

Based on this LDIFDE dump the only “token” that was being monitored for complexity was t_tUser in the sAMAccountName attribute. The admin was able to pass Test and User in the password. When password complexity is enforced, displayName and sAMAccountName attributes are checked. The new password is checked for “tokens”. It is important to note that a “token” must be three or more characters in length. Even then, only the whole “token” is used in the complexity check, not portions of the tokens. This comes into play when trying to evaluate tokens against displayName.

At this point, password complexity is being enforced as designed, but not what we would expect to see based on the user’s account information. The action plan that was provided to the customer was to populate the displayName attribute and test. This worked in testing, but ultimately the customer had to work with their vendor to resolve the issue with the sync process, allowing the displayName attribute to be populated from the SUN Identity Manager to Active Directory.

For more info on Password Complexity rules, see the recent blog post Understanding Password Policies.

- David Everett and Scott Goad, the Wonder Twins

I had started posting some of the interesting Directory Services-related posts from other blogs, but I figure it makes more sense to aggregate them in this weekly new KB articles posting. That way I won’t be muddying up people’s RSS feeds with reposts, since some of you subscribe to many of the blogs that I monitor. I will also include the previous week’s postings to AskDS so you can use this one post as a quick snapshot of what is going on with Directory Services.

If you have any thoughts on how we can better keep you informed about new content, drop us some feedback in the comments section.

KB Articles

Blog Posts