June, 2009

Hello everyone, Mark from DS again. With more and more companies using virtualization, such as Microsoft Virtual Server, Server 2008 Hyper-V or VMWare, in their environments these days you may end up in the following situation I recently worked on:

1) Customer wanted to roll back one of his DC’s in his test environment to basically “back out” of some changes that had been made recently. This was a single domain forest that consisted on two Domain Controllers. Both of the DC’s were running Windows 2003 SP2.

2) Virtual Machine snapshots were being taken instead of normal system state backups.

3) They restored one of the DC’s from one of the snapshots.

4) Replication was broken.

Replication symptoms consisted of the following:

1) The Netlogon service is in a paused state.

2) In the Directory Service event log a replication error was logged, Source was NTDS Replication with the Event ID 2095.

3) Also in the Directory Service event logs were two warnings, Source was NTDS General with the event ID’s 1113 and 1115.

Here are samples of the Directory Service event log events with the description of the event.

Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 2095
Date:
Time:
User:
Computer:
Description: During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers. Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC. If not resolved immediately, this scenario will result in inconsistencies in the Active Directory databases of this source DC and one or more direct and transitive replication partners. Specifically the consistency of users, computers and trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data may vary, affecting the ability to log on, find objects of interest and perform other critical operations. To determine if this misconfiguration exists, query this event ID using http://support.microsoft.com or contact your Microsoft product support. The most probable cause of this situation is the improper restore of Active Directory on the local domain controller. User Actions: If this situation occurred because of an improper or unintended restore, forcibly demote the DC.

Event Type: Warning
Event Source: NTDS General
Event Category: Replication
Event ID: 1113
Date:
Time:
User:
Computer:
Description: Inbound replication has been disabled by the user.
Event Type: Warning
Event Source: NTDS General
Event Category: Replication
Event ID: 1115
Date:
Time: 
User: 
Computer: 
Description: Outbound replication has been disabled by the user.

If you run the command repadmin /options <The DC Name> you can verify that inbound and outbound replication is disabled. You will see something similar to this:

Current DC Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL

With more and more companies using Virtualization to replace actual physical hardware, especially in test environments, I believe we are going see more issues such as this one. This can also happen in situations where you are converting physical hardware to virtual machines which we refer to as “PtoV” (physical to virtual).

First we need to understand some basic background information regarding Active Directory (AD) replication. Domain Controllers (DC’s) use Update Sequence Numbers (USN’s) to track the updates that need to be replicated between replication partners. Every time a change in made to the data in the directory the USN is incremented to indicate a change was made. For each directory the DC stores, USN’s are used to track the latest updates that a DC has received from each source replication partner. Each DC also has a table where it knows about every other DC highest USN that stores a replica of that directory partition. Each DC also has a value on its NTDS Settings object called an invocation ID. This value is used to indentify its version of its local AD database.

There are two values that use USN’s during the replication process. One is the up-to-dateness vector, the other is the high water mark. The up-to-dateness vector is a value that the destination DC maintains for tracking the originating updates that are received from its source DC’s. When the destination DC requests its updates for a directory partition it supplies its up-to-dateness to the source DC who can use that value to reduce the set of attributes it needs to send to the destination DC. The source DC will send its up-to-dateness vector value to the destination DC once the replication cycle has completed. The high water mark is a value that the destination DC maintains to keep track of the latest change it has received from a specific source DC for an object in a specific directory partition. This value prevents the source DC from sending out changes to the destination DC that have already been applied by the destination DC.

The invocation ID is a GUID value that identifies the directory database running on a DC and is maintained separately from the identity of the server object. The server object identity never changes but the identity of the directory database (invocation ID) will change when a system state is restored by using the Microsoft API’s. All the domain controllers keep track of the directory database on its source replication partners. Both the up-to-dateness vector and the high water mark refer to the invocation ID so that other DC’s know which copy of the AD the replication is coming from.

I know this can be confusing so let’s add some graphics that may help to understand this better. Let’s say we have two DC’s, DC1 and DC2. Both of these DC’s are running as Virtual Machines on a host machine running your favorite Virtualization Software. For all intents and purposes we are assuming that replication is working fine and both of the DC’s are up to date on replication. Before we start, we take a “snapshot” of DC1. As we can see below we add a new user “Jeff Smith” on DC1. The USN is incremented from 4710 to 4711 on DC1.

Now we replicate the new user to DC2. DC1 will notify DC2 that it has changes that it needs to replicate. DC2 will then request the changes and send DC1 what it thinks is DC1’s high water mark is. In this case DC2 thinks that value is 4710 so that is what it sends. When they are done replicating DC1 will send DC2 its up-to-dateness vector so DC2 will have the new value.

Now let’s suppose that other changes in the environment are occurring and replicating as they should. “Jeff” logs on and changes his password. When he does this DC2 is the DC where the change takes place. This will increment the USN on DC2 as it was 2452 and we increment the USN for DC2 to 2453.

Next we replicate that password change over to DC1. DC2 tells DC1 that it has changes it needs to get. DC1 will send DC2 what it thinks DC2’s USN is, in this case DC1 thinks DC2 is at 2452.

Once they are done replicating DC1 USN will be 5040 and DC2 will know it DC1 is at 5040. DC2 will be at 2453 and DC1 will know that value as well.

Now you want to roll that one DC back. You apply the snapshot to the DC as a restore procedure. When this happens, the invocation ID remains the same, the USN’s are “rolled back” to the time the snapshot was taken. Now when the replication process starts the “snapshot” DC requests changes from its source DC it sends the old up-to-dateness vector to the source DC. The source DC sees this value and it knows what the value should be and they are different. The value sent has a lower value then the source DC has in its table for the destination DC. The response sent back to the destination DC by the source DC basically telling the destination DC its database is out of date. When this happens we have built-in protection so that the destination DC will take measures not replicate with other. This is referred to as a “USN rollback” situation.

The protection that the USN rollback system will take will be is:

1) Pause the Netlogon service.

2) Disable the inbound and outbound replication.

To correct this situation we need to do the following on the DC that has the roll back issue.

1) Forcefully demote the DC by running dcpromo /forceremoval. This will remove AD from the server without attempting to replicate any changes off. Once it is done and you reboot the server and it will be a standalone serve in a workgroup.

2) Run a metadata cleanup of the DC that was demoted per KB article 216498 on one of the replication partners.

3) If the demoted server held any of the FSMO (Flexible Single Master Operations) roles then use the KB article 255504 to seize the roles to another DC.

4) Once replication has occurred end to end in your environment you can rejoin the demoted server back to the domain then promote to a DC.

To prevent this from happening adhere to the following best practices:

1) Do not use imaging software to take an image of the DC.

2) Do not take or apply snapshots of the DC.

3) Do not shut the Virtual Machine down and simply copy the virtual disk as a backup.

4) If you have the ability to “discard changes” as you do if you are running “Virtual Server 2005 R2”, do not enable this type of setting on a DC Virtual Machine.

5) Use NTBACKUP.EXE, WBADMIN.EXE, or any third party software that is available as long as it is certified to be AD-compatible to take system state backups.

6) Only restore a system state to the DC or restore a full backup.

References:

875495 How to detect and recover from a USN rollback in Windows Server 2003

http://support.microsoft.com/default.aspx?scid=kb;EN-US;875495

Appendix A: Virtualized Domain Controllers and Replication Issues

http://technet.microsoft.com/en-us/library/dd348479.aspx

Backup and Restore Considerations for Virtualized Domain Controllers

http://technet.microsoft.com/en-us/library/dd363545.aspx

- Mark Ramey

The series:

Designing and Implementing a PKI: Part I Design and Planning

Designing and Implementing a PKI: Part II

Designing and Implementing a PKI: Part III Certificate Templates

Chris here again. For those Security Architects and PKI implementers, you may have known that since Windows Server 2008 we have an Online Certificate Status Protocol (OCSP) responder, and since Windows Vista we have an OCSP client that is integrated with the operating system. I wanted to cover the in and outs of the OCSP responder, and walk through the installation.

So, you may be asking the question “OCSP what?” First a little background. One of the capabilities of a PKI and in particular a Certificate Authority, aside from issuing certificates, is to publish revocation information.

For example, let’s say you issue a User certificate to a user for authentication. When the user leaves the company you will most likely want to make sure no one can use that certificate for authentication so you log onto the Certificate Authority and revoke that certificate. Each CA has a period specified when it publishes what are called Certificate Revocation Lists or CRLs for short. When the next CRL is published it will contain the serial number of the certificate, the date and time it was revoked, and the reason that the certificate was revoked. Depending on the configuration the CA it will publish the CRL to a repository such as an LDAP server or a web server. In some instances a task or job must be created to copy the CRL to a repository.

Aside from CRLs, there are also delta CRLs. Delta CRLs simply contain the revocation information for certificates that have been revoked since the last Base CRL was published. In order to determine revocation status an application would examine the last base CRL, and the latest delta CRL. The reason for publishing delta CRLs is to provide revocation information that has more current data. Also, it can reduce bandwidth since if the base CRL is already cached on the client, just the delta CRL can be downloaded. More on this later.

In order for applications to determine if a certificate has been revoked, the application examines the CRL Distribution Point (CDP) extension in the certificate. This extension will have information on locations where the CRL can be obtained. These locations are normally either HTTP or LDAP locations.

The application then can go to those locations to download the CRL. There are, however, some potential issues with this scenario. CRLs over time can get rather large depending on the number of certificates issued and revoked. If CRLs grow to a large size, and many clients have to download CRLs, this can have a negative impact on network performance. More importantly, by default Windows clients will timeout after 15 seconds while trying to download a CRL. Additionally, CRLs have information about every currently valid certificate that has been revoked, which is an excessive amount of data given the fact that an application may only need the revocation status for a few certificates. So, aside from downloading the CRL, the application or the OS has to parse the CRL and find a match for the serial number of the certificate that has been revoked.

With the above limitations, which mostly revolve around scalability, it is clear that there are some drawbacks to using CRLs. Hence, the introduction of Online Certificate Status Protocol (OCSP). OCSP reduces the overhead associated with CRLs. There are server/client components to OCSP: The OCSP responder, which is the server component, and the OCSP Client. The OCSP Responder accepts status requests from OCSP Clients. When the OCSP Responder receives the request from the client it then needs to determine the status of the certificate using the serial number presented by the client. First the OCSP Responder determines if it has any cached responses for the same request. If it does, it can then send that response to the client. If there is no cached response, the OCSP Responder then checks to see if it has the CRL issued by the CA cached locally on the OCSP. If it does, it can check the revocation status locally, and send a response to the client stating whether the certificate is valid or revoked. The response is signed by the OCSP Signing Certificate that is selected during installation. If the OCSP does not have the CRL cached locally, the OCSP Responder can retrieve the CRL from the CDP locations listed in the certificate. The OCSP Responder then can parse the CRL to determine the revocation status, and send the appropriate response to the client.

OCSP Components

OCSP Client

The OCSP Client is a component that generates OCSP requests based on information stored in the AIA extension of the certificate it is validating. The Windows OCSP client supports the Lightweight OCSP Profile as specified in RFC 5019.

OCSP Responder

Web Proxy Cache

Web Proxy Cache is the Web service that receives requests, sends and caches responses.

Online Responder Service

The Online Responder Service is the component that is responsible for managing the configuration of the OCSP responder, retrieving revocation information from the Revocation Providers, signing responses, and auditing changes to the configuration of the OCSP responder (if configured to do so).

The Online Responder service runs under the Network Service account. When you create the Revocation Configuration you will assign the Signing Certificate that will be used by the Online Responder Service to digitally sign the responses sent back to a requesting client. If you are utilizing the OCSP in conjunction with an Enterprise CA you can choose to enroll for the signing certificate during the Revocation Configuration setup, and you can also choose to automatically reenroll for signing certificates. This eases management because the Signing Certificates are generally set to be valid for a short period of time.

The reason for the short validity periods is that the OCSP signing certificate contains the id-pkix-ocsp-nocheck extension. This extension tells the client that the certificate is valid for its entire lifetime so the revocation status of the certificate is never checked. The reason why this extension is included is to avoid circular revocation checking. If this extension was not included, the client would contact the OCSP Responder to verify the revocation status for a certificate. The OCSP Responder would then respond with a signed request. The client would then have perform revocation checking for the certificate that signed the response, before finishing revocation checking for the original certificate. At this point if there was an OCSP location specified for the signing certificate, you would run into a loop where the OCSP client would ask for the revocation status for the signing certificate from the OCSP and get a signed response. Then the client would again have to validate the revocation status for the signing certificate. This would occur over and over again. Or alternatively, if a CDP location was specified for the signing certificate, you would then need to download the CRL, and verify the signing certificate, in effect making the OCSP pointless, since you would have to download a CRL to validate the OCSP Signing Certificate. We avoid all of this with the inclusion of the id-pkix-ocsp-nocheck extension.

So, since we are not checking revocation status for the OCSP Signing certificate you should have a short validity period for the OCSP Signing Certificate to increase security.

Regardless you will have to the give permissions to the private key of the OCSP Signing Certificate to the Network Service Account since that is the identity under which the service runs. If you are using the OCSP with a Windows Server 2008 Enterprise CA, in the Request Handling tab of a Version 3 Certificate Template there is the option to Add Read permissions to Network Service on the private key. This option is enabled by default on the OCSP Response Signing template.

If you are using a certificate issued from a Windows Server 2008 Standalone CA, a Windows Server 2003 Enterprise CA or a Windows Server 2003 Standalone CA, you will need to manually grant permissions to the private Key of the OCSP Signing Response Certificate to the Network Service account.

1. To manually give the Network Service Account access to the private key, open up the Certificates MMC targeted for the Local Computer.

2. Right click on the certificate, then select All Tasks from the context menu, and then select Manage Private Keys…

3. Click Add on the Permissions dialog box.

4. Type Network Service, and then click Check Names to resolve the name. Then click OK.

5. The Network Service only needs read permissions to the Private Key, so deselect the Allow privilege for Full Control, and verify the Allow privilege is granted for Read, and click OK.

Revocation Configuration

A Revocation Configuration contains PKI components required to respond to an OCSP request. These include items such as the CA Certificate, OCSP Signing Certificate, and information about the Revocation Provider.

You can have multiple Revocation Configurations per OCSP Responder allowing the OCSP Responder to provide revocation information for multiple CAs.

When configuring the Revocation Configuration for the OCSP Responder you will designate the following

• The certificate of the CA for which you are providing revocation status
• The Signing Certificate (If the CA is an Enterprise CA, and you are using a certificate template)
• The Revocation Provider (Limited to Base and Delta CRLs in Windows Server 2008)

Revocation Provider is the component responsible for retrieving revocation information. In Windows Server 2008 the only revocation provider supported is the CRL based Revocation Provider. In other words the Windows Server 2008 OCSP Responder can only retrieve revocation information from published CRLs.

High Availability

OCSP Responders can be configured for high availability by placing the OCSP responders in an Array. The Array itself does not provide fault tolerances, but maintains the configurations of multiple OCSP responders that are part of the Array. The configuration is maintained by the OCSP Responder that is designated as the “Array Controller”.

Once the responders are arranged in an Array you can use Network Load Balancing to provide a highly available configuration.

I will cover the process of creating a highly available OCSP configuration in a future blog article.

Conclusion

I hope you found the information in this posting helpful. I plan on continuing the series on deploying an OCSP Responder. I will be posting the following blog entries soon, stay tuned:

Implementing an OCSP responder: Part I Introducing OCSP
Implementing an OCSP responder: Part II Preparing Certificate Authorities
Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 
Implementing an OCSP Responder: Part V High Availability
Implementing an OCSP Responder: Part VI Configuring Custom OCSP URIs via Group Policy

- Chris Delay

Chris here again. As promised I will be covering configuring an OCSP Responder to support Enterprise CA. I will also be covering validating your OCSP Configuration.

Installing OCSP Responder Role

The first step is to install the OCSP Responder Role.

To install the OCSP Responder: Open a command prompt and type: servermanagercmd.exe –install ADCS-Online-Cert.

Configuring the OCSP Responder

First we will add a Revocation Configuration to the OCSP Responder.

Right click on the Revocation Configuration and select Add Revocation Configuration from the context menu.

The Add Revocation Configuration wizard opens. Click Next to continue.

Give a Friendly Name to the Revocation Configuration, and click Next. It is a good idea to include the name of the CA for which you are setting up this Revocation Configuration, especially if this OCSP Responder will handle requests for multiple CAs.

On the Select CA Certificate page, you will need to select a CA certificate. This is where you determine the CA for which you will be providing revocation information.

Select a certificate for an Existing enterprise CA, and click Next

Select Browse CA certificates published in Active Directory, and click Browse.

Select the appropriate CA, and click OK

Next you will need to select a certificate that will be used for signing OCSP responses. For a particular Revocation Configuration, the OCSP Signing certificate must be issued by the CA for which the OCSP Responder will answer revocation status requests.

Select Automatically select a signing certificate. If you wish to automatically enroll for the OCSP Response Signing Certificate, make sure the Auto-Enroll for an OCSP signing certificate is checked. Select the certificate template that you configured for use with the OCSP Responder, then click Next.

On the Revocation Provider page, you can click Provider to select revocation providers. The Windows Server 2008 OCSP Responder can only use CRLs for revocation information. If you have the CDP Extension available in the signing certificate, the Revocation Providers will be populated from the information in the CDP Extension from the OCSP Response Signing Certificate.

You can add the repository locations for your CRLs and Delta CRLs if appropriate. By default these will be populated from information included in the CDP extension of the Signing certificate. After you have reviewed the configuration or made any changes, click OK.

That completes the initial Configuration of the OCSP Responder. If you would like to modify the configuration of the OCSP Responder, you can right click on the Revocation Configuration and select Properties from the context menu.

The Local CRL tab allows you to configure a Local CRL. You can add revocation information for certificates which you wish to consider revoked. It is recommended that you do not use this option, as it adds unnecessary complexity to the revocation configuration.

The Revocation Provider tab allows you to modify the location of the CRLs and Delta CRLs that will be used for providing revocation information.

Signing Tab

In the signing tab you can:

• Modify the hash algorithm used to sign responses.
• Do not prompt for credentials for cryptographic operations. This setting may need to be disabled if you are using an HSM to protect the private key of the OCSP Signing certificate. Disabling this setting allows you to be prompted for the password that is associated with the operator card on the HSM.
• Use renewed certificates for signing certificates. This option is enabled by default, when you use the OCSP Responder with an Enterprise CA and automatically renew certificates. If you use OCSP Responder with a standalone CA, the OCSP responder will use renewed signing certificates even if this setting is not enabled.
• Enable NONCE extension support allows the user to attach the NONCE sent in the request with the OCSP response. If this setting is used, you will not be able to utilize cached responses.
• Use any valid OCSP signing certificate. Not recommended if the OCSP Responder is supporting Vista clients since they do not support this option. This allows the OCSP responder to use any certificate that the OCSP Signing configured in the Extended Key Usage extension of the certificate. Vista clients will only accept OCSP responses that are signed by the same CA for which the OCSP Responder is providing revocation information.
• All responses will included the following Online Responder identifies: This setting determines whether a Key Hash or Subject will be included in the response. RFC 2560 specifies the structure of the response. In section 4.2.1 of the RFC it is specified that the Responder ID field can either be populated with a Name or Key hash. This setting determines which is included in the response. The Key hash is a hash of the OCSP Responder’s public key. The Name is the distinguished name of the subject of the OCSP signing certificate.

Verify OCSP Configuration

After configuring the OCSP Responder, you will want to verify that the OCSP responder is functioning properly. The easiest way to verify that the OCSP is functioning is to use the Certutil URL Retrieval tool.

First request a certificate from the CA. Place a copy of that cert on the file system, and run the following command: certutil –URL <Certificate Name>. This will open the URL Retrieval Tool

Select OCSP, and click on the Retrieve button.

If the certificate is valid you will get the following response.

If the certificate is revoked, you will get the following response.

And if it fails, the status will be listed as Failed.

You can also use the PKIView tool to verify the configurations of the OCSP Responder.

Conclusion

This concludes configuring an OCSP Responder to support an Enterprise CA. If you follow the steps listed here you now have your OCSP configured to support your Windows Server 2003 or Windows Server 2008 CA. In the next part of this series, I will be configuring an OCSP Responder to support Standalone CA.

Implementing an OCSP responder: Part I Introducing OCSP
Implementing an OCSP responder: Part II Preparing Certificate Authorities
Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 
Implementing an OCSP Responder: Part V High Availability
Implementing an OCSP Responder: Part VI Configuring Custom OCSP URIs via Group Policy

- Chris Delay

Chris here again. In Part I we covered some of the basics and background information on the reason for the OCSP Responder and a basic understanding of how the OCSP Responder functions. So now we look towards implementing the OCSP Responder. However, before we move forward with the Install of the OCSP Responder we must first configure the CA to support OCSP for revocation status checking.

As discussed in the first part of this series, the OCSP Responder provides revocation information to clients or application requesting revocation status for a specific certificate. In order for this to be accomplished there are certain prerequisites that need to be in place.

Some of the prerequisites are different depending on which version of the CA you are using, and whether you are using a Standalone or Enterprise CA.

Configuring AIA Extension to support OCSP

To advertise that revocation status information for a particular CA can be obtained via OCSP, the CA must include a pointer to the OCSP Responder in the certificate. This is done by adding an OCSP URI to the AIA extension of the certificate.

Although this is mentioned as a prerequisite, you may want to do this after the OCSP Responder is configured. The reason being is that if you issue certificates before the Responder is available you will create unnecessary traffic to the soon to be OCSP location.

1. Open the Certification Authority Snap-in on the CA, as an Enterprise Administrator.

2. Right click on the CA name, and select Properties

3. Click on the Extension Tab. From the Select Extension drop down Box, select Authority Information Access (AIA).

4. Then click on the Add… button to add the OCSP location

5. Type the location for the OCSP responder. This will typically be:

http://<fqdn of the ocsp responder>/ocsp

6. Then click OK.

7. Check the Checkbox for Include in the online certificate status protocol (OCSP) extension.

8. And click OK, to close the CA Properties.

Preparing Windows Server 2003 Standalone CA for use with OCSP Responder

OCSP Signing Certificates

In order to be able to deploy the OCSP Signing Certificate used by the OCSP Responder, there are some configuration changes that need to be made on a Windows Server 2003 CA.

A signing certificate includes the id-pkix-ocsp-nocheck extension. This extension informs the OCSP client that the OCSP signing certificate should not be checked for revocation during the lifetime of the certificate. The OCSP Signing certificate should therefore have a short lifetime. By default, a Windows Server 2003 CA will ignore the id-pkix-ocsp-nocheck extension in a certificate request and will not include that extension in the issued certificate. To change this behavior, you must allow custom extensions to be used in certificate requests.

To enable support for custom extensions, run the following command on the CA:

The extension object ID (OID) for the id-pkix-ocsp-nocheck extension is1.3.6.1.5.5.7.48.1.5. The above command instructs the CA to include that extension in the issued certificate if it is found in the request.

Preparing Windows Server 2003 Enterprise CA for use with OCSP Responder

If you plan on using a Windows Server 2003 Enterprise CA to issue the OCSP Signing Certificate you will need to follow the instructions outlined in the previous section for enabling the use of custom extensions.

If you plan on using a certificate template on the Windows Server 2003 Enterprise CA, you must have at least 1 Windows Server 2008 Enterprise CA in the environment. The reason is you will be duplicating the Version 3 OCSP Signing Template on the Windows Server 2008 CA for use with the Windows Server 2003 CA. Both the Windows Server 2003 and Windows Server 2008 CA, must be running Enterprise Edition. This is due to the fact that only Version 1 templates are supported in the Standard Editions of the Server OS.

Duplicating the OCSP Signing Template

1. Logon to a Windows Server 2008 Enterprise CA, with an account that is a member of the Enterprise Admins group.

2. Open up the Certificate Template management console (certtmpl.msc).

3. Right click on OCSP Response Signing Template, and select Duplicate Template from the context menu, as illustrated below:

4. From the Duplicate Template dialog box, select Windows Server 2003 Server, Enterprise Edition, and click OK. Selecting Windows Server 2003 Server, Enterprise Edition, creates a Version 2 Template instead of a Version 3 Template.

 
5. Give a Name to the Duplicated Template, and click OK.

6. Log on to the Windows Server 2003 CA, and open the Certificate Authority Snapin (Certsrv.msc), and right click on Certificate Templates, and select New, then Certificate Template to Issue from the context menu.

7. Select the Duplicated Template, and click on OK.

Preparing Windows Server 2008 Standalone CA for use with OCSP Responder

In the previous section on preparing the Windows Server 2003 Standalone CA, we had to enable the CA to accept custom extensions sent in the request. This was to allow us to request a certificate with the id-pkix-ocsp-nocheck extension. Windows Server 2008 natively supports the id-pkix-ocsp-nocheck extension, so there is no need to allow custom extensions. On the Windows Server 2008 Enterprise CA there is no action necessary to support the id-pkix-ocsp-nocheck extension. However, on the Windows Server 2008 Standalone CA, we need to run the following command to add support for the id-pkix-ocsp-nocheck extension:

Preparing Windows Server 2008 Enterprise CA for use with OCSP Responder

The only preparation required for the Windows Server 2008 Enterprise CA, is to give permissions to the templates to the OCSP Servers, and to make the template available for issuance.

1. Open the Certificate Template Management console (certtmpl.msc)

2. Locate the OCSP Certificate Template, Right-click, and select Properties

3. On the Security Tab, add the hostname of the soon to be OCSP Server, and give the server Read and Enroll permissions to the template. Note: A more scalable solution, as seen in the illustration below, is to create a security group, assign permissions to the security group, and add any OCSP servers to the Security Group.

4. Go back to the Certification Authority management console, Right-click on the Certificates Templates node, and from the context menu, select New and then "Certificate Template to issue.

5. Select the OCSP Response Signing Template, and select OK.

Conclusion

You should now have your Certificate Authorities configured to support the OCSP Responder as a source of revocation status. In the next part of this series I will cover installing and the configuring the OCSP Responder to support Enterprise CAs.

Implementing an OCSP responder: Part I Introducing OCSP
Implementing an OCSP responder: Part II Preparing Certificate Authorities
Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 
Implementing an OCSP Responder: Part V High Availability
Implementing an OCSP Responder: Part VI Configuring Custom OCSP URIs via Group Policy

- Chris Delay

Ned here again. Have you ever visited Snopes.com? It’s a terrific urban legend reference where they research folklore. Snopes is the place you go to find out if eating Thanksgiving turkey makes you sleepy (it doesn’t), if Coca Cola can dissolve a tooth overnight (it can’t), or if a man really did live in a Paris airport for 8 years (he did!).

Today I’m going to talk about another urban legend – that removing the Remote Differential Compression feature from Windows Vista will make your file copying faster over the network.

Background on RDC

Remote Differential Compression (RDC) is a Microsoft algorithm that was originally created for DFSR five years ago. RDC divides a file’s data into chunks by using signatures. When a file exists on two computers and the file is modified, only the differing chunks need to be sent to the other computer.

An application needs to be specifically written to support RDC. Windows Vista and Windows 7 include MSRDC.DLL to allow apps like Windows Live Messenger to use that functionality.

The feature can be turned on and off within the Control Panel “Program and Features” applet.

When turned on, the MSRDC.DLL will exist in the %SYSTEMROOT%\System32 directory. When it’s turned off, this DLL is removed.

The Myth

Unfortunately, the Internet is full of people telling you that RDC will somehow make your network communication slower. I have no idea how this got started, but this nonsense has been reprinted on thousands of websites by people unfamiliar with the Scientific Method. Folks have actually convinced themselves that turning this feature on or off has some affect on file transfer speeds. While there are a variety of things you can do to speed up Vista file copy performance, this isn’t one of them.

The Method

So after hearing this baloney for the umpteenth time, I set out to debunk it once and for all:

• I setup a Windows Server 2008 SP2 file server and a Windows Vista SP2 client as virtual guests inside a Hyper-V host.
• Each guest got 1GB of RAM and a virtual gigabit NIC.
• I dropped a 3GB ISO file onto a share on the file server.
• I used robocopy.exe (which automatically reports bytes per second and megabytes per minute) to copy that ISO file five times with MSRDC.DLL installed, then five more times with MSRDC.DLL removed.
• In between every copy, I rebooted both the client and server to ensure that there was no caching of the file that might artificially improve my results.

The Results

Here is what I found after the ten total passes, with and without RDC installed:

Hmmm… What if I sort my data points highest to lowest?

Interesting. Let’s look at the actual averages:

With MSRDC.DLL installed: 1209.6838 MB/min

With MSRDC.DLL removed: 1208.1666 MB/min

Wait – so removing RDC actually made it slower? Not really – the variance there is well within any respectable margin of error. These results mean that the two sets of copies were 99.87% identical. Removing RDC did nothing at all. There are going to be various performance differences when copying a file, depending on what else happens on the network, what the computers are doing, and I think when people claim RDC removal made things ‘faster’, it’s because they are not testing repeatedly over time to see variance.

Maybe you want more proof? Alright, let’s go to the debugger.

• I installed Notmyfault on my Vista client and configured the computer for a complete memory dump
• I started yet another file copy, then I listed out the process and service information with TASKLIST.EXE and TASKLIST.EXE /SVC
• After a few seconds, I intentionally crash dumped (the so-called “Blue Screen of Death”) the Vista computer. I then cracked open the MEMORY.DMP file in the Windows debugger

First, I listed loaded modules - all libraries and drivers loaded in the kernel memory space:

Note how MSRDC.DLL is not loaded in memory in the Kernel space. It’s still possible that a given process or service might have it loaded though, so then I listed all processes to see which ones would be interesting and likely to be involved in file copies. The TASKLIST output comes in handy here to see which PID is which hexadecimal CID value. In my case though I dumped them all just for exploratory purposes.

I know that the Workstation Service is responsible for SMB file copying, and the robocopy process is definitely doing work, so I examined those.

Note how neither process has MSRDC.DLL loaded either. It’s simply not being used, and a module that is not being used cannot possibly affect anyone. Remember, an application has to be coded to use RDC. Nothing in the Kernel, in Robocopy, or in the Workstation service uses RDC at all in Vista or Win7.

Still don’t believe me? Here is the Microsoft Remote File Systems development team stating it as well.

Changes that can truly improve file copy performance

By now you want me to get to the helpful part. Here’s a short list of some things that can improve your file copy network performance on Windows Vista and Windows Server 2008:

• Install Service Pack 2
• Install the latest NIC drivers from your vendor.
• Try disabling Receive Side Scaling, Chimney Offload, and NetDMA support, then testing like I did above to see if the results are measurably different after many copies of the same file. Note that this just disables the Windows implementation of those components – your vendor may also support them through their NIC configuration and it will need to be turned off there as well. While these components are intended to help performance, mileage can vary based on how good your hardware and vendor drivers are.
• Use robocopy.exe rather than Explorer – the price of the friendly shell showing progress and browsing folders is slightly slower performance.

Wrapup

It’s amazing that a component that was designed to speed up network file performance can somehow be vilified as a cause of bad performance; especially when it’s not even being used. I welcome people following my steps and telling me what you find out.

Don’t believe everything you read on the Internet. Unless I wrote it. :-)

- Ned ‘Rick Rolled’ Pyle

Warren here again. The purpose of this blog post is to outline the proper steps to move from an unsupported one-way replication deployment when using DFSR from Windows 2003 R2 and Windows 2008 to a supported configuration of two-way replication.

Before we get started, here is some good news. Starting with Windows 2008 R2 we will now have Read Only replicas support in DFSR. Details of this feature can be found here:

https://blogs.technet.com/filecab/archive/2009/01/21/read-only-replicated-folders-on-windows-server-2008-r2.aspx

Recovering from One Way Replication

First let’s define some terminology so the rest of this post will make more sense.

• Upstream server – For this article “Upstream” server will refer to the DFSR server that is able to replicate changes to its partner.
• Downstream server – For this article the “Downstream server” will refer to the DFSR server that is not allowed to replicate changes to its partner.

Note: In practice any server in a multi-master replication implementation can be Upstream or Downstream. It all depends on who has the changes “Upstream” and where the change is getting replicated to “Downstream”.

• RG – Replication Group - a set of servers, known as members, that participates in the replication of one or more replicated folders
• RF – Replicated Folder - defines a set of files and folders to be kept in sync across multiple servers in a replication group

See these links for more information on RG’s and RF’s

http://msdn.microsoft.com/en-us/library/bb540026(VS.85).aspx

http://technet.microsoft.com/en-us/library/cc759803.aspx

What Not To Do

1. You do not want to simply enable replication from the Downstream server to the Upstream server. That is a BAD idea. The reasons why are listed in the section below “Why One-way Replication is not Supported”. You may end up with very old data in your Replicated Folder and some upset users wondering why they are looking at last year’s report.
2. Do not remove and re-add the Downstream server to the Replication Group again. It is a common misconception that doing this will trigger DFSR to perform an Initial Sync and Initial Replication.

What To Do

To successfully recover from a one way replication deployment we must force the downstream server to perform an Initial Synchronization and Initial Replication of the RF so that all queued changes on the Downstream server are discarded.

I will provide two methods for you to consider. These would be the ones I would use based on key considerations:

1. How divergent is the data?
2. How slow are the links?

Method 1 will generate some file replication traffic as the downstream member is most likely out of date for at least some of its data. If the data is fairly consistent then I would use this method

Method 2 will generate the least amount of file replication as you will be pre-staging a recent backup of the data taken from the Upstream server on the Downstream. I would use this method if the data is very different or the degree of divergence is unknown but suspected to be high. Also if the links are slow Method 2 is a good choice.

If you use method 2 make sure you read Ned’s blog post on the proper method of pre-staging files. If you pre-stage the files incorrectly you will generate more replication traffic than if you had just used Method 1.

http://blogs.technet.com/askds/archive/2008/02/12/get-out-and-push-getting-the-most-out-of-dfsr-pre-staging.aspx

Whatever you do, always backup the data on your Upstream and Downstream server before making any changes to your one way replication configuration. Backing up your data is a matter of best practice so you should be doing this nightly already. As with any setup of replication, the work should be done off hours or on a weekend when possible to minimize user interruption.

Patching

Make sure your DFSR servers patch levels are up to date per the KB Articles linked below before implementing any other changes. Do not skip this point.

2003 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;958802

2008 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;968429

Important Note: DFSR stores its configuration in AD. When changes are made to the DFSR configuration the update will take place on the DC that the DFSR server is connected to. Those changes are then replicated to all DC’s in the domain. DFSR will pick up the change on its own during its next poll of AD.

In the methods mentioned below all changes to the DFSR configuration will be made on the Upstream server. We will then force AD replication and finally force a poll of AD on the DFSR servers. The steps to do this is listed below and the referenced in both Methods 1 and 2.

Forcing AD Synchronization

a. To find out what DC a DFSR server is connected to, use WMIC.

WMIC /namespace:\\root\microsoftdfs path DfsrReplicationGroupConfig get LastChangeSource

b. To force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain run this command on the DC returned in step A. i.e <dc name> = the DC returned in step A.

repadmin /syncall /d /e /P < dc name> <Naming Context>

c. On the Upstream and Downstream servers run this command. “DFSRDIAG Pollad”.

Tip: If you want to remote the pollad command you can by specifying the target server with the “/mem:” switch.

Method 1 – Disabling the Downstream server’s membership in the RF to force Initial Sync and Initial Replication when the membership is re-enabled

1. Get a full backup of the Replicated Folder(s) from the Upstream server.

2. If there are changes you want to keep on the Downstream server, back them up now.

Note: Any files that are different or unique on the downstream will be moved to the ConflictAndDeleted or pre-existing directories. The data in ConflictAndDeleted will be permanently removed if the quota is reached. The quota by default is 660 MB. Make sure you backup the Downstream if there is any data there you want to keep. See http://technet.microsoft.com/en-us/library/cc782648.aspx

3. On the Upstream sever open the DFSR management snap-in and highlight the Replication Group that has the affected Replicated Folder. Make sure you are on the Memberships Tab. Right click the Downstream server and select Disable.

Figure 1 - Disabling the Downstream servers membership in the Replication Group

You will get prompted for verification that you want to disable the membership of the server. Depending on if the RF is published in a DFS Namespace or not you will get a different set of prompts.

If your RF is not published you will see only the prompt in Figure 2

Select yes.

Figure 2 - Disable Membership prompt when the Replicated Folder is not published in a DFS namespace

If your RF is published in a DFS Namespace you see the prompts in Figure 3 and 4.

Click yes and Ok.

Figure 3 First popup you will get disabling membership on an RF that is published via a DFS namespace

Figure 4 Second popup that you get disabling membership on a RF that is Published via a DFS Namespace

4. On the Upstream server use the DFS Management console to enable the connection from the Downstream server to the Upstream server. This is done on the Connections Tab. If there is no connection from Downstream to Upstream create it at this time.

Figure 5 - Enabling the connection from Downstream to Upstream

5. On the Upstream find out what DC it is connected to -Step A in Forcing AD Synchronization.

6. Force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain. Step B in Forcing AD Synchronization.

7. Force the Upstream and Downstream servers to poll AD. Step C in Forcing AD Synchronization.

Once the downstream server detects the change to its membership the downstream server will then log events 4114 and 4008. Once you confirm these events are logged you can proceed to step 8. Event 4114 informs the admin that the data in the in the RF will be seen as pre-existing. The data will be treated like any other pre-staged data during Initial Sync and Replication when its membership is re-enabled.

Figure 6 Event ID 4008 logged

 

Figure 7 Event ID 4114

8. Enable the membership of the Downstream server in the Replicated Folder using the DFS Management snap-in. This is located on the Memberships Tab.

Figure 8 - Enabling the membership of the Downstream server in the Replication Group.

Depending if your RF is published in a DFS Namespace or not you will a get set of different prompts.

If the RF is not published in DFS you will get prompted with the dialog box in figure 9. Verify the path and click OK.

Figure 9 Enabling Membership on a server where the RF is not published via DFS

If your RF is published in a DFS namespace you will get the popup in figure 10. You will not be able to click OK until you set the share permissions and share name by clicking on the Edit button. Clicking in the Edit button will bring up the popup in figure 11. Set the perms and share names as needed (defaults are fine if suitable). The extra prompts are presented when the RF is published because the member is being added again as a target for the folder in the DFS namespace.

Figure 10 First popup presented when enabling membership on an RF that is published in DFS.

Figure 11 Second popup displayed when enabling membership on an RF that is published in DFS

9. Repeat steps 5-7. After those steps are done the Downstream server will log Event 4102 when Initial Replication begins and 4104 when it is complete. You will also more than likely see 4412 events due to the servers having different versions of some files. If the data is very different on the servers you will see a large amount of these.

Figure 12 Event ID 4102

Figure 13 Event ID 4412 “Conflict Event”

Figure 14 Event 4104

10. Once you see event ID 4104 on the Downstream server for the replicated folder(s), you are finished with setting up two-way replication.

Method 2 – Total Recreation of the RG using pre-seeded data on the Downstream server.

1. On the Upstream server get a full Backup of the replicated folder.

2. Ship the Backup to the Downstream server’s site.

3. On the Upstream server delete the Replication Group.

4. On the Upstream server find out what DC is connected to. – Step A in Forcing AD Synchronization.

5. Force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain. – Step B in Forcing AD Synchronization.

6. On the Upstream and Downstream servers force them to PollAD. – Step C in Forcing AD Synchronization.

7. On both the Upstream and Downstream server s you will log events 4010 and 3006 noting that the RF and RG have been removed from the configuration:

 

Figure 15 Event ID 4010 logged when a RF is removed from a RG

Figure 16 Event ID 3006 Logged when a RG is removed

8. Pre-seed the backup on the downstream server. See Ned’s blog post on how to do this correctly: http://blogs.technet.com/askds/archive/2008/02/12/get-out-and-push-getting-the-most-out-of-dfsr-pre-staging.aspx. In my experience most pre-seeding attempts that fail are due to mismatched NTFS permissions. The entire directory tree must have matching ACLs or the files will be different and file replication will occur. See Ned’s post for options on how to get ACLS to match. (I like icacls.exe but there are more options)

9. On the Upstream server recreate the Replication Group and Replicated Folder(s), specifying that the Upstream server is the Primary server for the content. Make sure to set your staging areas to the largest size possible up to the size of the RF if possible.

10. Repeat Steps 4-6. The upstream server will log event 4112. The Downstream server will log 4102 when initial replication begins and 4104 when it is finished:

Figure 17 Event ID 4112 logged on the Primary Member of a new RF.

Figure 18 Event ID 4102 on the downstream when the RF is initialized

Figure 19 Event 4104 on the downstream when Intial Replication is finished.

11. Once you see the event ID 4104 on the downstream server Initial Replication is done and you are finished with your task.

Why One-way Replication is not Recommended or Supported

Here is a snippet from the blog post by DFSR PM Mahesh Unnikrishnan covering the reasons why it is not recommended nor supported to use one way replication in DFSR in Windows 2003 R2 and Windows 2008. The full post can be found here:

https://blogs.technet.com/filecab/archive/2007/08/16/using-one-way-connections-in-dfs-replication.aspx

“We recommend that customers avoid configuring such one way connections to the extent possible since:

a)       The DFS Replication service’s conflict resolution algorithms are severely hampered if the outbound connection from a member server is deleted (or disabled). Therefore, scenarios where the DFS Replication service is unable to over-write undesired updates occurring on the ‘read-only’ member server with the authoritative contents of the hub/datacenter server may arise.

b)       Accidental deletions on the ‘read-only’ server (in this case, site server ‘design.contoso.com’) could cause issues with the replication updates being trapped on that server. Further, as described above, updates from the authoritative server can potentially not be applied since the parent folder could have been deleted locally. Therefore, with time it is possible to see substantial divergence in the contents of the replicated folders across all replication member servers.

c)       Problems with the deployment are difficult to detect without regular and meticulous monitoring. There might be a lot of false positives in the health report and system eventlogs owing to the fact that the replication topology is being set up to do something DFSR wasn’t designed to handle. Mining through these false positives and monitoring servers can be a challenge.

d)       Administrators need to develop their own scripts to identify which files are backlogged on the ‘read-only’ member (in this case site server ‘design.contoso.com’) and replicate authoritative content back to that ‘read-only’ site server. This can be quite tricky to get right and might need a lot of very close monitoring (perhaps, at times on a per-file basis). Microsoft does not supply any tools for this purpose.

e)       There is a risk of administrators inadvertently creating the missing connection and causing backlogs to flow to and corrupt the contents of an authoritative server. With these changes getting replicated out further from the authoritative server, the contents of the replicated folder could get out of sync and corrupt on all replication member servers very quickly.

“Please note that configuring one way connections is not a configuration supported by Microsoft Product Support Services.”

Reducing Unwanted Changes

To reduce unwanted changes at certain servers you have a few items in your arsenal. These work for UNC connections.

1. If it is not necessary to share the data on some servers, don’t share it. DFSR does not require that a replicated folder be shared.

2. Set Share permissions (not NTFS permissions) to read only.

3. If you never want anyone accessing the data through the DFS Namespace, disable the referral for that target or delete the server as a folder target.

Hopefully you will never find yourself in the situation where you need to use the information in this blog post.

- Warren “Don’t Call Me Warren” Williams

Chris here again. In part I of this series we covered the basics of how OCSP works. We also covered the underlying reasons for deploying an OCSP Responder. In Part II we covered configuring the Certificate Authorities for whom which the OCSP Responder will check revocation status for on behalf of the clients. In Part III we covered configuring and OCSP Responder to support an Enterprise CAs. You may use Standalone CAs in your environment. In this blog post, I will be covering deploying a Revocation Configuration to support a Standalone CA.

Enterprise CAs are very tightly integrated with Active Directory. As such the certificates for the Root CA and for intermediate CAs are published to Active Directory. These certificates are automatically placed in the appropriate certificate stores on the clients. If you publish the Root CA certificate that the issuing CA chains up to; in Active Directory the clients will have that Root CA certificate published to the Trusted Root Certification Authorities container in the user and machine store. If you have not, or do not plan to deploy the Root CA certificate through Active Directory and Group Policy you will need to manually publish the Root Certificates in the Trusted Root Certification Authority store.

Installing OCSP Responder Role

The first step is to install the OCSP Responder Role.

To install the OCSP Responder: Open a command prompt and type: servermanagercmd.exe –install ADCS-Online-Cert

Requesting and Installing the OCSP Responder Signing Certificate

The next step is to request the OCSP Response Signing Certificate from the Standalone CA. Since a Standalone CA does not have certificate templates we must manually request the attributes we would like in the certificate. To do this we use a utility called certreq.exe. More information for Certreq is available here: http://technet.microsoft.com/en-us/library/cc736326.aspx.

To use certreq we must first generate a configuration file. FIgure 1 shows a sample configuration file. The key items that must be included is the OCSP Signing OID, and the OCSP No Revocation Check Extension, otherwise known as the id-pkix-ocsp-nocheck extension.

Let us take a look at this configuration file.

· First we have [NewRequest] which is a required section indicating that this is for a new certificate request.

· Then we have the subject in X.500 format. You can also use the ldap format which is derived from X.500. For example: CN=FCOCSP01,DC=Fourthcoffe,DC=Com. Alternatively, you could use just the common name, such as CN=FCOCSP01.

· PrivateKeyArchive=False since we will not be archiving the private key.

· Exportable=True which gives us the option to export the private key if so desired.

· UserProtected=False which disables strong key protection.

· MachineKeySet =True which is used to indicte that the resulting certificate will be stored in the machine store.

· ProviderName=”Microsoft Enhanced Cryptographic Provider v1.0” specifies the Cryptographic Service Provider (CSP) that will be used.

· UseExistingKey Set=False indicates that this request is for a new certificate, with a new key pair.

· RequestType=CMC tells certreq to generate the request in CMC format.

· Then we specify the new section [EnhancedKeyUsageExtension] which indicates what extensions should be placed in the EKU Extension in the certificate. Under that extension we specify that this certificate can be used for OCSP Signing by specifying the OCSP Signing OID (OID=”1.3.6.1.5.5.7.3.9).

· We then start a new section called [Extensions] and specify that the id-pkix-ocsp-nocheck extension should be included in the certificate.

Below are the steps for generating the request and installing the signing certificate:

1. First we use certreq to generate the request file. We specify the configuration file and the output request file. The key pair for this certificate is generated at the same time the request file is created by Certreq.

2. Next, we must submit the request to the CA. Copy the request file over to the Standalone CA. From the Certification Authority MMC, right click on the CA Name, and select All Tasks from the context menu, and then Submit New Request.

3. Browse to the request file, and select Open.

4. The request will then show up in Pending Requests. Right click on the request, and select All Tasks from the context menu, then select Issue.

5. You will now find the requested Certificate under Issued Certificates. Double click on the certificate to view its properties.

6. Verify the certificate. Key things to look for here are the presence of the OCSP No Revocation Checking Extension. And that OCSP Signing is specified in the Enhanced Key Usage (EKU) Extension.

Exporting the Certificate from the CA

1. First select Copy to File from the Details Tab of the Certificate Properties. This will open the Certificate Export Wizard.

2. Click Next at the Welcome Screen.

3. Select DER encoded binary x.509 (.CER), and click Next.

4. Browse to the location where you which to save the resulting certificate, and give the certificate a name, and click on Save.

5. Click Finish at the Completing the Certificate Export Wizard screen.

6. You will be prompted that The export was successful. Click OK.

Installing the OCSP Response Signing Certificate

Copy the resulting certificate to the OCSP Server. Open up a command prompt. Navigate to the location where you saved the certificate file, and run certreq –accept <Certificate Name>, to complete the installation of the certificate.

Configuring Private Key Permissions

The Online Responder Service runs under the Network Service account. By default the Network Service account does not have access to private keys of certificates located in the Local Computer Personal store. To give the Network Service access, perform the following steps:

1. Open up the Certificates MMC targeted for the Local Computer.

2. Right click on the certificate, then select “All Tasks” from the context menu, and then select Manage Private Keys….

3. Click Add on the Permissions dialog box.

4. Type Network Service,and then click Check Names to resolve the name. Then click OK.

5. The Network Service only needs read permissions to the Private Key, so deselect the Allow privilege for Full Control, and verify the Allow privilege is granted for Read, and click OK.

Now that we have installed the OCSP Response Signing certificate, and configured Private Key permissions, we must now configure the Revocation Configuration for the CA, on the OCSP Responder. Open the OCSP Management Console. Follow the following steps to configure the Revocation Configuration:

1. Right click on Revocation Configuration, and select Add Revocation Configuration from the context menu.

2. This will start the Add Revocation Configuration wizard. Click Next, when presented with the Getting started with adding a revocation configuration screen.

3. On the Name the Revocation Configuration screen, give a name to the configuration, and click Next. Note: It is a good idea to name the configuration for the CA server, in case this Responder will be used for multiple CAs.

4. On the Select CA Certificate Location screen, Select a certificate from the Local certificate store, and click Next.

5. On the Choose CA Certificate screen, click Browse.

6. Select the CA certificate, for the CA you are configuring on the OCSP Responder, and click OK.

7. You will then be returned to the Choose CA Certificate screen. The CA that you selected will be displayed. Click Next to continue.

8. You will now need to select a signing certificate, on the Select Signing Certificate screen. Select Manually select a signing certificate, and click Next.

9. You will then be returned to the Revocation Provider screen, click Finish to complete the wizard.

Assigning the Signing Certificate

After completing the Wizard, you will notice under the “Revocation Configuration Status” portion of the “Online Responder Configuration” page that the OCSP Configuration that you just added has an error indicating “Bad Signing certificate on Array controller. No need to panic at this point. This error is generated because we have not assigned the OCSP Response Signing certificate yet.

Now let us go ahead and assign the Signing certificate.

1. In the OCSP MMC, expand Array Configuration, and click on the name of the OCSP Server. Then in the center pane of the console, select the appropriate Revocation Configuration, then right click on that revocation configuration, and elect Assign Signing Certificate from the context menu.

2. You will then be prompted select the Signing certificate. Select the appropriate Signing certificate, and click OK.

At this point you will now see some warnings. If you look under the Revocation Configuration Status for the Revocation Configuration you are configuring, you will notice this error:

Also, on the Online Responder Configuration page you will notice this error:

This is due to the fact that the Revocation Provider has not yet been verified. To verify the Revocation Provider, right click on Array Configuration, and select Refresh Revocation Data.

Once the Revocation Provider has been verified, you should see this under Revocation Configuration Status for the Revocation Configuration you are configuring.

And that OCSP Signing is specified in the Enhanced Key Usage (EKU) Extension.

Verify OCSP Configuration

To verify your ocsp configuration please follow the Verify OCSP Configuration section in Part III of this series.

Conclusion

This concludes Part IV of this Series. I hope you enjoyed the first four parts of the series and find them useful. I plan to cover other PKI topics in the near future.

Implementing an OCSP responder: Part I Introducing OCSP
Implementing an OCSP responder: Part II Preparing Certificate Authorities
Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 
Implementing an OCSP Responder: Part V High Availability
Implementing an OCSP Responder: Part VI Configuring Custom OCSP URIs via Group Policy

- Chris Delay

Hey Rob here again, I thought that I would share with you some of the things that we see where Internet Explorer Kerberos authentication fails.

It is important to understand the default behavior of Internet Explorer and its support for Kerberos authentication so that you don’t start ripping out your hair (can’t speak to what Ned does here). I have listed three very common problems that we typically see when Kerberos authentication is failing with web-based applications.

Scenario 1: Website does not use the standard TCP/IP port of (80/443)

Web Server Configuration:

• Webserver1 has two different IIS sites running on it.
• Website1 runs with a web application pool account assigned to NetworkService.
• Website2 runs with a web application pool account assigned to a domain user account.
• The website2 is configured to listen on Port 8080.
• The following SPN’s have been defined on the website2 application pool account.
  • http/webserver1.contoso.com:8080
  • http/webserver1:8080

In this scenario you can see why a non-standard port is being used since multiple websites have been configured on the same web server. When this happens you need to specify the port to be used when you add the Service Principal Name, otherwise there is going to be a high likely hood that you will get a Kerberos ticket for the wrong web application pool account.

Client Workaround

In this scenario you need to make sure that when Internet Explorer accesses Website2 that it asks for a Service Principal name with the port number defined. However, the default behavior of IE is to not add the port number to the Kerberos ticket request. When this ticket is presented to IIS you will see a KRB_AP_ERR_MODIFIED message back.

You will need to use the below KB article change the default behavior on all IE client versions. For Internet Explorer 6 it will require the QFE Brach of Wininet.dll to be installed before the registry change will actually work.

908209 Internet Explorer 6 cannot use the Kerberos authentication protocol to connect to a Web site that uses a non-standard port in Windows XP and in Windows Server 2003 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;908209

I can tell you that there is not a version of this KB article for IE7 and above, but you do have to make the same registry change for these versions of IE also.

Server Workaround:

There really is not a good workaround to the issue other than to use host headers for one of the websites and adding a DNS HOST record for the host header in your DNS configuration. You will see shortly why we are not recommending a CNAME record in DNS.

Scenario 2: CNAME DNS RR is used for a website.

Web Server Configuration:

• Webserver1 has two different sites running on it.
• Website1 runs with a web application pool account assigned to NetworkService.
• Website2 runs with a web application pool account assigned to a domain user account.
• Website2 is configured to us a host header of app1.contoso.com.
• In DNS a CNAME record for app1.contoso.com was created and pointed to webserver1.contoso.com HOST record.
• The following SPN’s have been defined on the website2 application pool account.
  • http/app1.contoso.com
  • http/app1

In this scenario it appears that this should work just fine. When a user goes to app1.contoso.com the client machine is going to do a DNS lookup, and the DNS server is going to respond with the CNAME record and point to the webserver1.contoso.com HOST record. We can also see that the Service Principal Name configuration is properly configured on the web application pool account for website2.

The default behavior of Internet Explorer is to generate the Kerberos ticket request for the HOST record that is returned from a CNAME record, not the actual CNAME record itself. So IE specifically asks for a Kerberos ticket for http/webserver1.contoso.com which will result in a Kerberos ticket being generated encrypted with the WebServer1 computer’s password. This will in turn generate a KRB_AP_ERR_MODIFIED from IIS back to IE when the user attempts to visit the app1.contoso.com website.

Client Workaround

You will need to use the KB articles below to change the default behavior on all IE versions. For IE 6 it will require the QFE Brach of Wininet.dll to be installed before the registry key change will actually work.

For Internet Explorer 6:

911149 Error message in Internet Explorer when you try to access a Web site that requires Kerberos authentication on a Windows XP-based computer: "HTTP Error 401 - Unauthorized: Access is denied due to invalid credentials" - http://support.microsoft.com/default.aspx?scid=kb;EN-US;911149

For Internet Explorer 7 and above:

938305 Error message when you try to log on to a Web site that requires Kerberos authentication by using Internet Explorer 7: "Access is denied due to invalid credentials" - http://support.microsoft.com/default.aspx?scid=kb;EN-US;938305

Server Workaround:

The only work around is to remove the DNS CNAME RR and replace it with a HOST RR.

Scenario 3: Website works on first access but starts failing 30 minutes later

Web Server Configuration:

• Computer Webserver1 one site on it.
• Website1 runs with a web application pool account assigned to a domain user account.
• The following SPN’s have been defined on the website1 application pool account.
  • http/webserver1.contoso.com

When a user visits the website they use the NETBIOS computer name for the URL to visit. For example: http://webserver1.

In this scenario there does not seem to be much wrong here, except that there is only the FQDN version of the Service Principal Name defined. Yeah, I know all of our documentation around Kerberos always states to add FQDN as well as NETBIOS name versions of the SPN. Believe it or not, we see this all the time where the user did not register both of them, but stick with me here.

The default behavior of Internet Explorer is to add on the computer’s DNS suffix or use DNS suffix search order if defined on the machine to whatever the user types in the URL if it is not a dotted name. If your DNS configuration is correct it will resolve to webserver1.contoso.com. Once IE finds this name it stores the DNS entry in its own DNS cache. Just like most caches it times out - for IE’s cache it is 30 minutes. After 30 minutes IE again has to resolve the name, however the next time it does not try to resolve the name through DNS again, it tries just NetBIOS name resolution (hopefully there is WINS in the environment; otherwise it will just fail). Based on your configuration you could expect one of the following Kerberos errors:

• KRB_AP_ERR_MODIFIED – Expect to get this error if web site name is the same name as your web server’s computer name. That is because it is going to ask for an http/webserver1 SPN and will resolve to HOST/webserver1 which is assigned to the computer account.
• KRB_ERR_S_PRINCIPAL_UNKNOWN – Expect to get this error if the web site name is something like app1.contoso.com. That is because it is going to ask for an http/app1 SPN and will not resolve to any account in the domain.

Client Workaround:

Thankfully there is a fix that can be implemented for Internet Explorer.

899417 You may receive an "Access is denied" error message you use the WWW-Authenticate: Negotiate method of HTTP authentication to connect to a Web server - http://support.microsoft.com/default.aspx?scid=kb;EN-US;899417

I will tell you that there is not a version of this KB article for IE7 and above, but you do have to make the registry key change for these versions of IE also before the functionality is supported.

Server Workaround:

You can register the NetBIOS version of the Service Principal Name to the account, using SETSPN.EXE.

I hope that you found this post interesting. As always it is easier to spot these type of issues by reviewing network trace taken at the client side (where IE is being used) to find the root cause of the issue.

- Rob “I Speak Tampa” Greene