Ask the Directory Services Team

Getting Over Replmon

Ned here again. The AD Replication Monitor utility (Replmon.exe) was introduced with the Windows Server 2000 Support Tools many years ago as a GUI mechanism for performing certain DC admin tasks. With the release of Window Server 2008 Replmon was not included and we stopped making add-on Support Tools. Every few weeks someone asks me ‘where do I download the Windows Server 2008 version of Replmon? Nowhere. It’s done. Buried. Gone. Kaput. If you want it, you must run the old Windows Server 2003 version. Today I will talk about moving on with its supported replacement, Repadmin.exe.

Background

Replmon grew out of the need for a more useful tool than the AD Sites and Services snap-in (dssites.msc). DS Sites offered only a basic view of the topology, and had very limited options for forcing replication or seeing errors in the topology.

On the other hand, Replmon exposed more information and had a deeper view into the AD partitioning structure. It was designed not by the Windows Product Group but rather by an engineer in PSS. Like all Support Tools, it was not truly supported but instead provided ‘AS IS’.

Replmon gave an administrator the ability to quickly force replication, get basic status reports, and see information about the environmental configuration.

It was also written in a graphical format rather than a command-line interface. When Active Directory was first released 10 years ago, most tools were given more attention in their GUI rather than command-line versions. Customers were already overwhelmed with the radical changes of AD over NT and having a GUI was a highly desirable feature for a complex and not well-understood product like Active Directory.

That was then.

Now AD is as ubiquitous as Windows in most customer environments. Windows administrators are much more comfortable with the command-line, and that’s great – because repadmin.exe is now equal or superior to Replmon. Don’t believe me? Let’s compare.

Moving On with Repadmin Syncall

The most common operation with Replmon was to have it ”push” AD replication outbound from a given DC where someone had made a change and wanted it to propagate quickly to all partners. I put ”push” in quotes because AD replication is always pull-based; there is no such thing as push. What Replmon was actually doing was contacting the partner DCs and telling them to replicate inbound immediately. To do this you would:

1. Start Replmon.

2. Type in (or search for) the DC.

3. Right click the DC or partition and choose Synchronize to force replication.

4. Select if you wanted to pull or push, cross AD sites or not, and disable transitivity or not.

Then you waited for it to finish. There were no immediate results to view, and you weren’t always going to see useful messages when something was shown at all. The only progress was a small status bar:

And you might also see:

If there was a problem you would get an error, but it could be misleading. For example, here is the error you get when forcing replication of the Domain partition and one of the DCs is offline for maintenance:

Now contrast this with the Repadmin.exe steps for the same server, doing a push replication of all partitions:

1. Run:

Repadmin /syncall DC_name /APed

2. There is no step 2, we’re done. :-)

By running a repadmin /syncall with the /A(ll partitions) P(ush) e(nterprise, cross sites) d(istinguished names) parameters, you have duplicated exactly what Replmon is doing. Except that you did it in one step, not many. And with the benefit of seeing immediate results on how the operations are proceeding. If I am running it on the DC itself, I don’t even have to specify the server name.

What about the situation I showed earlier where one of the DCs was offline for maintenance? In this case I am going to have Repadmin synchronize just the Domain partition, pushing across site boundaries:

Repadmin /syncall /Pe dc_name naming_context

With Repadmin we get a much more specific error:

Those are legitimate errors that are documented and can be researched.

Status Checking

Replmon had the option to generate a status report text file. It could tell you which servers were configured to replicate with each other, if they had any errors, and so on. It was pretty useful actually, and one of the main reasons people liked the tool.

Repadmin.exe offers similar functionality within a few of its command line options. For example, we can get a summary report:

Repadmin /replsummary *

Several DCs have been taken offline. Repadmin shows the correct error of 58 – that the other DCs are not available and cannot tell you their status.

You can also use more verbose commands with Repadmin to see details about which DCs are or are not replicating:

Repadmin /showrepl *

If you want to generate a ‘repadmin status report’ that generates a bunch of useful status information, give this simple batch file a try:

Copy and paste into notepad, save as a CMD file and run it with a server name, a partial server name with wildcards, or an asterisk. It supports whatever Repadmin supports.

So to get data from one server, like with Replmon:

Replreport.cmd server1

Or to get data from all DC’s (which Replmon cannot do):

Replreport.cmd *

Or to get data from all servers that have names starting with “SANFRAN“:

Replreport.cmd sanfran*

It will output to a text file called replreport.txt. Anything Repadmin can do, you can do in this batch file.

More More More

Repadmin can do even more for monitoring. Such as:

Tell you the last time your DCs were backed up, by reading the DSASignature attribute from all servers:

Repadmin /showbackup *

Or output all replication summary information from all DCs to a CSV format that you can open in a spreadsheet or database. Here I’ve brought my DCs back online and replicated any pending changes. Then I get a replication report:

Repadmin /showrepl * /csv

Or you can see what your replication backlog is currently in the queue, like here:

Repadmin /queue *

Or you can see which changes have not yet replicated from a server, as well as what changes have replicated since the last time the command was run, with /showchanges:

repadmin /showchanges destination_DC source_DSA_GUID domain_DN

(69) add CN=Ned Pyle,CN=Users,DC=adatum,DC=com
1> parentGUID: a90a9633-2682-4896-be86-21220cf24f0c
1> objectGUID: e8f0e0a2-69aa-4e4e-9f74-3db79ad6f3b7
4> objectClass: top; person; organizationalPerson; user
1> sn: Pyle
1> givenName: Ned
1> instanceType: 0x4 = ( WRITE )
1> whenCreated: 6/21/2009 9:05:32 AM Pacific Daylight Time
1> displayName: Ned Pyle
1> nTSecurityDescriptor: O:DAG:DAD:AI
1> name: Ned Pyle
1> userAccountControl: 0x10200 = ( NORMAL_ACCOUNT | DONT_EXPIRE_PASSWD )
1> codePage: 0
1> countryCode: 0
1> pwdLastSet: 6/21/2009 9:05:32 AM Pacific Daylight Time
1> primaryGroupID: 513 = ( GROUP_RID_USERS )
1> objectSid: S-1-5-21-3776065869-1984782319-1196103478-1107
1> accountExpires: (never)
1> sAMAccountName: nedpyle
1> sAMAccountType: 805306368 = ( NORMAL_USER_ACCOUNT )
1> userPrincipalName: nedpyle@adatum.com
1> objectCategory: <GUID=4ed8da23575bed48b12cd36061257c14>;CN=Person,CN=Schema,CN=Configuration,DC=adatum,DC=com

Neat right? That’s a user I created while the other DC was offline, in the list of pending changes. I snipped out another long list of changes that were also pending. Pretty useful to see if a DC that has not been replicating for a while is worth spending time trying to fix or is better off demoting.

Other Repadmin capabilities

Repadmin has plenty of other secrets you can use for monitoring, administering, and troubleshooting – most of which Replmon cannot do:

• Replicate a single specific object
• View and modify RODC password policies as well as trigger password caching
• Create, modify, and delete replication topology
• Remove lingering objects
• Manipulate Global Catalog partitions
• Set replication registry values
• Export data to Excel-ready text
• Way more cool stuff…

Need to see all the help?

Basic help - Repadmin /?

Help on selecting DCs - Repadmin /listhelp

Advanced command help - Repadmin /experthelp

Help and examples for every parameter- Repadmin /?:Your specific parameter here

Wrapping it up

Repadmin may not be as pretty as Replmon or Dssites.msc, but it is far more powerful than both of those utilities combined. Being an AD administrator brings a lot of responsibility – you are accountable for identity management, authentication, authorization, and general network availability for your entire company. You owe it to yourself to learn and use AD’s most powerful tools, even if they don’t support a mouse.

Update July 2 2009: One of our readers also points out that we have an excellent whitepaper on using Repadmin to troubleshoot problems.

Grab it here: http://www.microsoft.com/downloads/details.aspx?familyid=c6054092-ee1e-4b57-b175-5aabde591c5f&displaylang=en

(Nice one, Mike!)

- Ned ‘Cursor’ Pyle

Implementing an OCSP responder: Part IV - Configuring OCSP for use with Standalone CAs

Chris here again. In part I of this series we covered the basics of how OCSP works. We also covered the underlying reasons for deploying an OCSP Responder. In Part II we covered configuring the Certificate Authorities for whom which the OCSP Responder will check revocation status for on behalf of the clients. In Part III we covered configuring and OCSP Responder to support an Enterprise CAs. You may use Standalone CAs in your environment. In this blog post, I will be covering deploying a Revocation Configuration to support a Standalone CA.

Enterprise CAs are very tightly integrated with Active Directory. As such the certificates for the Root CA and for intermediate CAs are published to Active Directory. These certificates are automatically placed in the appropriate certificate stores on the clients. If you publish the Root CA certificate that the issuing CA chains up to; in Active Directory the clients will have that Root CA certificate published to the Trusted Root Certification Authorities container in the user and machine store. If you have not, or do not plan to deploy the Root CA certificate through Active Directory and Group Policy you will need to manually publish the Root Certificates in the Trusted Root Certification Authority store.

Installing OCSP Responder Role

The first step is to install the OCSP Responder Role.

To install the OCSP Responder: Open a command prompt and type: servermanagercmd.exe –install ADCS-Online-Cert

Requesting and Installing the OCSP Responder Signing Certificate

The next step is to request the OCSP Response Signing Certificate from the Standalone CA. Since a Standalone CA does not have certificate templates we must manually request the attributes we would like in the certificate. To do this we use a utility called certreq.exe. More information for Certreq is available here: http://technet.microsoft.com/en-us/library/cc736326.aspx.

To use certreq we must first generate a configuration file. FIgure 1 shows a sample configuration file. The key items that must be included is the OCSP Signing OID, and the OCSP No Revocation Check Extension, otherwise known as the id-pkix-ocsp-nocheck extension.

Let us take a look at this configuration file.

· First with have [NewRequest] which is a required section indicating that this is for a new certificate request.

· Then we have the subject in X.500 format. You can also use the ldap format which is derived from X.500. For example: CN=FCOCSP01,DC=Fourthcoffe,DC=Com. Alternatively, you could use just the common name, such as CN=FCOCSP01.

· PrivateKeyArchive=False since we will not be archiving the private key.

· Exportable=True which gives us the option to export the private key if so desired.

· UserProtected=False which disables strong key protection.

· MachineKeySet =True which is used to indicte that the resulting certificate will be stored in the machine store.

· ProviderName=”Microsoft Enhanced Cryptographic Provider v1.0” specifies the Cryptographic Service Provider (CSP) that will be used.

· UseExistingKey Set=False indicates that this request is for a new certificate, with a new key pair.

· RequestType=CMC tells certreq to generate the request in CMC format.

· Then we specify the new section [EnhancedKeyUsageExtension] which indicates what extensions should be placed in the EKU Extension in the certificate. Under that extension we specify that this certificate can be used OCSP Signing by specifying the OCSP Signing OID (OID=”1.3.6.1.5.5.7.3.9).

· We then start a new section called [Extensions] and specify that the id-pkix-ocsp-nocheck extension should be included in the certificate.

Below are the steps for generating the request and installing the signing certificate:

1. First we use certreq to generate the request file. We specify the configuration file and the output request file. The key pair for this certificate is generated at the same time the request file is created by Certreq.

2. Next, we must submit the request to the CA. Copy the request file over to the Standalone CA. From the Certification Authority MMC, right click on the CA Name, and select All Tasks from the context menu, and then Submit New Request.

3. Browse to the request file, and select Open.

4. The request will then show up in Pending Requests. Right click on the request, and select All Tasks from the context menu, then select Issue.

5. You will now find the requested Certificate under Issued Certificates. Double click on the certificate to view its properties.

6. Verify the certificate. Key things to look for here are the presence of the OCSP No Revocation Checking Extension. And that OCSP Signing is specified in the Enhanced Key Usage (EKU) Extension.

Exporting the Certificate from the CA

1. First select Copy to File from the Details Tab of the Certificate Properties. This will open the Certificate Export Wizard.

2. Click Next at the Welcome Screen.

3. Select DER encoded binary x.509 (.CER), and click Next.

4. Browse to the location where you which to save the resulting certificate, and give the certificate a name, and click on Save.

5. Click Finish at the Completing the Certificate Export Wizard screen.

6. You will be prompted that The export was successful. Click OK.

Installing the OCSP Response Signing Certificate

Copy the resulting certificate to the OCSP Server. Open up a command prompt. Navigate to the location where you saved the certificate file, and run certreq –accept <Certificate Name>, to complete the installation of the certificate.

The Online Responder Service runs under the Network Service account. By default the Network Service account does not have access to private keys of certificates located in the Local Computer Personal store. To give the Network Service access, perform the following steps:

1. Open up the Certificates MMC targeted for the Local Computer.

2. Right click on the certificate, then select “All Tasks” from the context menu, and then select Manage Private Keys….

3. Click Add on the Permissions dialog box.

4. Type Network Service,and then click Check Names to resolve the name. Then click OK.

5. The Network Service only needs read permissions to the Private Key, so deselect the Allow privilege for Full Control, and verify the Allow privilege is granted for Read, and click OK.

Configuring Private Key Permissions

Now that we have installed the OCSP Response Signing certificate, and configured Private Key permissions, we must now configure the Revocation Configuration for the CA, on the OCSP Responder. Open the OCSP Management Console. Follow the following steps to configure the Revocation Configuration:

1. Right click on Revocation Configuration, and select Add Revocation Configuration from the context menu.

2. This will start the Add Revocation Configuration wizard. Click Next, when presented with the Getting started with adding a revocation configuration screen.

3. On the Name the Revocation Configuration screen, give a name to the configuration, and click Next. Note: It is a good idea to name the configuration for the CA server, in case this Responder will be used for multiple CAs.

4. On the Select CA Certificate Location screen, Select a certificate from the Local certificate store, and click Next.

5. On the Choose CA Certificate screen, click Browse.

6. Select the CA certificate, for the CA you are configuring on the OCSP Responder, and click OK.

7. You will then be returned to the Choose CA Certificate screen. The CA that you selected will be displayed. Click Next to continue.

8. You will now need to select a signing certificate, on the Select Signing Certificate screen. Select Manually select a signing certificate, and click Next.

9. You will then be returned to the Revocation Provider screen, click Finish to complete the wizard.

Assigning the Signing Certificate

After completing the Wizard, you will notice under the “Revocation Configuration Status” portion of the “Online Responder Configuration” page that the OCSP Configuration that you just added has an error indicating “Bad Signing certificate on Array controller. No need to panic at this point. This error is generated because we have not assigned the OCSP Response Signing certificate yet.

Now let us go ahead and assign the Signing certificate.

1. In the OCSP MMC, expand Array Configuration, and click on the name of the OCSP Server. Then in the center pane of the console, select the appropriate Revocation Configuration, then right click on that revocation configuration, and elect Assign Signing Certificate from the context menu.

2. You will then be prompted select the Signing certificate. Select the appropriate Signing certificate, and click OK.

At this point you will now see some warnings. If you look under the Revocation Configuration Status for the Revocation Configuration you are configuring, you will notice this error:

Also, on the Online Responder Configuration page you will notice this error:

This is due to the fact that the Revocation Provider has not yet been verified. To verify the Revocation Provider, right click on Array Configuration, and select Refresh Revocation Data.

Once the Revocation Provider has been verified, you should see this under Revocation Configuration Status for the Revocation Configuration you are configuring.

And that OCSP Signing is specified in the Enhanced Key Usage (EKU) Extension.

Verify OCSP Configuration

To verify your ocsp configuration please follow the Verify OCSP Configuration section in Part III of this series.

Conclusion

This concludes Part IV of this Series. I hope you enjoyed the first four parts of the series and find them useful. I plan to cover other PKI topics in the near future.

Implementing an OCSP responder: Part I Introducing OCSP
Implementing an OCSP responder: Part II Preparing Certificate Authorities
Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 

- Chris Delay

New Directory Services KB Articles/Blogs 6/21-6/27

KB Articles

Blogs

· Internet Explorer behaviors with Kerberos Authentication

· Recovering from Unsupported One-Way Replication in DFSR Windows Server 2003 R2 and Windows Server 2008.

· Implementing an OCSP responder Part 1 (Part 2, Part 3)

· RSAT and ADUC for Vista - Update to add tabs for Terminal Services Profile, Environment, Sessions, and Remote Control

· Debunking the Vista Remote Differential Compression Myth

· Deploying DFSR Clustering in Windows Server 2008 R2

· How many DFS-N namespace servers do you need?

· Active Directory Enhancements - Windows Server 2008 R2 Demo Screencast 4 of 4

· Detection logic for PowerShell installation

· Discovery options with R2 AD Cmdlets

· Active Directory Port Requirements

· Use Active Directory Powershell to manage Windows 2003/2008 DCs

Deploying DFSR Clustering in Windows Server 2008 R2

Ned here again. The DFSR development team has posted a new series for anyone wanting top learn more about the new DFSR clustering capabilities in Windows Server 2008 R2. It's pretty indepth and walks you through the entire process.

1. Deploying DFS Replication on a Windows Failover Cluster – Part I: Explains how to create a new Windows Server 2008 R2 failover cluster.
2. Deploying DFS Replication on a Windows Failover Cluster – Part II: Explains how to configure DFS Replication service for high availability on the failover cluster.
3. Deploying DFS Replication on a Windows Failover Cluster – Part III: Explains how to add the failover cluster as a member server in a DFS replication group.

Nice stuff, Mahesh!

- Ned Pyle

Implementing an OCSP responder: Part III - Configuring OCSP for use with Enterprise CAs

Chris here again. As promised I will be covering configuring an OCSP Responder to support Enterprise CA. I will also be covering validating your OCSP Configuration.

Installing OCSP Responder Role

The first step is to install the OCSP Responder Role.

To install the OCSP Responder: Open a command prompt and type: servermanagercmd.exe –install ADCS-Online-Cert.

Configuring the OCSP Responder

First we will add a Revocation Configuration to the OCSP Responder.

Right click on the Revocation Configuration and select Add Revocation Configuration from the context menu.

The Add Revocation Configuration wizard opens. Click Next to continue.

Give a Friendly Name to the Revocation Configuration, and click Next. It is a good idea to include the name of the CA for which you are setting up this Revocation Configuration, especially if this OCSP Responder will handle requests for multiple CAs.

On the Select CA Certificate page, you will need to select a CA certificate. This is where you determine the CA for which you will be providing revocation information.

Select a certificate for an Existing enterprise CA, and click Next

Select Browse CA certificates published in Active Directory, and click Browse.

Select the appropriate CA, and click OK

Next you will need to select a certificate that will be used for signing OCSP responses. For a particular Revocation Configuration, the OCSP Signing certificate must be issued by the CA for which the OCSP Responder will answer revocation status requests.

Select Automatically select a signing certificate. If you wish to automatically enroll for the OCSP Response Signing Certificate, make user the Auto-Enroll for an OCSP signing certificate is checked. Select the certificate template that you configured for use with the OCSP Responder, then click Next.

On the Revocation Provider page, you can click Provider to select revocation providers. The Windows Server 2008 OCSP Responder can only use CRLs for revocation information. If you have the CDP Extension available in the signing certificate, the Revocation Providers will be populated from the information in the CDP Extension from the OCSP Response Signing Certificate.

You can add the repository locations for your CRLs and Delta CRLs if appropriate. By default these will be populated from information included in the CDP extension of the Signing certificate. After you have reviewed the configuration or made any changes, click OK.

That completes the initial Configuration of the OCSP Responder. If you would like to modify the configuration of the OCSP Responder, you can right click on the Revocation Configuration and select Properties from the context menu.

The Local CRL tab allows you to configure a Local CRL. You can add revocation information for certificates which you wish to consider revoked. It is recommended that you do not use this option, as it adds unnecessary complexity to the revocation configuration.

The Revocation Provider tab allows you to modify the location of the CRLs and Delta CRLs that will be used for providing revocation information.

Signing Tab

In the signing tab you can:

• Modify the hash algorithm used to sign responses.
• Do not prompt for credentials for cryptographic operations. This setting may need to be disabled if you are using an HSM to protect the private key of the OCSP Signing certificate. Disabling this setting allows you to be prompted for the password that is associated operator card on the HSM.
• Use renewed certificates for signing certificates. This option is enabled by default, when you use the OCSP Responder with an Enterprise CA and automatically renew certificates. If you use OCSP Responder with a standalone CA, the OCSP responder will use renewed signing certificates even if this setting is not enabled.
• Enable NONCE extension support allows the user to attach the NONCE sent in the request with the OCSP response. If this setting is used, you will not be able to utilize cached responses.
• Use any valid OCSP signing certificate. Not recommended if the OCSP Responder is supporting Vista clients since they do not support this option. This allows the OCSP responder to use any certificate that the OCSP Signing configured in the Extended Key Usage extension of the certificate. Vista clients will only accept OCSP response that is signed from by the same CA for which the OCSP Responder is providing revocation information.
• All responses will included the following Online Responder identifies: This setting etermines whether a Key Hash or Subject will be included in the response. RFC 2560 specifies the structure of the response. In section 4.2.1 of the RFC it is specified that the Responder ID field can either be populated with a Name or Key hash. This setting determines which is included in the response. The Key hash is a hash of the OCSP Responder’s public key. The Name is the distinguished name of the subject of the OCSP signing certificate.

Verify OCSP Configuration

After configuring the OCSP Responder, you will want to verify that the OCSP responder is functioning properly. The easiest way to verify that the OCSP is functioning is to use the Certutil URL Retrieval tool.

First request a certificate from the CA. Place a copy of that cert on the file system, and run the following command: certutil –URL <Certificate Name>. This will open the URL Retrieval Tool

Select OCSP, and click on the Retrieve button.

If the certificate is valid you will get the following response.

If the certificate is revoked, you will get the following response.

And if it fails, the status will be listed as Failed.

You can also use the PKIView tool to verify the configurations of the OCSP Responder.

Conclusion

This concludes configuring an OCSP Responder to support an Enterprise CA. If you follow the steps listed here you now have your OCSP configured to support your Windows Server 2003 or Windows Server 2008 CA. In the next part of this series, I will be configuring an OCSP Responder to support Standalone CA.

Implementing an OCSP responder: Part I Introducing OCSP
Implementing an OCSP responder: Part II Preparing Certificate Authorities
Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 

- Chris Delay

Debunking the Vista Remote Differential Compression Myth

Ned here again. Have you ever visited Snopes.com? It’s a terrific urban legend reference where they research folklore. Snopes is the place you go to find out if eating Thanksgiving turkey makes you sleepy (it doesn’t), if Coca Cola can dissolve a tooth overnight (it can’t), or if a man really did live in a Paris airport for 8 years (he did!).

Today I’m going to talk about another urban legend – that removing the Remote Differential Compression feature from Windows Vista will make your file copying faster over the network.

Background on RDC

Remote Differential Compression (RDC) is a Microsoft algorithm that was originally created for DFSR five years ago. RDC divides a file’s data into chunks by using signatures. When a file exists on two computers and the file is modified, only the differing chunks need to be sent to the other computer.

An application needs to be specifically written to support RDC. Windows Vista and Windows 7 include MSRDC.DLL to allow apps like Windows Live Messenger to use that functionality.

The feature can be turned on and off within the Control Panel “Program and Features” applet.

When turned on, the MSRDC.DLL will exist in the %SYSTEMROOT%\System32 directory. When it’s turned off, this DLL is removed.

The Myth

Unfortunately, the Internet is full of people telling you that RDC will somehow make your network communication slower. I have no idea how this got started, but this nonsense has been reprinted on thousands of websites by people unfamiliar with the Scientific Method. Folks have actually convinced themselves that turning this feature on or off has some affect on file transfer speeds. While there are a variety of things you can do to speed up Vista file copy performance, this isn’t one of them.

The Method

So after hearing this baloney for the umpteenth time, I set out to debunk it once and for all:

• I setup a Windows Server 2008 SP2 file server and a Windows Vista SP2 client as virtual guests inside a Hyper-V host.
• Each guest got 1GB of RAM and a virtual gigabit NIC.
• I dropped a 3GB ISO file onto a share on the file server.
• I used robocopy.exe (which automatically reports bytes per second and megabytes per minute) to copy that ISO file five times with MSRDC.DLL installed, then five more times with MSRDC.DLL removed.
• In between every copy, I rebooted both the client and server to ensure that there was no caching of the file that might artificially improve my results.

The Results

Here is what I found after the ten total passes, with and without RDC installed:

Hmmm… What if I sort my data points highest to lowest?

Interesting. Let’s look at the actual averages:

With MSRDC.DLL installed: 1209.6838 MB/min

With MSRDC.DLL removed: 1208.1666 MB/min

Wait – so removing RDC actually made it slower? Not really – the variance there is well within any respectable margin of error. These results mean that the two sets of copies were 99.87% identical. Removing RDC did nothing at all. There are going to be various performance differences when copying a file, depending on what else happens on the network, what the computers are doing, and I think when people claim RDC removal made things ‘faster’, it’s because they are not testing repeatedly over time to see variance.

Maybe you want more proof? Alright, let’s go to the debugger.

• I installed Notmyfault on my Vista client and configured the computer for a complete memory dump
• I started yet another file copy, then I listed out the process and service information with TASKLIST.EXE and TASKLIST.EXE /SVC
• After a few seconds, I intentionally crash dumped (the so-called “Blue Screen of Death”) the Vista computer. I then cracked open the MEMORY.DMP file in the Windows debugger

First, I listed loaded modules - all libraries and drivers loaded in the kernel memory space:

Note how MSRDC.DLL is not loaded in memory in the Kernel space. It’s still possible that a given process or service might have it loaded though, so then I listed all processes to see which ones would be interesting and likely to be involved in file copies. The TASKLIST output comes in handy here to see which PID is which hexadecimal CID value. In my case though I dumped them all just for exploratory purposes.

I know that the Workstation Service is responsible for SMB file copying, and the robocopy process is definitely doing work, so I examined those.

Note how neither process has MSRDC.DLL loaded either. It’s simply not being used, and a module that is not being used cannot possibly affect anyone. Remember, an application has to be coded to use RDC. Nothing in the Kernel, in Robocopy, or in the Workstation service uses RDC at all in Vista or Win7.

Still don’t believe me? Here is the Microsoft Remote File Systems development team stating it as well.

Changes that can truly improve file copy performance

By now you want me to get to the helpful part. Here’s a short list of some things that can improve your file copy network performance on Windows Vista and Windows Server 2008:

• Install Service Pack 2
• Install the latest NIC drivers from your vendor.
• Try disabling Receive Side Scaling, Chimney Offload, and NetDMA support, then testing like I did above to see if the results are measurably different after many copies of the same file. Note that this just disables the Windows implementation of those components – your vendor may also support them through their NIC configuration and it will need to be turned off there as well. While these components are intended to help performance, mileage can vary based on how good your hardware and vendor drivers are.
• Use robocopy.exe rather than Explorer – the price of the friendly shell showing progress and browsing folders is slightly slower performance.

Wrapup

It’s amazing that a component that was designed to speed up network file performance can somehow be vilified as a cause of bad performance; especially when it’s not even being used. I welcome people following my steps and telling me what you find out.

Don’t believe everything you read on the Internet. Unless I wrote it. :-)

- Ned ‘Rick Rolled’ Pyle

Implementing an OCSP responder: Part II - Preparing Certificate Authorities

Chris here again. In Part I we covered some of the basics and background information on the reason for the OCSP Responder and a basic understanding of how the OCSP Responder functions. So now we look towards implementing the OCSP Responder. However, before we move forward with the Install of the OCSP Responder we must first configure the CA to support OCSP for revocation status checking.

As discussed in the first part of this series, the OCSP Responder provides revocation information to clients or application requesting revocation status for a specific certificate. In order for this to be accomplished there are certain prerequisites that need to be in place.

Some of the prerequisites are different depending on which version of the CA you are using, and whether you are using a Standalone or Enterprise CA.

Configuring AIA Extension to support OCSP

To advertise that revocation status information for a particular CA can be obtained via OCSP, the CA must include a pointer to the OCSP Responder in the certificate. This is done by adding an OCSP URI to the AIA extension of the certificate.

Although this is mentioned as a prerequisite, you may want to do this after the OCSP Responder is configured. The reason being is that if you issue certificates before the Responder is available you will create unnecessary traffic to the soon to be OCSP location.

1. Open the Certification Authority Snap-in on the CA, as an Enterprise Administrator.

2. Right click on the CA name, and select Properties

3. Click on the Extension Tab. From the Select Extension drop down Box, select Authority Information Access (AIA).

4. Then click on the Add… button to add the OCSP location

5. Type the location for the OCSP responder. This will typically be:

http://<fqdn of the ocsp responder>/ocsp

6. Then click OK.

7. Check the Checkbox for Include in the online certificate status protocol (OCSP) extension.

8. And click OK, to close the CA Properties.

Preparing Windows Server 2003 Standalone CA for use with OCSP Responder

OCSP Signing Certificates

In order to be able to deploy the OCSP Signing Certificate used by the OCSP Responder, there are some configuration changes that need to be made on a Windows Server 2003 CA.

A signing certificate includes the id-pkix-ocsp-nocheck extension. This extension informs the OCSP client that the OCSP signing certificate should not be checked for revocation during the lifetime of the certificate. The OCSP Signing certificate should therefore have a short lifetime. By default, a Windows Server 2003 CA will ignore the id-pkix-ocsp-nocheck extension in a certificate request and will not include that extension in the issued certificate. To change this behavior, you must allow custom extensions to be used in certificate requests.

To enable support for custom extensions, run the following command on the CA:

The extension object ID (OID) for the id-pkix-ocsp-nocheck extension is1.3.6.1.5.5.7.48.1.5. The above command instructs the CA to include that extension in the issued certificate if it is found in the request.

Preparing Windows Server 2003 Enterprise CA for use with OCSP Responder

If you plan on using a Windows Server 2003 Enterprise CA to issue the OCSP Signing Certificate you will need to follow the instructions outlined in the previous section for enabling the use of custom extensions.

If you plan on using a certificate template on the Windows Server 2003 Enterprise CA, you must have at least 1 Windows Server 2008 Enterprise CA in the environment. The reason is you will be duplicating the Version 3 OCSP Signing Template on the Windows Server 2008 CA for use with the Windows Server 2003 CA. Both the Windows Server 2003 and Windows Server 2008 CA, must be running Enterprise Edition. This is due to the fact that only Version 1 templates are supported in the Standard Editions of the Server OS.

Duplicating the OCSP Signing Template

1. Logon to a Windows Server 2008 Enterprise CA, with an account that is a member of the Enterprise Admins group.

2. Open up the Certificate Template management console (certtmpl.msc).

3. Right click on OCSP Response Signing Template, and select Duplicate Template from the context menu, as illustrated below:

4. From the Duplicate Template dialog box, select Windows Server 2003 Server, Enterprise Edition, and click OK. Selecting Windows Server 2003 Server, Enterprise Edition, creates a Version 2 Template instead of a Version 3 Template.

 
5. Give a Name to the Duplicated Template, and click OK.

6. Log on to the Windows Server 2003 CA, and open the Certificate Authority Snapin (Certsrv.msc), and right click on Certificate Templates, and select New, then Certificate Template to Issue from the context menu.

7. Select the Duplicated Template, and click on OK.

Preparing Windows Server 2008 Standalone CA for use with OCSP Responder

In the previous section on preparing the Windows Server 2003 Standalone CA, we had to enable the CA to accept custom extensions sent in the request. This was to allow us to request a certificate with the id-pkix-ocsp-nocheck extension. Windows Server 2008 natively supports the id-pkix-ocsp-nocheck extension, so there is no need to allow custom extensions. On the Windows Server 2008 Enterprise CA there is no action necessary to support the id-pkix-ocsp-nocheck extension. However, on the Windows Server 2008 Standalone CA, we need to run the following command to add support for the id-pkix-ocsp-nocheck extension:

Preparing Windows Server 2008 Enterprise CA for use with OCSP Responder

The only preparation required for the Windows Server 2008 Enterprise CA, is to give permissions to the templates to the OCSP Servers, and to make the template available for issuance.

1. Open the Certificate Template Management console (certtmpl.msc)

2. Locate the OCSP Certificate Template, Right-click, and select Properties

3. On the Security Tab, add the hostname of the soon to be OCSP Server, and give the server Read and Enroll permissions to the template. Note: A more scalable solution, as seen in the illustration below, is to create a security group, assign permissions to the security group, and add any OCSP servers to the Security Group.

4. Go back to the Certification Authority management console, Right-click on the Certificates Templates node, and from the context menu, select New and then "Certificate Template to issue.

5. Select the OCSP Response Signing Template, and select OK.

Conclusion

You should now have your Certificate Authorities configured to support the OCSP Responder as a source of revocation status. In the next part of this series I will cover installing the Configuring the OCSP Responder to support Enterprise CAs.

Implementing an OCSP responder: Part I Introducing OCSP
Implementing an OCSP responder: Part II Preparing Certificate Authorities
Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 

- Chris Delay

RSAT and ADUC for Vista - Update to add tabs for Terminal Services Profile, Environment, Sessions, and Remote Control

Ned here. After much strife, here is the hotfix to get RSAT AD Users and Computers to include tabs for:

• Terminal Services Profile 
• Environment 
• Sessions 
• Remote Control 

960890 Some tabs are not available in the properties of a user account in the Active Directory Users and Computers MMC snap-in after you install Remote Server Administration Tools (RSAT) on a computer that is running Windows Vista
http://support.microsoft.com/default.aspx?scid=kb;EN-US;960890

If you have been hacking away at this previously to make it work, make sure to unregister and remove your server DLL's before installing this update.

 Linda Taylor rules.

- Ned Pyle

New Directory Services KB Articles/Blog Posts 6/14-6/20

KB Articles

Blog Posts

Implementing an OCSP responder: Part I - Introducing OCSP

Chris here again. For those Security Architects and PKI implementers, you may have known that since Windows Server 2008 we have an Online Certificate Status Protocol (OCSP) responder, and since Windows Vista we have an OCSP client that is integrated with the operating system. I wanted to cover the in and outs of the OCSP responder, and walk through the installation.

So, you may be asking the question “OCSP what?” First a little background. One of the capabilities of a PKI and in particular a Certificate Authority, aside from issuing certificates, is to publish revocation information.

For example, let’s say you issue a User certificate to a user for authentication. When the user leaves the company you will most likely want to make sure no one can use that certificate for authentication so you log onto the Certificate Authority and revoke that certificate. Each CA has a period specified when it publishes what are called Certificate Revocation Lists or CRLs for short. When the next CRL is published it will contain the serial number of the certificate, the date and time it was revoked, and the reason that the certificate was revoked. Depending on the configuration the CA it will publish the CRL to a repository such as an LDAP server or a web server. In some instances a task or job must be created to copy the CRL to a repository.

Aside from CRLs, there are also delta CRLs. Delta CRLs simply contains the revocation information for certificates that have been revoked since the last Base CRL was published. In order to determine revocation status an application would examine the last base CRL, and the latest delta CRL. The reason for publishing delta CRLs is to provide revocation information that has more current data. Also, it can reduce bandwidth since if the base CRL is already cached on the client, just the delta CRL can be downloaded. More on this later.

In order for applications to determine if a certificate has been revoked, the application examines the CRL Distribution Point (CDP) extension in the certificate. This extension will have information on locations where the CRL can be obtained. These locations are normally either HTTP or LDAP locations.

The application then can go to those locations to download the CRL. There are, however, some potential issues with this scenario. CRLs over time can get rather large depending on the number of certificates issued and revoked. If CRLs grow to a large size, and many clients have to download CRLs, this can have a negative impact on network performance. More importantly, by default Windows clients will timeout after 15 second while trying to download a CRL. Additionally, CRLs have information about every currently valid certificate that has been revoked, which is an excessive amount of data given the fact that an application may only need the revocation status for a few certificates. So, aside from downloading the CRL, the application or the OS has to parse the CRL and find a match for the serial number of the certificate that has been revoked.

With the above limitations, which mostly revolve around scalability, it is clear that there are some drawbacks to using CRLs. Hence, the introduction of Online Certificate Status Protocol (OCSP). OCSP reduces the overhead associated with CRLs. There are server/client components to OCSP: The OCSP responder, which is the server component, and the OCSP Client. The OCSP Responder accepts status requests from OCSP Clients. When the OCSP Responder receives the request from the client it then needs to determine the status of the certificate using the serial number presented by the client. First the OCSP Responder determines if it has any cached responses for the same request. If it does, it can then send that response to the client. If there is no cached response, the OCSP Responder then checks to see if it has the CRL issued by the CA cached locally on the OCSP. If it does, it can check the revocation status locally, and send a response to the client stating whether the certificate is valid or revoked. The response is signed by the OCSP Signing Certificate that is selected during installation. If the OCSP does not have the CRL cached locally, the OCSP Responder can retrieve the CRL from the CDP locations listed in the certificate. The OCSP Responder then can parse the CRL to determine the revocation status, and send the appropriate response to the client.

OCSP Components

OCSP Client

The OCSP Client is a component that generates OCSP requests based on information stored in the AIA extension of the certificate it is validating. The Windows OCSP client supports the Lightweight OCSP Profile as specified in RFC 5019.

OCSP Responder

Web Proxy Cache

Web Proxy Cache is the Web service that receives requests, sends and caches responses.

Online Responder Service

The Online Responder Service is the component that is responsible for managing the configuration of the OCSP responder, retrieving revocation information from the Revocation Providers, signing responses, and auditing changes to the configuration of the OCSP responder (if configured to do so).

The Online Responder service runs under the Network Service account. When you create the Revocation Configuration you will assign the Signing Certificate that will be used by the Online Responder Service to digitally sign the responses sent back to a requesting client. If you are utilizing the OCSP in conjunction with an Enterprise CA you can choose to enroll for the signing certificate during the Revocation Configuration setup, and you can also choose to automatically reenroll for signing certificates. This eases management because the Signing Certificates are generally set to be valid for a short period of time.

The reason for the short validity periods is that the OCSP signing certificate contains the id-pkix-ocsp-nocheck extension. This extension tells the client that the certificate is valid for its entire lifetime so the revocation status of the certificate is never checked. The reason why this extension is included is to avoid circular revocation checking. If this extension was not included, the client would contact the OCSP Responder to verify the revocation status for a certificate. The OCSP Responder would then respond with a signed request. And then the client would then have perform revocation checking for the certificate that signed the response, before finishing revocation checking for the original certificate. At this point if there was an OCSP location specified for the signing certificate, you would run into a loop where the OCSP client would ask for the revocation status for the signing certificate from the OCSP and get a signed response. Then the client would again have to validate the revocation status for the signing certificate. This would occur over and over again. Or alternatively, if a CDP location was specified for the signing certificate, you would then need to download the CRL, and verify the signing certificate, in effect making the OCSP pointless, since you would have to download a CRL to validate the OCSP Signing Certificate. We avoid all of this with the inclusion of the id-pkix-ocsp-nocheck extension.

So, since we are not checking revocation status for the OCSP Signing certificate you should have a short validity period for the OCSP Signing Certificate to increase security.

Regardless you will have to the give permissions to the private key of the OCSP Signing Certificate to the Network Service Account since that is the identity under which the service runs. If you are using the OCSP with a Windows Server 2008 Enterprise CA, in the Request Handling tab of a Version 3 Certificate Template there is the option to Add Read permissions to Network Service on the private key. This option is enabled by default on the OCSP Response Signing template.

If you are using a certificate issued from a Windows Server 2008 Standalone CA, a Windows Server 2003 Enterprise CA or a Windows Server 2003 Standalone CA, you will need to manually grant permissions to the private Key of the OCSP Signing Response Certificate to the Network Service account.

1. To manually give the Network Service Account access to the private key, open up the Certificates MMC targeted for the Local Computer.

2. Right click on the certificate, then select All Tasks from the context menu, and then select Manage Private Keys…

3. Click Add on the Permissions dialog box.

4. Type Network Service, and then click Check Names to resolve the name. Then click OK.

5. The Network Service only needs read permissions to the Private Key, so deselect the Allow privilege for Full Control, and verify the Allow privilege is granted for Read, and click OK.

Revocation Configuration

A Revocation Configuration contains PKI components required to respond to an OCSP request. These include items such as the CA Certificate, OCSP Signing Certificate, and information about the Revocation Provider.

You can have multiple Revocation Configurations per OCSP Responder allowing the OCSP Responder to provide revocation information for multiple CAs.

When configuring the Revocation Configuration for the OCSP Responder you will designate the following

• The certificate of the CA for which you are providing revocation status
• The Signing Certificate (If the CA is an Enterprise CA, and you are using a certificate template)
• The Revocation Provider (Limited to Base and Delta CRLs in Windows Server 2008)

Revocation Provider is the component responsible for retrieving revocation information. In Windows Server 2008 the only revocation provider supported is the CRL based Revocation Provider. In other words the Windows Server 2008 OCSP Responder can only retrieve revocation information from published CRLs.

High Availability

OCSP Responders can be configured for high availability by placing the OCSP responders in an Array. The Array itself does not provide fault tolerances, but maintains the configurations of multiple OCSP responders that are part of the Array. The configuration is maintained by the OCSP Responder that is designated as the “Array Controller”.

Once the responders are arranged in an Array you can use Network Load Balancing to provide a highly available configuration.

I will cover the process of creating a highly available OCSP configuration in a future blog article.

Conclusion

I hope you found the information in this posting helpful. I plan on continuing the series on deploying an OCSP Responder. I will be posting the following blog entries soon, stay tuned:

Implementing an OCSP responder: Part I Introducing OCSP
Implementing an OCSP responder: Part II Preparing Certificate Authorities
Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 

- Chris Delay

Recovering from Unsupported One-Way Replication in DFSR Windows Server 2003 R2 and Windows Server 2008.

Warren here again. The purpose of this blog post is to outline the proper steps to move from an unsupported one-way replication deployment when using DFSR from Windows 2003 R2 and Windows 2008 to a supported configuration of two-way replication.

Before we get started, here is some good news. Starting with Windows 2008 R2 we will now have Read Only replicas support in DFSR. Details of this feature can be found here:

https://blogs.technet.com/filecab/archive/2009/01/21/read-only-replicated-folders-on-windows-server-2008-r2.aspx

Recovering from One Way Replication

First let’s define some terminology so the rest of this post will make more sense.

• Upstream server – For this article “Upstream” server will refer to the DFSR server that is able to replicate changes to its partner.
  
• Downstream server – For this article the “Downstream server” will refer to the DFSR server that is not allowed to replicate changes to its partner.

Note: In practice any server in a multi-master replication implementation can be Upstream or Downstream. It all depends on who has the changes “Upstream” and where the change is getting replicated to “Downstream”.

• RG – Replication Group - a set of servers, known as members, that participates in the replication of one or more replicated folders
  
• RF – Replicated Folder - defines a set of files and folders to be kept in sync across multiple servers in a replication group

See these links for more information on RG’s and RF’s

http://msdn.microsoft.com/en-us/library/bb540026(VS.85).aspx

http://technet.microsoft.com/en-us/library/cc759803.aspx

What Not To Do

1. You do not want to simply enable replication from the Downstream server to the Upstream server. That is a BAD idea. The reasons why are listed in the section below “Why One-way Replication is not Supported”. You may end up with very old data in your Replicated Folder and some upset users wondering why they are looking at last year’s report.
   
2. Do not remove and re-add the Downstream server to the Replication Group again. It is a common misconception that doing this will trigger DFSR to perform an Initial Sync and Initial Replication.

What To Do

To successfully recover from a one way replication deployment we must force the downstream server to perform an Initial Synchronization and Initial Replication of the RF so that all queued changes on the Downstream server are discarded.

I will provide two methods for you to consider. These would be the ones I would use based on key considerations:

1. How divergent is the data?
   
2. How slow are the links?

Method 1 will generate some file replication traffic as the downstream member is most likely out of date for at least some of its data. If the data is fairly consistent then I would use this method

Method 2 will generate the least amount of file replication as you will be pre-staging a recent backup of the data taken from the Upstream server on the Downstream. I would use this method if the data is very different or the degree of divergence is unknown but suspected to be high. Also if the links are slow Method 2 is a good choice.

If you use method 2 make sure you read Ned’s blog post on the proper method of pre-staging files. If you pre-stage the files incorrectly you will generate more replication traffic than if you had just used Method 1.

http://blogs.technet.com/askds/archive/2008/02/12/get-out-and-push-getting-the-most-out-of-dfsr-pre-staging.aspx

Whatever you do, always backup the data on your Upstream and Downstream server before making any changes to your one way replication configuration. Backing up your data is a matter of best practice so you should be doing this nightly already. As with any setup of replication, the work should be done off hours or on a weekend when possible to minimize user interruption.

Patching

Make sure your DFSR servers patch levels are up to date per the KB Articles linked below before implementing any other changes. Do not skip this point.

2003 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;958802

2008 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;968429

Important Note: DFSR stores its configuration in AD. When changes are made to the DFSR configuration the update will take place on the DC that the DFSR server is connected to. Those changes are then replicated to all DC’s in the domain. DFSR will pick up the change on its own during its next poll of AD.

In the methods mentioned below all changes to the DFSR configuration will be made on the Upstream server. We will then force AD replication and finally force a poll of AD on the DFSR servers. The steps to do this is listed below and the referenced in both Methods 1 and 2.

Forcing AD Synchronization

a. To find out what DC a DFSR server is connected to, use WMIC.

WMIC /namespace:\\root\microsoftdfs path DfsrReplicationGroupConfig get LastChangeSource

b. To force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain run this command on the DC returned in step A. i.e <dc name> = the DC returned in step A.

repadmin /syncall /d /e /P < dc name> <Naming Context>

c. On the Upstream and Downstream servers run this command. “DFSRDIAG Pollad”.

Tip: If you want to remote the pollad command you can by specifying the target server with the “/mem:” switch.

Method 1 – Disabling the Downstream server’s membership in the RF to force Initial Sync and Initial Replication when the membership is re-enabled

1. Get a full backup of the Replicated Folder(s) from the Upstream server.

2. If there are changes you want to keep on the Downstream server, back them up now.

Note: Any files that are different or unique on the downstream will be moved to the ConflictAndDeleted or pre-existing directories. The data in ConflictAndDeleted will be permanently removed if the quota is reached. The quota by default is 660 MB. Make sure you backup the Downstream if there is any data there you want to keep. See http://technet.microsoft.com/en-us/library/cc782648.aspx

3. On the Upstream sever open the DFSR management snap-in and highlight the Replication Group that has the affected Replicated Folder. Make sure you are on the Memberships Tab. Right click the Downstream server and select Disable.

Figure 1 - Disabling the Downstream servers membership in the Replication Group

You will get prompted for verification that you want to disable the membership of the server. Depending on if the RF is published in a DFS Namespace or not you will get a different set of prompts.

If your RF is not published you will see only the prompt in Figure 2

Select yes.

Figure 2 - Disable Membership prompt when the Replicated Folder is not published in a DFS namespace

If your RF is published in a DFS Namespace you see the prompts in Figure 3 and 4.

Click yes and Ok.

Figure 3 First popup you will get disabling membership on an RF that is published via a DFS namespace

Figure 4 Second popup that you get disabling membership on a RF that is Published via a DFS Namespace

4. On the Upstream server use the DFS Management console to enable the connection from the Downstream server to the Upstream server. This is done on the Connections Tab. If there is no connection from Downstream to Upstream create it at this time.

Figure 5 - Enabling the connection from Downstream to Upstream

5. On the Upstream find out what DC it is connected to -Step A in Forcing AD Synchronization.

6. Force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain. Step B in Forcing AD Synchronization.

7. Force the Upstream and Downstream servers to poll AD. Step C in Forcing AD Synchronization.

Once the downstream server detects the change to its membership the downstream server will then log events 4114 and 4008. Once you confirm these events are logged you can proceed to step 8. Event 4114 informs the admin that the data in the in the RF will be seen as pre-existing. The data will be treated like any other pre-staged data during Initial Sync and Replication when its membership is re-enabled.

Figure 6 Event ID 4008 logged

 

Figure 7 Event ID 4114

8. Enable the membership of the Downstream server in the Replicated Folder using the DFS Management snap-in. This is located on the Memberships Tab.

Figure 8 - Enabling the membership of the Downstream server in the Replication Group.

Depending if your RF is published in a DFS Namespace or not you will a get set of different prompts.

If the RF is not published in DFS you will get prompted with the dialog box in figure 9. Verify the path and click OK.

Figure 9 Enabling Membership on a server where the RF is not published via DFS

If your RF is published in a DFS namespace you will get the popup in figure 10. You will not be able to click OK until you set the share permissions and share name by clicking on the Edit button. Clicking in the Edit button will bring up the popup in figure 11. Set the perms and share names as needed (defaults are fine if suitable). The extra prompts are presented when the RF is published because the member is being added again as a target for the folder in the DFS namespace.

Figure 10 First popup presented when enabling membership on an RF that is published in DFS.

Figure 11 Second popup displayed when enabling membership on an RF that is published in DFS

9. Repeat steps 5-7. After those steps are done the Downstream server will log Event 4102 when Initial Replication begins and 4104 when it is complete. You will also more than likely see 4412 events due to the servers having different versions of some files. If the data is very different on the servers you will see a large amount of these.

Figure 12 Event ID 4102

Figure 13 Event ID 4412 “Conflict Event”

Figure 14 Event 4104

10. Once you see event ID 4104 on the Downstream server for the replicated folder(s), you are finished with setting up two-way replication.

Method 2 – Total Recreation of the RG using pre-seeded data on the Downstream server.

1. On the Upstream server get a full Backup of the replicated folder.

2. Ship the Backup to the Downstream server’s site.

3. On the Upstream server delete the Replication Group.

4. On the Upstream server find out what DC is connected to. – Step A in Forcing AD Synchronization.

5. Force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain. – Step B in Forcing AD Synchronization.

6. On the Upstream and Downstream servers force them to PollAD. – Step C in Forcing AD Synchronization.

7. On both the Upstream and Downstream server s you will log events 4010 and 3006 noting that the RF and RG have been removed from the configuration:

 

Figure 15 Event ID 4010 logged when a RF is removed from a RG

Figure 16 Event ID 3006 Logged when a RG is removed

8. Pre-seed the backup on the downstream server. See Ned’s blog post on how to do this correctly: http://blogs.technet.com/askds/archive/2008/02/12/get-out-and-push-getting-the-most-out-of-dfsr-pre-staging.aspx. In my experience most pre-seeding attempts that fail are due to mismatched NTFS permissions. The entire directory tree must have matching ACLs or the files will be different and file replication will occur. See Ned’s post for options on how to get ACLS to match. (I like icacls.exe but there are more options)

9. On the Upstream server recreate the Replication Group and Replicated Folder(s), specifying that the Upstream server is the Primary server for the content. Make sure to set your staging areas to the largest size possible up to the size of the RF if possible.

10. Repeat Steps 4-6. The upstream server will log event 4112. The Downstream server will log 4102 when initial replication begins and 4104 when it is finished:

Figure 17 Event ID 4112 logged on the Primary Member of a new RF.

Figure 18 Event ID 4102 on the downstream when the RF is initialized

Figure 19 Event 4104 on the downstream when Intial Replication is finished.

11. Once you see the event ID 4104 on the downstream server Initial Replication is done and you are finished with your task.

Why One-way Replication is not Recommended or Supported

Here is a snippet from the blog post by DFSR PM Mahesh Unnikrishnan covering the reasons why it is not recommended nor supported to use one way replication in DFSR in Windows 2003 R2 and Windows 2008. The full post can be found here:

https://blogs.technet.com/filecab/archive/2007/08/16/using-one-way-connections-in-dfs-replication.aspx

“We recommend that customers avoid configuring such one way connections to the extent possible since:

a)       The DFS Replication service’s conflict resolution algorithms are severely hampered if the outbound connection from a member server is deleted (or disabled). Therefore, scenarios where the DFS Replication service is unable to over-write undesired updates occurring on the ‘read-only’ member server with the authoritative contents of the hub/datacenter server may arise.

b)       Accidental deletions on the ‘read-only’ server (in this case, site server ‘design.contoso.com’) could cause issues with the replication updates being trapped on that server. Further, as described above, updates from the authoritative server can potentially not be applied since the parent folder could have been deleted locally. Therefore, with time it is possible to see substantial divergence in the contents of the replicated folders across all replication member servers.

c)       Problems with the deployment are difficult to detect without regular and meticulous monitoring. There might be a lot of false positives in the health report and system eventlogs owing to the fact that the replication topology is being set up to do something DFSR wasn’t designed to handle. Mining through these false positives and monitoring servers can be a challenge.

d)       Administrators need to develop their own scripts to identify which files are backlogged on the ‘read-only’ member (in this case site server ‘design.contoso.com’) and replicate authoritative content back to that ‘read-only’ site server. This can be quite tricky to get right and might need a lot of very close monitoring (perhaps, at times on a per-file basis). Microsoft does not supply any tools for this purpose.

e)       There is a risk of administrators inadvertently creating the missing connection and causing backlogs to flow to and corrupt the contents of an authoritative server. With these changes getting replicated out further from the authoritative server, the contents of the replicated folder could get out of sync and corrupt on all replication member servers very quickly.

“Please note that configuring one way connections is not a configuration supported by Microsoft Product Support Services.”

Reducing Unwanted Changes

To reduce unwanted changes at certain servers you have a few items in your arsenal. These work for UNC connections.

1. If it is not necessary to share the data on some servers, don’t share it. DFSR does not require that a replicated folder be shared.

2. Set Share permissions (not NTFS permissions) to read only.

3. If you never want anyone accessing the data through the DFS Namespace, disable the referral for that target or delete the server as a folder target.

Hopefully you will never find yourself in the situation where you need to use the information in this blog post.

- Warren “Don’t Call Me Warren” Williams

Internet Explorer behaviors with Kerberos Authentication

Hey Rob here again, I thought that I would share with you some of the things that we see where Internet Explorer Kerberos authentication fails.

It is important to understand the default behavior of Internet Explorer and its support for Kerberos authentication so that you don’t start ripping out your hair (can’t speak to what Ned does here). I have listed three very common problems that we typically see when Kerberos authentication is failing with web-based applications.

Scenario 1: Website does not use the standard TCP/IP port of (80/443)

Web Server Configuration:

• Webserver1 has two different IIS sites running on it.
• Website1 runs with a web application pool account assigned to NetworkService.
• Website2 runs with a web application pool account assigned to a domain user account.
• The website2 is configured to listen on Port 8080.
• The following SPN’s have been defined on the website2 application pool account.
  • http/webserver1.contoso.com:8080
  • http/webserver1:8080

In this scenario you can see why a non-standard port is being used since multiple websites have been configured on the same web server. When this happens you need to specify the port to be used when you add the Service Principal Name, otherwise there is going to be a high likely hood that you will get a Kerberos ticket for the wrong web application pool account.

Client Workaround

In this scenario you need to make sure that when Internet Explorer accesses Website2 that it asks for a Service Principal name with the port number defined. However, the default behavior of IE is to not add the port number to the Kerberos ticket request. When this ticket is presented to IIS you will see a KRB_AP_ERR_MODIFIED message back.

You will need to use the below KB article change the default behavior on all IE client versions. For Internet Explorer 6 it will require the QFE Brach of Wininet.dll to be installed before the registry change will actually work.

908209 Internet Explorer 6 cannot use the Kerberos authentication protocol to connect to a Web site that uses a non-standard port in Windows XP and in Windows Server 2003 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;908209

I can tell you that there is not a version of this KB article for IE7 and above, but you do have to make the same registry change for these versions of IE also.

Server Workaround:

There really is not a good workaround to the issue other than to use host headers for one of the websites and adding a DNS HOST record for the host header in your DNS configuration. You will see shortly why we are not recommending a CNAME record in DNS.

Scenario 2: CNAME DNS RR is used for a website.

Web Server Configuration:

• Webserver1 has two different sites running on it.
• Website1 runs with a web application pool account assigned to NetworkService.
• Website2 runs with a web application pool account assigned to a domain user account.
• Website2 is configured to us a host header of app1.contoso.com.
• In DNS a CNAME record for app1.contoso.com was created and pointed to webserver1.contoso.com HOST record.
• The following SPN’s have been defined on the website2 application pool account.
  • http/app1.contoso.com
  • http/app1

In this scenario it appears that this should work just fine. When a user goes to app1.contoso.com the client machine is going to do a DNS lookup, and the DNS server is going to respond with the CNAME record and point to the webserver1.contoso.com HOST record. We can also see that the Service Principal Name configuration is properly configured on the web application pool account for website2.

The default behavior of Internet Explorer is to generate the Kerberos ticket request for the HOST record that is returned from a CNAME record, not the actual CNAME record itself. So IE specifically asks for a Kerberos ticket for http/webserver1.contoso.com which will result in a Kerberos ticket being generated encrypted with the WebServer1 computer’s password. This will in turn generate a KRB_AP_ERR_MODIFIED from IIS back to IE when the user attempts to visit the app1.contoso.com website.

Client Workaround

You will need to use the KB articles below to change the default behavior on all IE versions. For IE 6 it will require the QFE Brach of Wininet.dll to be installed before the registry key change will actually work.

For Internet Explorer 6:

911149 Error message in Internet Explorer when you try to access a Web site that requires Kerberos authentication on a Windows XP-based computer: "HTTP Error 401 - Unauthorized: Access is denied due to invalid credentials" - http://support.microsoft.com/default.aspx?scid=kb;EN-US;911149

For Internet Explorer 7 and above:

938305 Error message when you try to log on to a Web site that requires Kerberos authentication by using Internet Explorer 7: "Access is denied due to invalid credentials" - http://support.microsoft.com/default.aspx?scid=kb;EN-US;938305

Server Workaround:

The only work around is to remove the DNS CNAME RR and replace it with a HOST RR.

Scenario 3: Website works on first access but starts failing 30 minutes later

Web Server Configuration:

• Computer Webserver1 one site on it.
• Website1 runs with a web application pool account assigned to a domain user account.
• The following SPN’s have been defined on the website1 application pool account.
  • http/webserver1.contoso.com

When a user visits the website they use the NETBIOS computer name for the URL to visit. For example: http://webserver1.

In this scenario there does not seem to be much wrong here, except that there is only the FQDN version of the Service Principal Name defined. Yeah, I know all of our documentation around Kerberos always states to add FQDN as well as NETBIOS name versions of the SPN. Believe it or not, we see this all the time where the user did not register both of them, but stick with me here.

The default behavior of Internet Explorer is to add on the computer’s DNS suffix or use DNS suffix search order if defined on the machine to whatever the user types in the URL if it is not a dotted name. If your DNS configuration is correct it will resolve to webserver1.contoso.com. Once IE finds this name it stores the DNS entry in its own DNS cache. Just like most caches it times out - for IE’s cache it is 30 minutes. After 30 minutes IE again has to resolve the name, however the next time it does not try to resolve the name through DNS again, it tries just NetBIOS name resolution (hopefully there is WINS in the environment; otherwise it will just fail). Based on your configuration you could expect one of the following Kerberos errors:

• KRB_AP_ERR_MODIFIED – Expect to get this error if web site name is the same name as your web server’s computer name. That is because it is going to ask for an http/webserver1 SPN and will resolve to HOST/webserver1 which is assigned to the computer account.
• KRB_ERR_S_PRINCIPAL_UNKNOWN – Expect to get this error if the web site name is something like app1.contoso.com. That is because it is going to ask for an http/app1 SPN and will not resolve to any account in the domain.

Client Workaround:

Thankfully there is a fix that can be implemented for Internet Explorer.

899417 You may receive an "Access is denied" error message you use the WWW-Authenticate: Negotiate method of HTTP authentication to connect to a Web server - http://support.microsoft.com/default.aspx?scid=kb;EN-US;899417

I will tell you that there is not a version of this KB article for IE7 and above, but you do have to make the registry key change for these versions of IE also before the functionality is supported.

Server Workaround:

You can register the NetBIOS version of the Service Principal Name to the account, using SETSPN.EXE.

I hope that you found this post interesting. As always it is easier to spot these type of issues by reviewing network trace taken at the client side (where IE is being used) to find the root cause of the issue.

- Rob “I Speak Tampa” Greene

Friday Silliness

Ned here. I don't feel like being particularly technical today, so check out a good chuckle:

http://www.codesqueeze.com/the-ultimate-top-25-chuck-norris-the-programmer-jokes/

If you don't know who Chuck Norris is:

1. What is your problem?
2. He is the greatest actor of our generation.
3. He does not sleep. He waits...

- Ned 'The Octagon' Pyle

Migrating from PolicyMaker to Group Policy Preferences with GPPMIG

PolicyMaker customers rejoice—Microsoft has a way for you to migrate from PolicyMaker 2.x to the new Group Policy preferences released with Windows Server 2008 and included in the Remote Server Administration Tools for Windows Vista Service Pack 1 or higher.

Download GPPMIG: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=35791cb6-710b-48c4-aaa1-90db170bcf2a

PolicyMaker to Preferences… how to get there

If you’ve been using PolicyMaker then you already know how to use Group Policy Preferences. It is all managed using the Group Policy Management Console included with Windows Server 2008 or, using Windows Vista Service Pack1 by installing the Remote Server Administration Tools. However, Group Policy preferences cannot process PolicyMaker data and vice versa. Therefore, you need to have a strategy to migrate from PolicyMaker to Group Policy Preferences. Hopefully, this should help. Everything discussed below is also included in the GPPMIG installer as the ‘GPPMIG Migration Guide'.

Prerequisites

I want to take a few minutes to discuss some of the prerequisites before we jump right into the migration strategy. We have two categories Management and Client.

Management

Policymaker’s management looks and feels the same as managing other Group Policy setting. The same look and feel returns using Preferences. One thing to consider is-- each instance (PolicyMaker or Group Policy Preferences) cannot edit the others data. For this reason, you may need to leave one or more Windows XP computer, with the PolicyMaker administrative tools installed, until you’ve completed your migration. If your migration follows a staged approach, then you may encounter a small period of time where you may need to manage using both Windows Vista and Windows XP. Or, you may be the weekend warrior type and have your migration complete from Friday to Monday. The choice and freedom are there, but the requirement remains—PolicyMaker administrative additions can only edit PolicyMaker items. Server 2008 and the RSAT tools can only edit Preferences. Read Microsoft Knowledgebase article 941314, Description of Windows Server 2008 Remote Server Administration Tools for Windows Vista Service Pack 1 for more information.

Client

The critical component that actual makes PolicyMaker and/or Preferences work is the client side extensions (CSEs), which you must install on the client computer. The CSEs make normal Group Policy processing PolicyMaker/Preferences aware. The same rules apply to the client portion—PolicyMaker CSEs only process PolicyMaker data and Preference CSEs only process Preference data. Also, installing the Group Policy Preference CSEs automatically removes PolicyMaker CSEs. The new Group Policy preference client side extensions installs on

• Windows Vista RTM and Service Pack 1
• Windows Server 2003 Service Pack 1
• Windows XP Service Pack 2

Both Windows Server 2003 and Windows XP require the installation of XmlLite prior to installing the CSEs. Preference CSEs are included in Windows Server 2008. Read Microsoft Knowledgebase article 943729, Information about new Group Policy preferences in Windows Server 2008 for more information.

Testing

It goes without saying—you can never test enough and this scenario is not any different. Make sure you have backups… and they actually work. If you are going to use GPMC to backup your GPOs, then remember to use the correct version. GPMC backups are not interchangeable. If you backup with pre-Server 2008 GPMC, then you must restore with the same version. Back up some of your most complex or important GPOs and then important them into isolated test GPOs in a test OU with a single user and computer. Run through your entire migration strategy—noting what works and what does not— refining the plan with each pass. All efforts spent in planning usually pay off during implementation.

Group Policy Preference Migration utility

Now that we have the planning stuff out of the way—on to the good stuff. GPPMIG is a console application developed with version 3.0 of the .NET framework. Use GPPMIG to migrate PolicyMaker items to Group Policy Preference items into the same or a different Group Policy object. GPPMIG does not migrate PolicyMaker Application or Mail Profile data as Group Policy Preferences do not included client-side extensions for these items.

What it does

Let us take a few moments to discuss how GPPMIG works. For starters, GPPMIG always uses the domain of the currently logged on user. You’ll want to remember this so you can log on with domain administrator account for the domain GPOs you want to migrate. And, you must be a domain administrator as GPPMIG write to SYSVOL and Active Directory. One last point is that GPPMIG always connects to the PDC of the user domain—for reading and writing to Active Directory and SYSVOL. So, you’ll want to run GPPMIG from a computer close (same subnet) as the PDC emulator.

With GPPMIG, you can target a single GPO to migrate or, you can choose to migrate all GPOs. GPPMIG performs a paged LDAP query to the PDC to retrieves a list of all the Group Policy objects in the user’s domain. GPPMIG then filters out any GPO in the list that is not configured for PolicyMaker items. Then, GPPMIG iterates through each GPO in the final list, looking for PolicyMaker specific client side extensions in each GPO. The entire GPO is evaluated before moving to the next. If a PolicyMaker setting is found, then GPPMIG ensures there is not an equivalent Group Policy Preference configuration, as it will not migrate PolicyMaker items into existing Group Policy Preference items. When GPPMIG completes its search for PolicyMaker items in the GPO, it then updates the Group Policy object to included Group Policy Preference client side extensions and then increases the version number for the user, computer, or both depending on what PolicyMaker items it migrated. In no way does a migration alter any PolicyMaker items for the GPO. All PolicyMaker items remain configured and available in the GPO. GPPMIG creates a migration log in the directory from which it ran.

Usage

You can use GPPMIG to migrate to Group Policy Preferences in staged approach or, you can create brand new GPOs to hold your new Group Policy Preference items and migrate to those new GPOs. The staged approach is a planned migration strategy and is the approach I’ll document here. After reading this, you should be able to alternate this strategy to best suit the needs of your environment. Generally, you’ll migrate from PolicyMaker to Group Policy Preferences in three stages (after you’ve done your testing).

• Stage 1— Identify GPOs containing PolicyMaker items and use GPMC 1.x to back up those GPOs
• Stage 2— Migrate PolicyMaker items to Group Policy Preference items in the same or a new Group Policy object. Then, deploy the Group Policy Preference CSEs to your client computers.
• Stage 3— Confirm Group Policy Preference items migrated and are successfully applying to user and computers. Use GPMC 6.0.0.1 to backup your GPOs (to a different back up location then Stage 1. Then remove PolicyMaker items from GPOs, if applicable

Download GPPMIG: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=35791cb6-710b-48c4-aaa1-90db170bcf2a

Commands

GPPMIG contains four basic commands:

• Whatif — display all the Group Policy objects that contain PolicyMaker items
• Migrate— migrates PolicyMaker items to Group Policy Preference items in the same GPO
• MigrateTo— migrates PolicyMaker items to Group Policy Preference items to a different GPO
• Remove— removes PolicyMaker items from a GPO

Stage 1 – Identify PolicyMaker GPOs

Begin your migration process by identifying GPOs containing PolicyMaker items. You can do this by using the –whatif command. Use the –all command afterwards to search all the GPOs in the user’s domain or, you can use the –name command and provided the display name of the GPO. Use GPMC to backup all of the GPOs identified to have PolicyMaker items.

Stage 2 – Migrate PolicyMaker Data to Group Policy Preferences

Next, you’ll want to migrate PolicyMaker items to Group Policy Preference items. You have a choice to migrate the setting within the same or to a different Group Policy object.

Note
The migration does not modify PolicyMaker items, regardless of the migration action you choose.

Use the –migrate command to migrate PolicyMaker items to Group Policy Preference items within the same GPO. Use the following syntax:

Gppmig –migrate –name:gpo_name

Alternatively, you replace the –name argument with –all to migrate all the GPOs in the users domain that contain PolicyMaker items.

Figure 1 GPPMIG In-place Migration

You may prefer to keep PolicyMaker GPOs separate from Group Policy Preference GPOs. You use the –migrateTo command to accomplish this task

Important
You must create the target GPO before using the -migrateTo command. GPPMIG does create the target Group Policy object.

Figure 2 - GPPMIG Source-target Migration

The –migrateTo command requires two additional arguments: -source: and –target: follows by the display name of the Group Policy object. Enclose the name of the GPO in quotes if the name contains spaces. Also, the –migrateTo command does not support the –all argument.

Deploy GPP Client

You’re now ready to deploy the Group Policy Preference client-side extensions after you’ve migrated all of your GPOs to include Group Policy Preference items. The migration does not modify any PolicyMaker items; so clients with the PolicyMaker CSE and the Group Policy preference CSEs process the same data

Note
GPPMIG does not migrate Application or Mail PolicyMaker items. Therefore, Group Policy Preference CSEs do not apply these items to users or computers. Leave the PolicyMaker CSE installed on computers that require these items and do not install the Group Policy Preference CSEs as the installation removes PolicyMaker CSEs).

You can apply Group Policy Preferences to several Microsoft operating systems. The minimum operating system requirements are:

• Windows Vista RTM or Windows Vista Service Pack 1 (32 or 64-bit)
• Windows Server 2003 Service Pack 1 or later (32 or 64-bit)
• Windows XP Service Pack 2 or later (32 or 64-bit)

Group Policy Preference client-side extensions are included in Windows Server 2008. You can use the links above to download the client-side extension installation packages. Or, you can download the extensions as an optional update from Windows Update.

Important
Remember-- installing Preference client-side extensions removes PolicyMaker Client Side Extensions.

Stage 3

The last stage in the migration process involves verifying your items migrated and apply correctly. Use GPMC to view the Group Policy object to which you migrated your items. Click the Settings tab to show the Preference items included in the GPO.

Figure 3 GPMC Settings Tab

Next, you'll want to apply the Group Policy object to your client computers. For in-place migrations, you'll want to apply the GPO to computers using PolicyMaker CSEs and computers using Preference CSEs. Also verify user PolicyMaker and Preference items apply to the appropriate user. GPOs that are targets of in-place migrations should apply items to both (PolicyMaker and Preferences). Source-target migrations migrate the PolicyMaker items to Preference items in the newly created GPO. This allows you to keep your existing PolicyMaker GPOs separate from your Preference GPOs. You apply GPOs containing Preference items to computers are users using the Group Policy Preference CSEs.

Use the Resultant Set of Policy (RSOP) management console to confirm PolicyMaker items are applying to computers or users. Use the Group Policy Results feature within GPMC to confirm Preference items are applying to computers or users.

Figure 4 GPMC Group Policy Results for testuser

The actual migration from PolicyMaker to Group Policy Preferences is complete. Computers running either Preferences or PolicyMaker should be applying their respective items. Source-target migrations contain both PolicyMaker and Preference items. After you've transitioned your client to use the Group Policy Preference CSEs, you'll want to remove the PolicyMaker data, which remains in the GPO. You can use GPPMIG with the -remove option to remove overlapping PolicyMaker and Preference items.

Figure 5 Removing PolicyMaker settings

Note
GPPMIG does not remove PolicyMaker Application and Mail items from the Group Policy object.

Source-target Migrations do not included PolicyMaker items. Therefore, once you've completed transitioning your client computers to use Preference CSEs, you can delete the source version of the GPO, which contains only PolicyMaker items.

Conclusion

You should consider backing up your Group Policy objects after you've completed your migration and cleanup of Group Policy objects. Use the Group Policy Management Console included in Windows Server 2008 and the Remote Server Administration tools to backup all of your Group Policy objects before you proceed with any further changes.

- Mike Stephens

Potential for Kerberos Issues When Using a Cisco VPN/ASA with Win2003 or later DC’s

Hey everyone, Rob Greene here back after a long hiatus from blogging. I had an interesting case come through that I thought many of you IT pros would be interested in.

Background

The customer had an issue with using Cisco VPN and Cisco ASA concentrators and authenticating the user with Kerberos authentication. After they upgraded all their Windows 2000 domain controllers to Windows Server 2003 domain controllers, users could not authenticate through the VPN.

After doing this they gave Cisco a call and they were told to start DSA.MSC and check the box under Account Options to “Do not require Kerberos preauthentication” for all user accounts that need to connect through the appliances. When they did this as a test it did allow the user to start authenticating through the devices. However the customer did not like this option since:

1. This change lowers the security of Kerberos authentication by disabling password replay attack protection.

2. Everything had worked previously with Windows 2000.

If you want to learn more about preauthentication you can click here and look under the section:

Example AS Administration.

If you search Cisco’s website you will find that they have older documentation stating that you must disable this feature on the user’s account before the VPN will start working.

I don’t have real network traces (as I can’t show you a real customer’s network!), but I can tell you what we saw in the traces that led us to a potential workaround.

VPN Appliance --> UDP Port 88 / KRB5 AS-REQ --> Windows DC

VPN Appliance <-- UDP Port 88 / KRB5KDC_ERR_REAUTH_REQUIRED <--Windows DC

VPN Appliance --> UDP Port 88 / AS-REQ (with Pre-Auth Data) --> Windows DC

VPN Appliance <-- UDP Port 88 / KRB5KDC_ERR_RESPONSE_TOO_BIG <-- Windows DC

A KRB5KDC_ERR_RESPONSE_TOO_BIG means that the requested Kerberos data was larger than the maximum default datagram size of 1465 bytes. The next step here usually would be for the client to start the Kerberos ticketing process over using TCP rather than UDP. If this was a Windows XP client instead of the Cisco VPN appliance we would have the customer implement the registry key change outlined in the below KB article.

244474 How to force Kerberos to use TCP instead of UDP in Windows
http://support.microsoft.com/default.aspx?scid=kb;EN-US;244474

So basically the real issue is that the Cisco appliance is being asked by the KDC to use TCP instead of UDP to do the AS-REQ with pre-authentication data present. However, the Cisco appliance does not continue at this point. I spent the past few weeks working with Cisco directly (big shoutout - they have been great!), and they have published the following new articles:

Cisco VPN 3000: CSCed27444 - VPN3000 does not switch to TCP upon receiving Kerberos error code 52

Cisco ASA platform: CSCsi32224 - ASA does not switch to TCP upon receiving Kerberos error code 52

Unfortunately, the Cisco VPN 3000 concentrator has been discontinued so there is no update available for the Cisco VPN 3000 to add this functionality. Also, you will need a support contract from Cisco to logon to those two articles and read them.

Please contact Cisco directly to find out more information about the issue, and with what IOS version it will be included in; we have provided this information to make it easier to find the content.

In the Meantime

So now the question becomes this: If that is what is going on, then how did it work when a Windows 2000 domain controller was used? And what can you do about it until an IOS update is released?

There was a change in the default datagram reply packet size used by the Kerberos Key Distribution Center (KDC) between Windows 2000 and Windows Server 2003. In Windows 2000 the default size is 2000 bytes, where it is 1465 bytes in Windows Server 2003. For those interested you can review the below KB article and find the registry key of MaxDatagramReplySize.

837361 Kerberos protocol registry entries and KDC configuration keys in Windows Server 2003
http://support.microsoft.com/default.aspx?scid=kb;EN-US;837361

So that means this customer’s Kerberos ticket size for the user accounts is less than 2000 bytes, but more than 1465 bytes. Keep in mind that if you are reading this blog, your environment may still not work with the registry key setting at 2000. This is because the Kerberos ticket size is going to vary based on the number of groups that the user belongs to. You can review the KB below to learn more:

327825 New resolution for problems with Kerberos authentication when users belong to many groups
http://support.microsoft.com/default.aspx?scid=kb;EN-US;327825

The only workaround that we can give a customer is to modify the MaxDatagramReplySize registry value to the size of your user’s largest Kerberos ticket size. It really depends on the group membership size of the user attempting to authenticate because of the PAC data that is included in the TGT. For those customers that have used the MaxTokenSize registry value should equal that value and restart the KDC service. If none of these acronyms are making sense you might want to check out my older blog post Kerberos for the Busy Admin.

Hey wow, this sounds like a pretty easy fix right? Well, you can actually have other problems because of this change. If your networking gear (switches, routers, firewalls) cannot handle large UDP packets without fragmenting them then all Kerberos authentication could start to fail in your environment. See KB244474 under more information to learn more about this.

Please keep in mind what you are doing when you implement the MaxDatagramReplySize registry value. You are in fact telling the KDC to never request the client send KRB5KDC_ERR_RESPONSE_TOO_BIG message back to a Kerberos client, and you should consider implementing the custom group policy setting included in KB244474 on all domain-joined machines to use TCP for Kerberos authentication always. If you are running purely Vista/2008 or later though, this is on by default.

Lastly, if you have to set this value very high, and you know that your networking gear will not support the packet sizes that had to be implemented, you may have to put a domain controller on the same IP subnet as the Cisco appliance to successfully get Kerberos tickets.

- Rob ‘I mean, you know’ Greene