January, 2009

New KB articles related to Directory Services for the weeks of 12/22-1/4.

Hi. Jim here again from the Directory Services team. Today I will break down some of the core components of credential roaming and how it functions. To secure critical transactions such as signing, encrypting, and decrypting e-mail or authenticating identity, many environments rely on certificates. The user certificates and the associated private keys are linked to the local profile on whatever machine the user logs onto. Let’s talk about the local user profile briefly. When a user logs onto a computer for the first time a local profile is created for that user. This profile sets the desktop environment, certificates, private keys, and all other user-specific configuration information. When the user logs back onto the same machine this profile is reused.

You can see where I am going here. What if the user logs onto multiple machines? In this scenario a user could end up having separate sets of certificates and private keys on each computer's local profile. Administrators would have to manage multiple certificates stored on multiple machines in multiple profiles. What if a profile on one machine gets deleted? What if that deleted profile contained certificates and private keys used to decrypt data? What if the user leaves the company? I hope you have key recovery configured.

Enter Credential Roaming –

Credential roaming functionality addresses these concerns by making it possible to roam only the user's credentials in a manageable, secure manner that is ultimately transparent to the user. For the purpose of this post we will only be discussing the roaming of certificates and private keys. The credential roaming implementation in Windows Vista and Windows Server 2008 is additionally able to roam stored user names and passwords. For more information on this go here - "How to Manage Stored User Names and Passwords on a Computer in a Domain in Windows XP” at:

http://support.microsoft.com/?id=306992

Remember, credential roaming only supports x.509 v3 certificates and RSA or DSA key pairs that are stored in the user’s credential store. Only credentials in the user’s MY and REQUEST stores are roamed.

Ok, Ok, enough already. Let’s get into how credential roaming works in a nutshell. What you need first is a functioning, healthy Active Directory environment. This is crucial as credential roaming relies on group policy, auto enrollment and healthy Active Directory and SYSVOL replication. Windows Server 2003 SP2 and XP SP3 both support credential roaming. The update below must be applied to XP SP2 computers as well as Windows Server SP1 machines to facilitate credential roaming.

907247 Description of the Credential Roaming service update for Windows Server 2003 and for Windows XP - http://support.microsoft.com/default.aspx?scid=kb;EN-US;907247

After you have verified you are running the correct versions of Windows and you have deployed the 907247 update, you can begin the initial configuration of credential roaming in your organization. In order to receive the script necessary to modify your Active Directory schema you must have an engagement with Microsoft Consulting Services or you can request the script from Microsoft Customer Support Services. For more information about the request procedure, see the Microsoft Knowledge Base article

http://support.microsoft.com/default.aspx?scid=kb;en-us;907247&sd=tech

I will not be covering the initial credential roaming configuration as the steps for configuration are explained here in exhaustive detail:

Configuring and Troubleshooting Certificate Services Client–Credential Roaming - http://technet.microsoft.com/en-us/library/cc700821.aspx

Once credential roaming functionality has been configured in your Active Directory environment, here is what happens on an XP computer with the 907247 update installed. This also assumes healthy active directory replication and successful group policy application in regard to credential roaming policy specifically.

1. A user logs on to a domain joined computer.

2. Group policy applies successfully and includes the policy setting for credential roaming.

Here is the location in the registry where the Credential Roaming Group Policy settings are written:

HKEY_CURRENT_USER\Software\Policies\Microsoft\Cryptography\Autoenrollment
    AEPolicy    REG_DWORD    0x7
    DIMSRoaming    REG_DWORD    0x1
    DIMSRoamingTombstoneDays    REG_DWORD    0x3c
    DIMSRoamingMaxNumTokens    REG_DWORD    0x7d0
    DIMSRoamingMaxTokenSize    REG_DWORD    0xffff

3. Coinciding with Winlogon and Userinit.exe the dimsntfy.dll (which has been updated by the 907247 update) sets off an event to activate credential roaming. The dimsntfy.dll lives within the WINDOWS\system32 folder.

4. Also located in the WINDOWS\system32 folder, the Dimsroam.dll (also updated by the 907247 update) is the Key Roaming DIMS Provider DLL. The Dimsroam.dll is compelled by group policy to count the number of tokens in the local certificate store. The maximum number of tokens allowed to roam can be configured within the following credential roaming group policy setting.

If there are more tokens available then what is specified (In the above screenshot the default maximum of 2000 per user is set) credential roaming will not work. If the total number of non-tombstoned credentials plus tombstoned credentials is less than two times the number indicated in the group policy setting above, then the credential roaming series of events proceeds on its merry way.

(Non Tombstoned + Tombstoned) < 2000 x 2 = credential roaming works

(Non Tombstoned + Tombstoned) > 2000 x 2 = credential roaming fails

You do the math ; )

5. The local credentials of the user and the credentials within the Active Directory object of the user are compared against the state.dat file. The state.dat is the current state file. The state.da~ is the previous state file. When credential roaming reconciles new certificates from Active Directory to the local store and from the local store back up to Active Directory the state.dat file records those transactions. Each time this occurs the original state.dat file is renamed to state.da~ and a new state.dat is created. Both the state.dat and the state.da~ files are located here on a Windows XP machine:

C:\Documents and Settings\”username”\Local Settings\Application Data\Microsoft\DIMS

The security on this directory by default is as follows:

  DomainName\UserName: F (Full Control – Inherited)
  NT AUTHORITY\SYSTEM: F (Full Control – Inherited)
  BUILTIN\Administrators: F (Full Control – Inherited)

On a Vista machine, it is located here –

  %LOCALAPPDATA%\Microsoft\DIMS

Any modification to these permissions may affect the functionality of credential roaming. I will speak more of this in a forthcoming credential roaming troubleshooting post. I must also point out in regard to Vista that the NTDS.DIT file on domain controllers grows continually larger after enabling credential roaming for Windows Vista clients. Please see the following for information on the hotfix, or install Vista Service Pack 1 to proactively avoid this issue:

934797 - The size of the Ntds.dit file on the domain controller grows continually larger after you enable the "Credential Roaming" feature for Windows Vista-based client computers in the domain

http://support.microsoft.com/default.aspx?scid=kb;EN-US;934797

Let’s press on shall we…

If there has been a modification of the certificates in the local store or the user object in Active Directory, the changes are either downloaded to the store or uploaded to Active Directory.

The reconciliation of the local store with Active Directory and vice versa is handled by the dimsroam.dll. The private key material is encrypted the DPAPI key and all credential roaming communication between the computer and Active Directory is Kerberos encrypted.

The dimsroam.dll also deletes certificates that are older than the tombstone lifetime in days set in the credential roaming group policy. These certificates with an older tombstone are deleted from both Active Directory and the user’s local store. Any reference to them is also removed from the state.dat file. The Tombstone value should be a value long enough so that the user is able to logon to all machines that they use so that the certificate is deleted from all the respective local certificate stores upon logon. If this value is too low, then previously deleted certificates can be put back into the AD Credentials store and cause the certificate to be repopulated on all machines that the user logs onto.

6. The maximum size of the user object's credentials attribute is the result of the maximum number of tokens multiplied by the maximum token size. A user object can become large if there are a large number of tokens that also have a large token size. With certificates the token size will vary depending on the key size. The size above (65535) is default. It is conceivable that a user’s credential size would be larger than what is defined in the policy, thus causing credential roaming to fail for that user.

7. If auto-enrollment is configured in group policy the pautoenrol.dll kicks off AFTER credential roaming code is finished. This prevents auto-enrollment from taking place needlessly. If auto-enrollment was to take place before credential roaming it is conceivable that a user logging on to a different machine would enroll for a certificate already procured on another machine and waiting to be roamed.

The Dimsroam.dll will fire off the credential roaming series of events described to this point when the following occurs:

• At user logon and logoff.
• The user locks or unlocks the machine console.
• certutil -user -pulse is performed at a Windows Vista or Windows Server 2008 command prompt.
• A change is made to the user's local certificate store by removing a certificate from the personal store, importing a certificate to the personal store, manually enrolling for a certificate with the MMC or certreq.exe, receiving a new certificate via auto-enrollment and a group policy refresh.

In addition Credential Roaming also roams the DPAPI master keys. For more explanation of the Windows Data Protection API (DPAPI) go here. For group policy only there may be a latency of up to four hours before the certificate store is refreshed. This is by design and was introduced to reduce high load on domain controllers when group policy is updated or created. Like anything else, credential roaming deployments require careful consideration and testing before enterprise wide roll outs. Go here for more exhaustive information on all things credential roaming.

- Jim “Lord of the Strings” Tierney

Hi, Ned here again. Today I'm going to go through some well-hidden information on DFSR SYSVOL migration; hopefully this sticks in your brain and someday allows you to enjoy your weekend rather than spending it fighting issues.

As you already know, Windows Server 2008 introduced the ability to use Distributed File System Replication (DFSR) to replicate your SYSVOL folder, rather than the legacy File Replication Service (FRS). And there was much rejoicing. The step-by-steps for this process are documented here:

1: SYSVOL Migration Series: Part 1 – Introduction to the SYSVOL migration process
2: SYSVOL Migration Series: Part 2 – Dfsrmig.exe: The SYSVOL migration tool
3: SYSVOL Migration Series: Part 3 – Migrating to the ‘PREPARED’ state
4: SYSVOL Migration Series: Part 4 – Migrating to the ‘REDIRECTED’ state
5: SYSVOL Migration Series: Part 5 – Migrating to the ‘ELIMINATED’ state

(And before you get wound up in the Comments section - Yes, there is a TechNet Whitepaper for this coming just as soon as we can get it published! Don't hurt me!)

So let's run through some Q & A.

Q: What are the Domain Controller availability requirements during my migration?

A: There are a couple.

1. The PDC Emulator must be online any time the DFSRMIG tool is being invoked for a read or write operation. If the PDC Emulator is offline or inaccessible for LDAP, the user of DFSRMIG will receive:

“Unable to connect to the Primary DC’s AD.

Please make sure that the PDC is reachable and try the command later.”

2. All DCs must remain online until they each complete their state steps. All DCs do not need to be accessible simultaneously. But the global state will never reach the Prepared, Redirected, or Eliminated state until all DCs have been able to complete their individual phases.

The PDC Emulator requirement is because by default, administrators always edit group policy on the PDCE, so in most environments it will have the most up to date knowledge of policy. That and we need to talk to someone unique, so it might as well be him.

It is recommended that a SYSVOL migration not be attempted unless all DCs are online and available. Change control blackouts should be scheduled to prevent modification to DCs that might impact their availability. This will minimize the window of time that the migration will take.

Q: Is there some super-secret way to return to using FRS after reaching the Eliminated phase of DFSR migration?

A: Microsoft does not support returning your domain to using FRS for SYSVOL replication after a completed DFSR migration (except to rebuild the domain). This is why the steps are done in a phased approach with two checkpoints where you can revert back to FRS without any consequences. Once you trigger the ELIMINATED phase to start, there is no going back, period.

Q: When does Robocopy run during the migration and what does it do?

A: The DFSR service uses robocopy at several stages to synchronize SYSVOL directories outside of normal replication when it detects a SYSVOL migration is underway; a set of ‘pre-seeding’ and ‘save the GP admins from themselves’ operations.

1. When Prepared state (DFSRMIG /SETGLOBALSTATE 1) is invoked, all DC’s robocopy their FRS SYSVOL data locally into the new DFSR content set. This is equivalent to ‘pre-seeding’ data and ensures that minimal file replication occurs to converge the content set. This is triggered by the DFSR service itself when:

• AD replication has converged between a DC and the PDCE.
• The DFSR service on that DC has polled (this runs every 5 minutes) and picks up the state change from CN=dfsr-LocalSettings

2. When entering the Redirected state, the PDC Emulator (only) robocopies the local differences of FRS SYSVOL data into the new local DFSR content set, on itself. The other servers receive new data via replication.

3. If you undo the Redirected state back to Prepared, the reverse happens. The PDC Emulator robocopies its local DFSR content set data into its local FRS content set. FRS replication synchronizes all other servers... eventually. Allow more time for this than entering Redirected, as FRS is not as fast to synchronize as DFSR.

For sharp-eyed readers: we won’t run into any of the old pre-seeding issues (the file hash being changed by robocopy) here because DFSR correctly creates the SYSVOL_DFSR folder ACL, so there are no inheritance issues when the contents are copied in and replicated.

Q: Event 8004 says something about RODC's. I don't have any RODC's. What the frak?

A: The following event is incorrectly written in the DFSR event log(s) on servers that are not Read-only Domain Controllers when setting elimination state using DFSRMIG.EXE:

Well, it finally happened – we had a bug. No, no, don’t try to tell me it’s not possible, it was bound to occur someday. :)

The text in the event log is completely cosmetic and benign. It is supposed be fixed in a later version of the OS. Just ignore it.

Q: What are all the AD and Registry state values that will be set at a given point in the migration?

A: Ok, you asked for it buddy.

=============

1. Prepared Phase - DFSRMIG /SETGLOBALSTATE 1

a. DFSRMIG contacts the PDC Emulator directly.

b. Global objects are created under:

CN=DFSR-GlobalSettings,CN=SYSTEM,DC=<domain>
CN=DOMAIN SYSTEM VOLUME
  CN=SYSVOL SHARE
   CN=CONTENT
    CN=TOPOLOGY

c. CN=DFSR-GlobalSettings now has msDFSR-Flags attribute set to 0.

d. As DC's pick up the globalstate change via AD replication and DFSR service polling, they create and write to registry entry:

HKLM\System\CurrentControlSet\Services\DFSR\Parameters\Sysvols\Migrating Sysvols
Local State = 4 [REG_DWORD]

e. The PDC Emulator creates the:

CN=dfsr-LocalSettings,CN=<servername>,DC=<domain>

objects for all DCs and sets this attribute to:

msDFSR-Flags = 80 (if RWDCs).
msDFSR-Flags = 64 (if RODCs - the RODC itself will set it to 80 later).

f. The DFSR service has now started and created the new local SYSVOL_DFSR structure. Robocopy has made a local copy of the FRS SYSVOL. All AD topology data has been written in to support the content set. Initial sync of the data has started (since robocopy has locally pre-seeded the data this should involve minimal replication data on the network). The registry on all DC's is:

Local State = 5 [REG_DWORD]

g. Once initial sync is done on all DCs:

Local State = 1 [DWORD]
'msDFSR-Flags' (on CN=dfsr-LocalSettings) = 16

h. If DFSRMIG /GETGLOBALSTATE returns that all DCs are prepared, 'msDFSR-Flags' on CN=dfsr- GlobalSettings has changed to 16 because all DCs are prepared. All DCs are currently replicating DFSR and FRS content sets, with FRS being shared as SYSVOL.

=============

2. Redirected Phase - DFSRMIG /SETGLOBALSTATE 2

a. DFSRMIG contacts the PDC Emulator directly.

b. CN=DFSR-LocalSettings now has msDFSR-Flags attribute set to 96 on all DCs and this replicates out through AD.

c. As DCs pick up the attribute from AD replication, their DFSR service sets:

Local State = 6 [REG_DWORD]

d. On the PDC Emulator only, robocopy syncs any changes between the FRS and DFSR's content sets, and this is replicated out through DFSR.

e. Once SYSVOL data is in sync, SYSVOL content set is set to be the active SYSVOL share on all servers. FRS and DFSR are both still replicating data.

f. When this is complete, for each DC:

Local State = 2 [DWORD]
'msDFSR-Flags' (on CN=dfsr-LocalSettings) = 32

g. If DFSRMIG /GETGLOBALSTATE returns that all DCs are redirected, 'msDFSR-Flags' on CN=dfsr- GlobalSettings has changed to 32 because all DCs are prepared. All DCs are currently replicating DFSR and FRS content sets, with DFSR being shared as SYSVOL.

==============

3. Eliminated Phase - DFSRMIG /SETGLOBALSTATE 3

a. DFSRMIG contacts the PDC Emulator directly. At this point it is not possible to undo the changes!

b. CN=DFSR-LocalSettings now has msDFSR-Flags attribute set to 112 on all DCs and this replicates throughout AD.

c. As DCs pick up the attribute from AD replication, their DFSR service sets:

Local State = 7 [REG_DWORD]

d. On the PDC, the FRS content set information is removed and this is replicated through AD. As each DC sees this change, their FRS service stops replicating the FRS content set. The FRS service is stopped (and restarted if custom FRS sets still exist on a given server).

e. When this is complete, for each DC:

Local State = 3 [DWORD]
'msDFSR-Flags' (on CN=dfsr-LocalSettings) = 48

f. If DFSRMIG /GETGLOBALSTATE returns that all DCs are eliminated, 'msDFSR-Flags' on CN=dfsr-GlobalSettings has changed to 48 because all DCs are prepared. All DCs are currently replicating DFSR only.

g. A final cleanup task on each DC will set their 'msDFSR-Flags' on CN=dfsr-LocalSettings to <NOT SET>. The same will happen from the PDC to CN=dfsr-GlobalSettings.

==============

If any 'undo' phases are entered (where an administrator has decided to go from redirected back to prepared, redirected back to start, or prepared back to start), the flow above happens in reverse, with the exception that the following two entries exist in the 'Local State' registry entries:

8 (Undo Redirecting)
9 (Undo Preparing)

Q: I'm not a huge fan of Ultrasound. Are there any other ways to validate the health of SYSVOL prior to and after migration?

A: Sure thing - already discussed in a previous blog post here.

And that’s it. If you have any questions on DFSR SYSVOL migration not covered by this blog post or by the FileCab step-by-step guide, don’t hesitate to start swinging in the Comments section below.

- Ned “Working on Windows 7 Beta But Still Has An Unnatural Love for Win2008 DFSR” Pyle