March, 2010

Hi all, Ned here again. There are a number of ways that DFSR can be tuned for better performance. This article will go through these configurations and explain the caveats. Even if you cannot deploy Windows Server 2008 R2 - for the absolute best performance - you can at least remove common bottlenecks from your older environments. If you are really serious about performance in higher node count DFSR environments though, Win2008 R2’s 3^rd generation DFSR is the answer.

If you’ve been following DFSR for the past few years, you already know about some improvements that were made to performance and scalability starting in Windows Server 2008:

But there’s more you can do, especially in 2008 R2.

Registry tuning

All registry values are REG_DWORD (and in the explanations below, are always in decimal). All registry tuning for DFSR in Win2008 and Win2008 R2 is made here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DFSR\Parameters\Settings

A restart of the DFSR service is required for the settings to take effect, but a reboot is not required. The list below is not complete, but instead covers the important values for performance. Do not assume that setting a value to the max will make it faster; some settings have a practical limitation before other bottlenecks make higher values irrelevant.

Important Note: None of these registry settings apply to Windows Server 2003 R2.

AsyncIoMaxBufferSizeBytes
Default value: 2097152
Possible values: 1048576, 2097152, 4194304, 8388608
Tested high performance value: 8388608
Set on: All DFSR nodes

RpcFileBufferSize
Default value: 262144
Possible values: 262144, 524288
Tested high performance value: 524288
Set on: All DFSR nodes

StagingThreadCount
Default value: 6
(Win2008 R2 only; cannot be changed on Win2008)
Possible values: 4-16
Tested high performance value: 8
Set on: All DFSR nodes. Setting to 16 may generate too much disk IO to be useful.

TotalCreditsMaxCount
Default value: 1024
Possible values: 256-4096
Tested high performance value: 4096
Set on: All DFSR nodes that are generally inbound replicating (so hubs if doing data collection, branches if doing data distribution, all servers if using no specific replication flow)

UpdateWorkerThreadCount
Default value: 16
Possible values (Win2008): 4-32
Possible values (Win2008 R2): 4-63*
Tested high performance value: 32

Set on: All DFSR nodes that are generally inbound replicating (so hubs if doing data collection, branches if doing data distribution, all servers if using no specific replication flow. The number being raised here is only valuable when replicating in from more servers than the value. I.e. if replicating in 32 servers, set to 32. If replicating in 45 servers set to 45.

*Important note: The actual top limit is 64. We have found that under certain circumstances though, setting to 64 can cause a deadlock that prevents DFSR replication altogether. If you exceed the maximum tested value of 32, set to 63 or lower. Do not set to 64 ever. The 32 max limit is recommended because we tested it carefully, and higher values were not rigorously tested. If you set this value to 64, periodically replication will stop working, the dfsrdiag replstate command hangs and does not return results, and the dfsrdiag backlog command hangs and does not return results.

 

When using all the above registry tuning on Windows Server 2008 R2, testing revealed that initial sync replication time was sometimes twice as fast compared to no registry settings in place. This was using 32 servers replicating a "data collection" topology to a single hub over thirty-two non-LAN networks with 32 RG's containing unique branch office data. The slower the network, the better the relative performance averaged:

On Windows Server 2008 the same registry values showed considerably less performance improvement; this is partly due to additional service improvements made to DFSR in Win2008 R2, especially around the Credit Manager. Just like your phone, “3G” DFSR is going to work better than older models…

Note: do not use this table to predict replication times. It is designed to show behavior trends only!

Topology tuning

Even if you are not using Windows Server 2008 R2 there are plenty of other factors to fast replication. Some of these I’ve talked about before, some are new. All are important:

• Minimize mixing of Win2003 and Win2008/Win2008 R2 - Windows Server 2008 introduced significant DFSR changes for RPC, inbound and outbound threading, and other aspects. However, if a Win2008 server is partnered with a Win2003 server for DFSR, most of those improvements are disabled for backwards compatibility. An ideal environment is 100% Windows Server 2008 R2, but a Win2008-only is still a huge improvement. Windows Server 2003 should be phased out of use as quickly as possible as it has numerous "1G" design issues that were improved on with experience in later OS's. Windows Server 2008 R2 credit manager and update worker improvements are most efficient when all operating systems are homogenous. If you are replacing Win2003 servers with newer OS, do the hub servers first as the increased number of files will provide some benefits even when talking to 2003 spokes.
• Consider multiple hubs - If using a large number of branch servers in a hub-and-spoke topology, adding “subsidiary hub” servers will help reduce load on the main hubs.
  
  So for example, this configuration would cause more bottlenecking:

And this configuration would cause less bottlenecking:

• Increase staging quota - The larger the replicated folder staging quotas are for each server, the less often files must be restaged when replicating inbound changes. In a perfect world, staging quota would be configured to match the size of the data being replicated. Since this is typically impossible, it should be made as large as reasonably possible. It must always be configured to be at least as large as the combined size of the count of the files controlled by UpdateWorkerThreadCount+16 on Win2008 and Win2008 R2. Why 16? Because that is the number of outbound files that could be replicated simultaneously.

This means that by default on Win2008/Win2008 R2, quota must be as large as the 32 largest files. If UpdateWorkerThreadCount is increased to 32, it must be as large as the 48 largest files (32+16). If any smaller then staging can become blocked when all 32 files are being replicated inbound and 16 outbound, preventing further replication until that queue is cleared. Frequent 4202 and 4204 staging events are indications of an inappropriately configured staging quota, especially if no longer in the initial sync phase of setting up DFSR for the first time.

Source : DFSR
Catagory : None
Event ID : 4202
Type : Warning
Description :
The DFS Replication service has detected that the staging space in use for the replicated folder at local path c:\foo is above the high watermark. The service will attempt to delete the oldest staging files. Performance may be affected.   

Source : DFSR
Catagory : None
Event ID : 4204
Type : Information
Description :
The DFS Replication service has successfully deleted old staging files for the replicated folder at local path c:\foo. The staging space is now below the high watermark.

If you get 4206 staging events you have really not correctly sized your staging, as you are now blocking replication behind large files.

Event Type: Warning
Event Source: DFSR
Event Category: None
Event ID: 4206
Date: 4/4/2009
Time: 3:57:21 PM
User: N/A
Computer: SRV
Description:
The DFS Replication service failed to clean up old staging files for the replicated folder at local path c:\foo. The service might fail to replicate some large files and the replicated folder might get out of sync. The service will automatically retry staging space cleanup in 1 minutes. The service may start cleanup earlier if it detects some staging files have been unlocked.

If still using Win2003 R2, staging quota would need to be as large as the 9 largest files. And if using read-only replication on Windows Server 2008 R2, at least 16 or the size specified in UpdateWorkerThreadCount – after all, a read-only replicated folder has no outbound replication.

So to recap the staging quota minimum recommendations:

- Windows Server 2003 R2: 9 largest files
- Windows Server 2008: 32 largest files (default registry)
- Windows Server 2008 R2: 32 largest files (default registry)
- Windows Server 2008 R2 Read-Only: 16 largest files

If you want to find the 32 largest files in a replicated folder, here’s a sample PowerShell command:

Get-ChildItem <replicatedfolderpath> -recurse | Sort-Object length -descending | select-object -first 32 | ft name,length -wrap –auto

• Consider read-only - Deploy Windows Server 2008 R2 read-only replication when possible. If users are not supposed to change data, mark those replicated folders as read-only. A read-only server cannot originate data and will prevent unwanted replication or change orders from occurring outbound to other servers. Unwanted changes generate load and lead to data overwrites – which to fix you will need to replicate back out from backups, consuming time and replication resources.

• Latest QFE and SP - Always run the latest service pack for that OS, and the latest DFSR.EXE/DFSRS.EXE for that OS. There are also updates for NTFS and other components that DFSR relies on. Hotfixes have been released that remove performance bugs or make DFSR more reliable; a more reliable DFSR is naturally faster too. These are documented in KB968429 and KB958802 but the articles aren’t always perfectly up to date, so here’s a trick: If you want to find the latest DFSR service updates, use these three searches and look for the highest KB number in the results:

Win2008 R2: http://www.bing.com/search?q=%22windows+server+2008+r2%22+%22dfsrs.exe%22+kbqfe+site%3Asupport.microsoft.com&go=&form=QBRE

Win2008: http://www.bing.com/search?q=%22windows+server+2008%22+%22dfsrs.exe%22+kbqfe+site%3Asupport.microsoft.com&form=QBRE&qs=n

Win2003 R2: http://www.bing.com/search?q=%22windows+server+2003+r2%22+%22dfsr.exe%22+kbqfe+site%3Asupport.microsoft.com&form=QBRE&qs=n

Remember, Win2003 mainstream support ends July 13, 2010. That’s the end of non-security updates for that OS.

People ask me all the time why I take such a hard line on DFSR hotfixes. I ask in return “Why don’t you take such a hard line?” These fixes cost us a fortune, we’re not writing them for our health. And that goes for all other components too, not just DFSR. It’s an issue intrinsic to all software. DFSR is not less reliable than many other Windows components – after all, NTFS is considered an extremely reliable file system but that hasn’t stopped it from having 168 hotfixes in its lifetime; DFSR just has a passionate group of Support Engineers and developers here at MS that want you to have the best experience.

• Turn off RDC on fast connections with mostly smaller files - later testing (not addressed in the chart below) showed 3-4 times faster replication when using LAN-speed networks (i.e. 1GBb or faster) on Win2008 R2. This is because it was faster to send files in their totality than send deltas, when the files were smaller and more dynamic and the network was incredibly fast. The performance improvements were roughly twice as fast on Win2008 non-R2. This should absolutely not be done on WAN networks under 100 Mbit though as it will likely have a very negative affect.

• Consider and test anti-virus exclusions – Most anti-virus software has no concept of the data types that make up DFSR’s working files and database. Additionally, those file types are not executables and are therefore very unlikely to contain a useful malicious payload. If you are seeing slow performance within DFSR, test the following anti-virus file exclusions; if DFSR performs considerably better, contact your AV vendor for an updated version of their software and an explanation around the performance gap.

<drive>:\system volume information\DFSR\

   $db_normal$
   FileIDTable_* 
   SimilarityTable_*

<drive>:\system volume information\DFSR\database_<guid>\

   $db_dirty$
   Dfsr.db
   Fsr.chk
   *.log
   Fsr*.jrs
   Tmp.edb

<drive>:\system volume information\DFSR\config\

   *.xml

<drive>:\<replicated folder>\dfsrprivate\staging\*

   *.frx

This should be validated carefully; many anti-virus products allow exclusions to be set but then do not actually abide by them. For maximum performance, you would exclude scanning any replicated files at all, but this is obviously unfeasible for most customers.

• Pre-seed the data when setting up a new replicated folder- Pre-seeding - often referred to as "pre-staging" - data on servers can lead to huge performance gains during initial sync. This is especially useful when creating new branch office servers; if they are being built in the home office, they can be quickly pre-seeded with data then sent out to the field for replication of the change delta. See the following article for pre-seeding recommendations.

Going back to those same tests I showed earlier with 32 spokes replicating back to a single hub, note the average performance behavior when the data was perfectly pre-seeded:

Test	Spokes	Hubs	Topology	GB/node	Unique	RG	Tuned	Staging	Net	Time to sync
C9	32	1	Collect	1	Yes	32	Y	4GB	1Gbps	0:49:21
C11	32	1	Collect	1	Yes	32	Y	4GB	512Kbps	0:46:34
C12	32	1	Collect	1	Yes	32	Y	4GB	256Kbps	0:46:08
C13	32	1	Collect	1	Yes	32	Y	4GB	64Kbps	0:48:29

Even the 64Kbps frame relay connection was nearly as fast as the LAN! This is because no files had to be sent, only file hashes.

Note: do not use this table to predict replication times. It is designed to show behavior trends only.

• Go native Windows Server 2008 R2 – Not to beat a dead horse but the highest performance gains - including registry tuning and the greatly improved Credit Manager code - will be realized by using Windows Server 2008 R2. Win2003 R2 was first generation DFSR, Win2008 was second generation, and Win2008 R2 is third generation; if you are serious about performance you must get to 2008 R2.

Hardware tuning

• Use 64-bit OS with as much RAM as possible on hubs - DFSR can become bound by RAM availability on busy hub servers, especially when using the registry performance values above. There is absolutely no reason to run a 32-bit file server in this day and age, and with the coming of Windows Server 2008 R2, it’s no longer possible. For spoke servers that tend to have far less load, you can cut more corners of course; the ten-user sales team in Hicksville doesn’t need 16GB of RAM in their file server.

As a side note, customers periodically open cases to report “memory leaks” in DFSR. What we discuss is that DFSR intentionally caches as much RAM as it can get its hands on – really though, it’s the ESE (Jet) database doing this. So the idler other processes on a DFSR server are, the more memory a DFSR process will be able to gobble up. You can see the same behavior in LSASS’s database on DC’s.

• Use the fastest disk subsystem you can afford on hubs - Much of DFSR will be disk bound - especially in staging and RDC operations - so high disk throughput will dramatically lower bottlenecks; this is especially true on hub servers. As always, a disk queue length greater than 2 in PerfMon is in indication of an over-used or under-powered disk subsystem. Talk to your hardware vendors about the performance and cost differences of SATA, SCSI, and FC. Don’t forget about reliability too – I have a job here for life thanks to all the customers that use the least expensive, off-brand, no warranty, low parity, practically consumer-grade iSCSI products they can find. You get what you pay for and ultimately your users do not care about anything but their data. The OS is just a thing to make applications access files so that the business can make money. Someday the Linux desktop folks will figure this out and get some applications; then we may actually be in trouble here.

If using iSCSI, make sure you have redundant network paths to the disks, using multiple switches and NIC’s. We have had quite a few cases lately of no fault tolerance iSCSI configs that would go down for hours in the middle of DFSR updating the database and transaction logs, and the results were obviously not pretty.

• Use reliable networks - They don't necessarily have to be fast, but they do need to stay up. Many DFSR performance issues are caused by using old network card drivers, using malfunctioning "Scalable Network" (TCP offload, RSS, etc.) settings, or using defective WANs. Network card vendors release frequent driver updates to increase performance and resolve problems; just like Windows service packs, the drivers should be installed to improve reliability and performance. Companies often deploy cost saving WAN solutions (with VPN tunnels, frame relay circuits, etc.) that in the end cost the company more in lost productivity than they ever saved in monthly expense. DFSR - like all RPC applications - is sensitive to constant network instability.
• Review our performance tuning guides – For much more detail on squeezing performance out of your hardware, including network, storage, and the rest, review:
  
  
  • Performance Tuning Guidelines for Windows Server 2008 R2
  • Performance Tuning Guidelines for Windows Server 2008

And that’s it.

- Ned “fork” Pyle

Hi Everyone, Mike here. Today I am discussing the default processing behavior for Group Policy scripts. Microsoft changed the default behavior of Group Policy startup and logon scripts processing from synchronous to asynchronous starting with Windows Vista and Windows Server 2008. This behavior is the same in Windows 7 and Windows Server 2008 R2. I’ve recently read some confusion regarding this policy setting.

Computer Startup Scripts:

The default processing behavior for computer startup scripts is asynchronous. Asynchronous scripts processing is when computer startup scripts no longer wait for the previous script to complete before starting the next startup script. Versions of Windows prior to Windows Vista defaulted to synchronous processing. This behavior can be changed by reading from the following locations, in the following order.

Computer Preference

Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
ValueName: RunStartupScriptSync

Computer Policy

Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System
ValueName: RunStartupScriptSync

User Logon Scripts:

The default processing behavior for Group Policy logon scripts is asynchronous. However, this behavior can be changed by reading from the following locations, in order.

User Preference

Key: HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
ValueName: RunLogonScriptSync

Computer Preferences

Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
ValueName: RunLogonScriptSync

User Policy

Key: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\System
ValueName: RunLogonScriptSync

Computer Policy

Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System
ValueName: RunLogonScriptSync

User Logoff/Computer Shutdown:

User Logoff and Computer Shutdown Group Policy Scripts always process synchronously.

Customers are noticing timing issues with their computer startup scripts when using Windows 7 and Windows Server 2008 R2. Yes, this is true because the default behavior changed to run these scripts asynchronously. If you relied on the previous behavior, then you need to configure computer startup scripts to run synchronously. Unfortunately, this is where some confusion surfaces.

You’ll soon notice that Windows does not include a policy setting to enable computer startup scripts to run synchronously, or does it? Yes, it does; however, not so intuitive. I’ve had some GP administrators break out the nice search feature in the Group Policy Management Editor and search for the keyword synchronous—this is not going to return the information you are seeking. The policy setting to enable this does not include the word synchronous in the explain text or in any other portion of the policy. You need to configure the Run startup scripts asynchronously policy setting to disabled to force computer startup scripts to process synchronously, which is the default behavior prior to Windows 7 and Windows Vista.

Hopefully, this will clear up any confusion with how Windows 7 process computer startup scripts and explain how to restore the older behavior.

- Mike “Tips and Tricks is Ned’s kryptonite” Stephens

It’s your guest writer Herbert Mauerer again.

A very common AD disaster is an unexpected deletion or modification of objects. Unlike a bad football match or family meeting, you can prepare for that and make the crisis more bearable. In this blog, I will discuss best practices of Windows Server 2003 and 2008 forest level backup and restore. I will not discuss Windows 2000 as it’s at the end of its lifetime, and also not Windows 2008 R2 because we have a pretty good solution for object deletion without the need for backups: AD Recycle Bin.

AD Object deletions and unwanted modifications are often due to human error. Sometimes a bad script does that or a provisioning system, but then these are also created by humans. The common factor is the data loss. Now there is quite some complexity in the current KB article on AD object restores:

840001 How to restore deleted user accounts and their group memberships in Active Directory
http://support.microsoft.com/default.aspx?scid=kb;EN-US;840001

We basically follow method 1 in the article 840001, with a few tweaks and preparations:

1. Windows Server 2003 Service Pack 2 or newer and Windows Server 2003 Forest Functional level. This allows the restoration of links using LDIF files written by NTDSUTIL.EXE.
2. Preparing for the restore by converting the LEGACY group memberships to Link-Value Replication (LVR) group memberships.

Step 2 requires that you remove and re-add all group members. This is relatively easy using the Windows Server 2003 command line tools for querying and changing AD objects:

Dsget group CN=GroupX,OU=Groups,DC=contoso,DC=com /members | dsmod group CN=GroupX,OU=Groups,DC=contoso,DC=com /chmbr

This command gets the members of the group and replaces the members with the output, thus adding them as LVR members. This may mean some replication traffic if you apply this to many groups within a short time. However, since the removal and re-adding of members happened in a very short time, you should not see the member go away, as all changes should be replicated in the same replication job.

There are two problems with this approach:

1. The version of DSGET.EXE in Windows Server 2003 gets a maximum of 1500 members, and the command above would discard the rest. In one of my test environments, DSGET does not show any members. I have not seen both problems with the Windows Server 2008 or later version of DSGET.
2. If a group has lots of members, the execution of DSMOD.EXE can overflow the AD database version store, and adding the members would fail. So although the change of the membership type is a one-time action, it’s certainly a good idea to first export the member list to files first. I describe that below.
3. Use updated operating system file versions, details also below.

How to Convert Groups for LVR

You can determine the state of linked attribute values like group memberships from the replication meta-data on the object. When you run "repadmin.exe /showobjmeta", the link "Type" says LEGACY:

Type     Attribute       Last Mod Time   Originating DC  Loc.USN Org.USN     Ver  Distinguished Name
=======  ========        =============   =============== ========= ========= ===  ==========================================
LEGACY  member                                                                  CN=test-user1,OU=Test-OU,DC=contoso,DC=com
PRESENT   member   2008-09-16 18:22:29   HQ\contoso-DC1  175379684 175379684   1  CN=test-user2,OU=Test-OU,DC=contoso,DC=com

The line for LEGACY does not have data on the latest change all these values share that with the attribute meta-data listed in the first part of the output.

Hint: I use the "for" command in CMD for loops for direct execution on the command line. When you use CMD scripts, you need to change loop variables like "%f" to "%%f".

The steps are:

1. Freeze the group memberships, stop all group member management.

2. Export the groups in your domain:

Dsquery group dc=contoso,dc=com /limit 0 > grouplist.txt

Review the list and remove all built-in groups. You cannot remove all members there, and these groups usually don’t have many members to begin with. Also, there’s hope nobody deletes the memberships for these.

3. Get all group members on a Windows Vista member with RSAT or Windows Server 2008 computer:

For "delims=" %f in (grouplist.txt) do DSGET %f /members > groupmembers\%f

Important: The DNs of the groups must not contain characters ":\/*?". If you need to rename a group or an OU, it’s sufficient to change the DN.

4. Now we count the members:

In object counting, I use an older version of the MKS toolkit WC.EXE which counts lines, words and characters. You need a similar tool that counts lines in text files. You may have a favorite tool for that, or use a text editor that provides a line count, and use that to get the line count for the bigger output files.

For "delims=" %f in (grouplist.txt) do wc groupmembers \%f >> membercount.txt

The file membercount.txt has the count of members in the first column. Groups with more than 5000 members require special treatment. I suggest moving their member list into a folder "big-groups1".

5. For all other groups, execute:

Dir /b groupmembers >grouplist-small.txt
For /f "delims=" %f in (grouplist-small.txt) do dsmod group "%f" /chmbr < " groupmembers\%f"

Hint: "%f" is in double quotes here as "dir /b" does not print quotes, and we want to handle names with spaces properly.

So after this, the small groups are off our radar.

6. Now for the big groups:

My suggestion is to split the membership lists into multiple text files of approximately 5000 lines and put these sections into separate folders with the same file names (the DN of the group):

Dir /b big-groups1> big-groups1.txt
For /f "delims=" %f in (big-groups1.txt) do dsmod group "%f" /chmbr < " big-groups1\%f"

Dir /b big-groups2> big-groups2.txt
For /f "delims=" %f in (big-groups2.txt) do dsmod group "%f" /chmbr < " big-groups2\%f"

7. Verify the change by getting the group meta-data:

For "delims=" %f in (grouplist.txt) do repadmin /showobjmeta . %f > group-meta\%f

The output files in “group-meta” should have "LEGACY" only for other attributes.

Well, this was quite some work. The good news is that we don’t have to worry about the primary group… :-)

I suggest you test-drive this with test groups, for these tests it does not matter that the members are in LVR mode already. Then expand the work to smaller OU trees until you tackle the rest of the domain. One idea would be to split out parts of the group-list.txt so you impact a few groups only.

Suggested Fixes

After Windows Server 2003 Service Pack 2 was released, we had two problems affecting replication after a (authoritative) restore:

937855 After you restore deleted objects by performing an authoritative restoration on a Windows Server 2003-based domain controller, the linked attributes of some objects are not replicated to the other domain controllers
http://support.microsoft.com/default.aspx?scid=kb;EN-US;937855

943576 Active Directory objects may not be replicated from the restored server after an authoritative restore on a Windows Server 2003-based domain controller
http://support.microsoft.com/default.aspx?scid=kb;EN-US;943576

When you install Fix 943576 on the DCs you backup on a regular basis, you should not see any issues. In the long run, you should get this fix on all DCs in the enterprise. This problem is fixed in Windows 2008.

On top of that, there are also problem with the tool we use to restore the objects, NTDSUTIL.

Problems with object names containing extended characters:
886689 The Ntdsutil authoritative restore operation is not successful if the distinguished name path contains extended characters in Windows Server 2003 and in Windows 2000
http://support.microsoft.com/default.aspx?scid=kb;EN-US;886689

910823 Error message when you try to import .ldf files on a computer that is running Windows Server 2003 with Service Pack 1: "Add error on line LineNumber: No such object"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;910823

Problems with excessive links in the LDIF files:

951320 The ntdsutil.exe utility in Windows Server 2003 writes out too many links to .ldf files during an authoritative restore process
http://support.microsoft.com/default.aspx?scid=kb;EN-US;951320

The good news is that having fix 951320 on the DCs you backup often also takes care of problem 910823. Fix 951320 is included in Windows Server 2008 Service Pack 2.

There is a new problem with authoritative restores where objects are not treated correctly in replication. We have a corrective fix in NTDSUTIL for that:

974803 The domain controller runs slower or stops responding when the garbage collection process runs
http://support.microsoft.com/default.aspx?scid=kb;EN-US;974803

This last problem is not fixed in Windows Server 2008 yet. The problem cannot happen on Windows Server 2008 R2.

How to Avoid Accidental Deletion

Most of the AD objects recovery cases we see in our support work are because of accidental deletions. Typically, there are whole OUs with lots of user, groups and computer accounts affected. These objects have attributes that can’t be recovered by re-creation as you can’t get the same Sid again.

For this problem, Microsoft has implemented an option in the Windows Server 2008 Admin Tools, also available for Windows Vista using RSAT (Remote Server Administration Toolkit). This version has a check box "Protect object from accidental deletion” when “advanced view” is enabled. This flag is set automatically on all OUs you create with this admin tool.

When this is used, the OU itself and its parent get ACEs (Access Control Entries) that deny "Everyone" the permission to delete the object itself and child objects. This works with Domains hosted on Windows Server 2003. You can add the ACEs this way using your own tools and scripts and you will get the same effect.

Thus such a mass deletion should not happen by accident anymore, as you need to step through clearing the check box. Yes, the deletion might still happen from a script, but then it’s written to remove these ACEs and brings the problem to the next level, the script programmer. At this point, you can’t call it accidental anymore.

A Solution

For the object deletion scenario, Windows Server 2008 R2 offers the AD Recycle Bin. The forest needs to run in the Windows Server 2008 R2 forest mode and the feature needs to be enabled separately.

All objects that are deleted will have no attributes removed anymore, and linked attributes will have a third state to signal they are inactive pointing to a deleted object. When the object is reactivated, the link state will be switched to active again.

The graphical user interface for this facility is not in this release, but the groundwork is done, PowerShell is available, and there are scripts to help you perform the recovery.

Until you can enable this feature, the task is to be prepared for object deletions. This article will still be useful for unwanted object attribute modifications.

Herbert “by way of Germany” Mauerer

Ned here again. I’ve been involved with DFSR since its late beta days in 2005. Through good luck and a previous interest in FRS, I decided to take on DFSR as a focus area; when the support cases started rolling in I dealt with most of the issues. My expertise in DFSR is pretty much happenstance. :-)

Since that first week, I have had hundreds of customers ask me about configuring DFSR “one-way” replication. I’ve also had many customers try to invent their own read-only configurations for DFSR, with results ranging from pretty decent – using share permissions - to “OMG cleanse it with fire!!!” topology hacking.

Thankfully, Windows Server 2008 R2 takes care of this now in a more sophisticated way. I am going to discuss the architecture of read-only replication in Win 2008 R2, cover some best practices, and describe a few troubleshooting techniques. If you are looking for introductory information about why you would use read-only DFSR or how to configure it step by step, I highly recommend the DFSR development teams blog posts and help file:

• Read-only replicated folders on Windows Server 2008 R2
• Configuring a read-only replicated folder on Windows Server 2008 R2
• The Windows DFSR Help (available in DFSMGMT.MSC)

On with the show.

Architecture

Read-only (RO) replication refers to the concept of “one-way” replication, where a downstream RO replicated folder is incapable of sending files outbound. It also prevents the creation, modification, or deletion of files in the RO content set locally. RO replicated folders sets can exist on a server that also contains other Read-Write (RW) replicated folders - the boundary of RW or RO is the replicated folder itself on a given server. This new system is ideal for static data that end users are never supposed to modify, such as application installation points, corporate procedural files, centralized backup locations etc.

RO replication relies on the new mini filter file system driver DFSRRO.SYS. This IO-blocking driver is at an altitude below most common filters, meaning that other drivers will have to abide by DFSRRO’s decisions – including anti-virus software. DFSRRO.SYS is implemented at the “FSFilter Content Screener” level:

Because the filter driver blocks write IOs, any applications or even many kernel-mode drivers that attempt to save data in the RO folder will receive "Access is denied" (0x5 ERROR_ACCESS_DENIED). This also means that if an application opens files with a WRITE disposition unnecessarily - even if the application only intends to read data – DFSR will return access denied. You can read more about CreateFile dispositions here and file system drivers here.

A Replication Group containing an RO replicated folder must still contain both inbound and outbound connections. Since a replication group could potentially maintain a mix of RO/RW, and because an RO can change to an RW dynamically, both connections must exist at all times. RO replicated folders can only directly replicate from RW replicated folders. An RO replicated folder cannot source from another RO replicated folder. Also, an RW cannot transitively replicate through an RO.

So these configurations are good:

And these configurations are bad:

Note: my arrows indicate direction of replication, not a single DFSR connection object! Don’t make me come over there.

An RW replicated folder can be converted to an RO replicated folder (and back) “on the fly”. Converting will cause a non-authoritative sync to occur on the replicated folder for the server being altered. This is done to ensure that the contents are up to date with the partner before data can flow inbound/outbound. This gives you DFSR event log messages 4620 or 4622:

Log Name:      DFS Replication
Source:        DFSR
Date:          2/22/2010 4:27:29 PM
Event ID:      4620
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      2008r2-f-01.contoso.com
Description:
The DFS Replication service has detected that ken, which used to be a read-write replicated folder has now been configured to be a read-only replicated folder. The DFS Replication service will not allow any changes to be made to the contents of this replicated folder. Any updates occurring on other read-write replicated folders will be replicated in and applied to the contents of this read-only replicated folder.
Additional Information:
Replicated Folder Name: ken
Replicated Folder Root: c:\ken
Replicated Folder ID: 171E6A7E-6182-496D-8277-1797FF18C05C
Replication Group Name: clu2
Replication Group ID: 8A645529-FE74-4430-9ECD-D1BDC0BA800A
Member ID: 0EAD46B4-A442-48EA-97F6-109714968E40

==========================

Log Name:      DFS Replication
Source:        DFSR
Date:          2/22/2010 4:31:31 PM
Event ID:      4622
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      2008r2-f-01.contoso.com
Description:
The DFS Replication service has detected that ken, which used to be a read-only replicated folder has now been configured to be a read-write replicated folder. The DFS Replication service will now allow any changes to be made to the contents of this replicated folder. Any updates occurring on this read-write replicated folder will be replicated out and applied to the contents of other replicated folders on other members.
Additional Information:
Replicated Folder Name: ken
Replicated Folder Root: c:\ken
Replicated Folder ID: 171E6A7E-6182-496D-8277-1797FF18C05C
Replication Group Name: clu2
Replication Group ID: 8A645529-FE74-4430-9ECD-D1BDC0BA800A
Member ID: 0EAD46B4-A442-48EA-97F6-109714968E40

These events would be followed by 4102 and 4104. There is also a new event that will write when someone tries to restore over a read-only RF:

Log Name: DFS Replication
Source: DFSR
Date: 11/21/2008 11:29:14 PM
Event ID: 1112
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: WIN7-6946-02.southridgevideo.com
Description:
The DFS Replication service failed a restore request. This could happen if an
attempt was made to restore the contents of a read-only replicated folder
authoritatively. Read-only replicated folders should only be restored
non-authoritatively. Authoritative restores should be performed only on read-write
replicated folders.

Additional Information:
Replicated Folder Name: foo
Replicated Folder ID: 02436A8B-D620-4D83-980A-657E2603FBA0

Finally, there are also new debug entries for RO, under the module ID’s BFLT, FREP, RFLT ad well as updated in VLMG and CCTX. If that makes no sense, review this insomnia cure.

Best Practices and Answers to Common Questions

Note: some of these will be repeats from this and other blog posts. I do that because no matter how many times I talk about certain best practices, people continue to ignore me. :-)

1. Before deploying read-only folders you must test! Your antivirus software, your backup software, and especially your line of business applications that open files from the replicated folder may all react badly to RO. There are no known MS applications that have any problems.

2. Converting a replicated folder from RW to RO (or the reverse) will cause a non-authoritative sync to occur on the replicated folder. Plan accordingly.

3. DFSR read-only is not a panacea! If users can access files on the read-write folders, RO will not prevent anything. If you are trying to stop administrators from changing data, get better administrators.

4. Tell your users! Unless you want to have lots of help desk tickets about “I get access denied errors”, make sure your users understand that you are marking previously writable data as read-only. Not that they will read the email where you told them this, but at least you tried.

5. You cannot authoritatively restore data to an RO folder. Duh. Getting backups from there is fine and highly recommended.

6. Don’t bother trying to circumvent RO. Even when running as SYSTEM, you will not be able to write files into an RO-protected folder. If you don’t want RO protection, don’t run RO. Make your changes on RW folders.

7. RO can replicate inbound from Windows Server 2003 R2 and Windows 2008. You will need to have extended the AD schema to at least Windows Server 2008 though.

8. An RO replicated folder can have more than one RW partner replicating inbound, as long as those RW servers have more than just the transitive RO connection.

Troubleshooting

There aren’t many new steps to troubleshooting read-only replication compared to classic read-write – make sure you have two connections between every server, in both directions, just like RW. Make sure you have the latest hotfixes. Review everything I described above about topology – most issues in RO have been admin-generated so far. :-)

The one thing you can add to your test environment troubleshooting is FLTMC.EXE. This tool allows you to dynamically load and unload the DFSRRO.SYS filter driver for testing purposes, using the LOAD and UNLOAD commands. This way if you are testing an application that mysteriously fails when talking to a read-only folder, you have a way to temporarily allow writing to the folder and confirm the application is trying to write data without the need to completely remove RO. There is a lazy sync worker thread that removes changes and replicates down changes from the RW server after the driver is reloaded, so that the file system does not get irrevocably out of sync. Remember to load the driver back up when done testing!

Wrap Up

It’s one more good reason to ditch FRS and deploy DFSR.

And a funny story about that previous interest in FRS I mentioned: most people in MS support really dislike FRS for all of its inadequacies, but I owe it a debt of gratitude. When I was in my hiring interview years ago, one of the questions I got asked was “Can you describe in as much detail as possible how FRS processes files?” The panel thought that was a real stumper. So I drew this on the whiteboard from memory and explained it:

And I got the job. So thanks FRS, I owe you this gig.

Until next time.

Ned “r” Pyle

Ned here again. As you begin the process of moving off your older operating systems to Windows 7 you may end up using USMT 4.0. With its powerful capabilities and intended audience of IT professionals, the tool can be unforgiving and often gives inscrutable messages. Today I’ll cover a few of the most common but unclear errors people see when they first run the tool.

Return code 27: Invalid store path

You ran a scanstate and now you’re trying to restore the data to a new target computer. You run:

Loadstate.exe c:\store\usmt /auto

And…you get a store error. Examining the loadstate log shows:

[0x000000] USMT Started at 2010/01/31:15:53:58.796
[0x000000] Command line: loadstate c:\store\usmt /auto
[0x000000] Activity: 'MIGACTIVITY_COMMAND_LINE_PROCESSING'
[0x000000] Failed.[gle=0x00000002]
[0x000000]   An error occurred processing the command line.
  Invalid store path; check the store parameter and/or file system permissions[gle=0x00000002]
[0x000000] USMT Completed at 2010/01/31:15:53:58.812[gle=0x00000002]
[0x000000] Entering MigShutdown method
[0x000000] Leaving MigShutdown method

But you look and that path definitely exists with a valid store MIG file inside it.

This is an easy mistake to make; when you specify a store path during scanstate, USMT automatically makes a sub-folder called “USMT” that contains your data. Admins often assume they need to specify the full path - you don’t. Just provide a path to the same folder level that was used in the scanstate. So in the above case:

Loadstate.exe c:\store /auto

Return code 28: Unable to find a script file specified by /i

You are running a scan or load with a perfectly valid command-line, but every time you start it:

You get error 28. You examine the directory where scanstate.exe is located and the XML files are definitely there. You examine the scanstate.log and see:

[0x000000] USMT Started at 2010/01/31:15:59:28.118
[0x000000] Command line: c:\temp\usmt\scanstate.exe c:\store /v:5 /i:migdocs.xml /i:migapp.xml /o
[0x000000] Activity: 'MIGACTIVITY_COMMAND_LINE_PROCESSING'
[0x000000] Script file is not present: 'C:\users\std\migdocs.xml'[gle=0x00000002]
[0x000000] Failed.[gle=0x00000002]
[0x000000]   An error occurred processing the command line.
  Unable to find a script file specified by /i[gle=0x00000002]
[0x000000] USMT Completed at 2010/01/31:15:59:28.149[gle=0x00000002]
[0x000000] Entering MigShutdown method
[0x000000] Leaving MigShutdown method

This error is returned because both scanstate.exe and loadstate.exe assume that the working folder is the current directory -- where the command is execute -- and not the directory wherein the executable is located. Since the XML files do not exist in my profile -- the current directory -- the command fails. To fix the issue, first switch to the directory containing your USMT files using the CD command, or specify full paths to the XML files with each /i.

Return code 26: Software malfunction or Unknown exception

Hooboy, that is a really helpful error. You examine the scanstate log and see:

[0x000000] USMT Started at 2010/01/31:16:11:54.383
[0x000000] Command line: scanstate c:\store /o /i:miguser.xml /i:migapp.xml /v:5
[0x000000] Activity: 'MIGACTIVITY_COMMAND_LINE_PROCESSING'
<snip>
[0x000000] Entering MigStartupOnline method
[0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
[0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
[0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
[0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
[0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
[0x0803ac] Initializing online WinNT platform (Read/Write)
[0x080000] Platform is not using admin privileges

Aha! As you hopefully know from your reading, USMT requires you to be an Administrator to run the scanstate with full functionality or to run loadstate at all. The problem here is that while you are logged on with an account that is a member of the Administrators group, User Account Control is turned on and this CMD prompt is not elevated. That means that your user account shows up as an Administrator but can’t do anything. If you were to lookup that 0x514 error you’d see it means “Not all privileges or groups referenced are assigned to the caller”. Indeed, they are not.

Elevate your CMD prompt, use an account that bypasses UAC, or turn off UAC – the issue will go away.

User running the scanstate is the only profile ever migrated

Why. Do. You. Ignore. My. Command. Line?!? ARRRRGGGHHHH!!!

This one is just unfair. As I mentioned above, you must be an administrator to run loadstate – it gives you a very specific error otherwise:

But if you run scanstate as a user that is not a member of the Administrators group, you get no errors. Goo gets gathered up; everything seems fine.

Except that nothing you put on the command-line for /ui is considered at all, and if you were to closely examine the store data, you’d find tons of operating system settings missing. I have two different accounts on this computer that contain “ned” in some way, that examination should have gotten them both.

Yes, this is by design -- more information is tucked away here. There is almost no good reason to run any USMT command as a standard user; just get into the habit of using an (elevated) administrator.

Conclusion

Migrating your computers to Windows 7 can be a very interesting exercise. Hopefully this blog post makes things a bit easier.

Oh - and why did I only say “may end up using USMT 4.0” at the start of this post? Because you have other options:

• MDT 2010 lite touch (uses USMT under a GUI)
• MDT 2010 zero touch with System Center Configuration Manager 2007 (ditto)
• Upgrade in place
• 3^rd parties

Until next time.

- Ned “The Riddler” Pyle

Ned here again. Because Windows XP cannot be in-place upgraded to Windows 7 and because XP has ruled supreme for longer than most IT staffers’ careers, everyone who managed to avoid USMT are now coming out of the woodwork. Perhaps the most asked question is “I am trying to block X from being migrated to the new computer but it always comes over anyway: what step am I missing?”

USMT 4.0 is incredibly flexible and powerful, to the point where its complexity can make tasks tricky. Exclusions are a case in point, and today I’ll go through the three common mistakes that prevent exclusions from working.

Consider the case of the frmcache that wouldn’t die. The frmcache.dat file is the part of Microsoft Outlook that stores forms and apparently is rather fragile. One of our customers found that a third party application had modified the frmcache and the cache no longer worked when opened by a later version of Outlook. Since the customer was migrating to new computers running Outlook 2007, they just wanted to block the frmcache.dat from migrating and let Outlook create a fresh one automagically. They created a custom XML file with an exclude rule.

Except that it didn’t work; the file kept migrating. They suspected – correctly – that USMT was still migrating the file because their custom rule was wrong. After a brief dig, they found it being migrated by the migapp.xml. For good reason they didn’t want to modify that included XML file directly. Why wasn’t the custom XML file working and how could you run into the same exact issue?

Common cause 1: Your XML uses exclude and not unconditionalExclude

The rules around USMT conflict and precedence handling are complex and often subtle – read more here if you want a cure for insomnia. Generally speaking, though, if you have some reason to specifically exclude a file or a registry setting there is no circumstance where you ever want it to get included. This means you should use <unconditionalExclude> in your custom XML. This special rule ignores any precedence or inclusion conflicts.

Here is a sample rule that will absolutely not, no matter what, allow any MP3 files to be migrated to a new destination computer.

<?xml version="1.0" encoding="utf-8" ?>
<migration urlid="http://www.microsoft.com/migration/1.0/migxmlext/excludefiles">
<component context="UserAndSystem" type="Documents">
  <displayName>NedSample</displayName>
  <role role="Data">
    <rules>
      <unconditionalExclude>
        <objectSet>
          <script>MigXmlHelper.GenerateDrivePatterns ("* [*.mp3]", "Fixed")</script>
        </objectSet>
      </unconditionalExclude>
    </rules>
  </role>
</component>
</migration>

Obviously, you need to be very careful about unconditional excludes – setting it for .DOC* files will not endear you to users. Exclusions should strive have as specific a path as possible, unless you know without a doubt that a file has no business being migrated. MP3’s are a good example of those to most businesses, if only for liability reasons!

Common cause 2: You have set unconditionalExclude in the wrong context

Within unconditionalExclude there is still a delineation of context for User, System and UserAndSystem. The context tag tells USMT when to care about a section of XML – if gathering a user’s profile it will use the rules for User and UserAndSystem. If gathering the system (aka computer’s) profile, it will use rules for System and UserAndSystem.

Here is a sample custom rule XML file that will prevent a specific file pattern (frmcache.dat) from being migrated from any user's local appdata profile (%CSIDL_LOCAL_APPDATA%) where the file exists in a specific path (\Microsoft\FORMS).

<?xml version="1.0" encoding="UTF-8"?>
<migration urlid="http://www.microsoft.com/migration/1.0/migxmlext/test">
  <component type="Documents" context="User">
    <displayName>Block migrating frmcache.dat</displayName>
    <role role="Data">
      <rules>
        <unconditionalExclude>
          <objectSet>
            <pattern type="File"> %CSIDL_LOCAL_APPDATA%\Microsoft\FORMS [frmcache.dat] </pattern>
          </objectSet>
        </unconditionalExclude>
      </rules>
    </role>
  </component>
</migration>

Common cause 3: You have not correctly provided the case for unconditionalExclude

Finally – and most insidiously - USMT XML tag handling is case-sensitive. This is not just for includes and excludes, so just get in the habit. If you set "unconditionalexclude" it will not work and neither will "Unconditionalexclude" or "UnconditionalExclude". But if you set "unconditionalExclude" your rule will process.

OMGXMLH8U!!!111elevenz

If you want to avoid these sorts of typo problems in the future, take a look at this previous blog post.

Until next time.

- Ned “tweener” Pyle

Hi all, Ned here again. I’ve seen an emerging issue with USMT that I need to address while our TechNet documentation is updated. If you are using USMT 4.0 for migrations from XP, this is required reading to avoid some very gnarly problems when using the /SF switch in loadstate. Onward.

Background

When reading our TechNet USMT documentation, you may find occasional examples that have the loadstate.exe using a switch called /SF. If you look at the USMT command-line documentation though, there is no mention of this switch at all. And in the great majority of our examples, the switch is never used. Running loadstate.exe /? shows:

/sf Restores shell folder redirection

What the what?

Shell folder redirection is an option within Windows Explorer to change the default locations of the built-in shell folders, such as My Documents. It’s typically configured through Folder Redirection group policy. In the grand scheme of hundreds of millions of business computers though, it’s strictly in the minority and rarely used compared to the defaults – especially in an unmanaged fashion; it’s not easy for the typical end user to find.

The USMT /SF option was designed to migrate the shell folder settings. Unfortunately, when loadstate runs it incorrectly updates the global registry path:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders

This happens with both online and offline migrations.

Symptoms

If you use the /SF switch to migrate from an XP computer to Windows 7 or Vista, you see the following issues:

• Attempting to open the Administrative Tools folder returns error:

"Location is not available
C:\Users\Public\Start Menu\Programs\Administrative Tools refers to a location that is unavailable. If could be on a hard drive on this computer, or on a network. Check to make sure the disk is properly inserted, or that you are connected to the Internet or your network, and then try again. If it still cannot be located, the information might have been moved to a different location."

• The “search programs and files” box does not allow you search for any files. For example, when typing in “regedit” it returns error:

“search:query=regedit
Windows cannot find ‘search:query=regedit’. Make sure you typed the name correctly, and then try again.”

• The “All Programs” menu only shows "Internet Explorer", "Accessories", "Maintenance", and "Startup".

There may be more, these are just the ones that I have been able to find or reproduce from customer feedback. Use that Comments section!

Workaround

There are a few ways to resolve these issues:

1. If you are not using redirected folders in Windows XP source computers, stop using the /SF switch on loadstate.

2. If you want to use /SF or are unsure if redirected shell folders are being configured on clients, you can add the following sample batch file to your steps. After running loadstate.exe, you would then run these in a CMD batch file:

I’ve attached this as a TXT file as well, since the formatting here is not pretty.

And that’s that. Please don’t ask about a bug fix or ETA on TechNet being updated, I’ve got nothing for you yet. This article is designed to get you out of the woods.

- Ned “documenting the undocumentable” Pyle

Hey everyone, Mike here again to discuss an interesting feature I learned about in Windows 7. Many Microsoft customers deploy Roaming User Profiles. In fact, many combine Roaming User profiles and Folder Redirection to get the best experience possible. However, one of the drawbacks with Roaming User profiles is the user must logoff before their settings are uploaded to the server. Folder redirection solves this problem for any of the known folders within the user profile namespace, such as Documents, Music, or Downloads—the data is highly available without requiring a user logoff. Now, if this could only occur with user’s registry settings.

Windows 7 solves this problem by allowing the User Profile service to upload the user’s registry settings of a Roaming User profile while the user remains logged on to the computer, or terminal services session (provided the session is hosted on Windows Server 2008 R2).

You enabled this feature using Group Policy. The policy setting Background upload of a roaming user profile’s registry file while user is logged on, is located under Computer Configuration\Policies\Administrative Templates\System\User Profiles. The policy setting offers two configuration settings: scheduled or interval. The scheduled method allows you to configure a time of day (represented on 24 hour time) at which to upload the user’s registry settings. The interval method allow you to choose a specific interval (represented in hours) at which to upload the user’s registry settings. This method accepts an interval range between 1 and 720 inclusively. Both settings include a random delay that does not exceed one hour.

Figure 1 Background upload policy setting in Windows 7

Background uploading only occurs with Roaming User profile. Also, background uploading does not alter uploading the entire profile when the user logs off. It is important to remember that background uploading only uploads the user’s registry settings (ntuser.dat).

Now those subtle changes made in the registry can be uploaded to the server while the user remains logged on. Pretty cool.

Mike “the other Mike Stephens” Stephens