March, 2010

Hi all, Ned here again. There are a number of ways that DFSR can be tuned for better performance. This article will go through these configurations and explain the caveats. Even if you cannot deploy Windows Server 2008 R2 - for the absolute best performance - you can at least remove common bottlenecks from your older environments. If you are really serious about performance in higher node count DFSR environments though, Win2008 R2’s 3^rd generation DFSR is the answer.

If you’ve been following DFSR for the past few years, you already know about some improvements that were made to performance and scalability starting in Windows Server 2008:

But there’s more you can do, especially in 2008 R2.

Registry tuning

All registry values are REG_DWORD (and in the explanations below, are always in decimal). All registry tuning for DFSR in Win2008 and Win2008 R2 is made here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DFSR\Parameters\Settings

A restart of the DFSR service is required for the settings to take effect, but a reboot is not required. The list below is not complete, but instead covers the important values for performance. Do not assume that setting a value to the max will make it faster; some settings have a practical limitation before other bottlenecks make higher values irrelevant.

Important Note: None of these registry settings apply to Windows Server 2003 R2.

AsyncIoMaxBufferSizeBytes
Default value: 2097152
Possible values: 1048576, 2097152, 4194304, 8388608
Tested high performance value: 8388608
Set on: All DFSR nodes

RpcFileBufferSize
Default value: 262144
Possible values: 262144, 524288
Tested high performance value: 524288
Set on: All DFSR nodes

StagingThreadCount
Default value: 6
(Win2008 R2 only; cannot be changed on Win2008)
Possible values: 4-16
Tested high performance value: 8
Set on: All DFSR nodes. Setting to 16 may generate too much disk IO to be useful.

TotalCreditsMaxCount
Default value: 1024
Possible values: 256-4096
Tested high performance value: 4096
Set on: All DFSR nodes that are generally inbound replicating (so hubs if doing data collection, branches if doing data distribution, all servers if using no specific replication flow)

UpdateWorkerThreadCount
Default value: 16
Possible values (Win2008): 4-32
Possible values (Win2008 R2): 4-63*
Tested high performance value: 32

Set on: All DFSR nodes that are generally inbound replicating (so hubs if doing data collection, branches if doing data distribution, all servers if using no specific replication flow. The number being raised here is only valuable when replicating in from more servers than the value. I.e. if replicating in 32 servers, set to 32. If replicating in 45 servers set to 45.

*Important note: The actual top limit is 64. We have found that under certain circumstances though, setting to 64 can cause a deadlock that prevents DFSR replication altogether. If you exceed the maximum tested value of 32, set to 63 or lower. Do not set to 64 ever. The 32 max limit is recommended because we tested it carefully, and higher values were not rigorously tested. If you set this value to 64, periodically replication will stop working, the dfsrdiag replstate command hangs and does not return results, and the dfsrdiag backlog command hangs and does not return results.

 

When using all the above registry tuning on Windows Server 2008 R2, testing revealed that initial sync replication time was sometimes twice as fast compared to no registry settings in place. This was using 32 servers replicating a "data collection" topology to a single hub over thirty-two non-LAN networks with 32 RG's containing unique branch office data. The slower the network, the better the relative performance averaged:

On Windows Server 2008 the same registry values showed considerably less performance improvement; this is partly due to additional service improvements made to DFSR in Win2008 R2, especially around the Credit Manager. Just like your phone, “3G” DFSR is going to work better than older models…

Note: do not use this table to predict replication times. It is designed to show behavior trends only!

Topology tuning

Even if you are not using Windows Server 2008 R2 there are plenty of other factors to fast replication. Some of these I’ve talked about before, some are new. All are important:

• Minimize mixing of Win2003 and Win2008/Win2008 R2 - Windows Server 2008 introduced significant DFSR changes for RPC, inbound and outbound threading, and other aspects. However, if a Win2008 server is partnered with a Win2003 server for DFSR, most of those improvements are disabled for backwards compatibility. An ideal environment is 100% Windows Server 2008 R2, but a Win2008-only is still a huge improvement. Windows Server 2003 should be phased out of use as quickly as possible as it has numerous "1G" design issues that were improved on with experience in later OS's. Windows Server 2008 R2 credit manager and update worker improvements are most efficient when all operating systems are homogenous. If you are replacing Win2003 servers with newer OS, do the hub servers first as the increased number of files will provide some benefits even when talking to 2003 spokes.
• Consider multiple hubs - If using a large number of branch servers in a hub-and-spoke topology, adding “subsidiary hub” servers will help reduce load on the main hubs.
  
  So for example, this configuration would cause more bottlenecking:

And this configuration would cause less bottlenecking:

• Increase staging quota - The larger the replicated folder staging quotas are for each server, the less often files must be restaged when replicating inbound changes. In a perfect world, staging quota would be configured to match the size of the data being replicated. Since this is typically impossible, it should be made as large as reasonably possible. It must always be configured to be at least as large as the combined size of the count of the files controlled by UpdateWorkerThreadCount+16 on Win2008 and Win2008 R2. Why 16? Because that is the number of outbound files that could be replicated simultaneously.

This means that by default on Win2008/Win2008 R2, quota must be as large as the 32 largest files. If UpdateWorkerThreadCount is increased to 32, it must be as large as the 48 largest files (32+16). If any smaller then staging can become blocked when all 32 files are being replicated inbound and 16 outbound, preventing further replication until that queue is cleared. Frequent 4202 and 4204 staging events are indications of an inappropriately configured staging quota, especially if no longer in the initial sync phase of setting up DFSR for the first time.

Source : DFSR
Catagory : None
Event ID : 4202
Type : Warning
Description :
The DFS Replication service has detected that the staging space in use for the replicated folder at local path c:\foo is above the high watermark. The service will attempt to delete the oldest staging files. Performance may be affected.   

Source : DFSR
Catagory : None
Event ID : 4204
Type : Information
Description :
The DFS Replication service has successfully deleted old staging files for the replicated folder at local path c:\foo. The staging space is now below the high watermark.

If you get 4206 staging events you have really not correctly sized your staging, as you are now blocking replication behind large files.

Event Type: Warning
Event Source: DFSR
Event Category: None
Event ID: 4206
Date: 4/4/2009
Time: 3:57:21 PM
User: N/A
Computer: SRV
Description:
The DFS Replication service failed to clean up old staging files for the replicated folder at local path c:\foo. The service might fail to replicate some large files and the replicated folder might get out of sync. The service will automatically retry staging space cleanup in 1 minutes. The service may start cleanup earlier if it detects some staging files have been unlocked.

If still using Win2003 R2, staging quota would need to be as large as the 9 largest files. And if using read-only replication on Windows Server 2008 R2, at least 16 or the size specified in UpdateWorkerThreadCount – after all, a read-only replicated folder has no outbound replication.

So to recap the staging quota minimum recommendations:

- Windows Server 2003 R2: 9 largest files
- Windows Server 2008: 32 largest files (default registry)
- Windows Server 2008 R2: 32 largest files (default registry)
- Windows Server 2008 R2 Read-Only: 16 largest files

If you want to find the 32 largest files in a replicated folder, here’s a sample PowerShell command:

Get-ChildItem <replicatedfolderpath> -recurse | Sort-Object length -descending | select-object -first 32 | ft name,length -wrap –auto

• Consider read-only - Deploy Windows Server 2008 R2 read-only replication when possible. If users are not supposed to change data, mark those replicated folders as read-only. A read-only server cannot originate data and will prevent unwanted replication or change orders from occurring outbound to other servers. Unwanted changes generate load and lead to data overwrites – which to fix you will need to replicate back out from backups, consuming time and replication resources.

• Latest QFE and SP - Always run the latest service pack for that OS, and the latest DFSR.EXE/DFSRS.EXE for that OS. There are also updates for NTFS and other components that DFSR relies on. Hotfixes have been released that remove performance bugs or make DFSR more reliable; a more reliable DFSR is naturally faster too. These are documented in KB968429 and KB958802 but the articles aren’t always perfectly up to date, so here’s a trick: If you want to find the latest DFSR service updates, use these three searches and look for the highest KB number in the results:

Win2008 R2: http://www.bing.com/search?q=%22windows+server+2008+r2%22+%22dfsrs.exe%22+kbqfe+site%3Asupport.microsoft.com&go=&form=QBRE

Win2008: http://www.bing.com/search?q=%22windows+server+2008%22+%22dfsrs.exe%22+kbqfe+site%3Asupport.microsoft.com&form=QBRE&qs=n

Win2003 R2: http://www.bing.com/search?q=%22windows+server+2003+r2%22+%22dfsr.exe%22+kbqfe+site%3Asupport.microsoft.com&form=QBRE&qs=n

Remember, Win2003 mainstream support ends July 13, 2010. That’s the end of non-security updates for that OS.

People ask me all the time why I take such a hard line on DFSR hotfixes. I ask in return “Why don’t you take such a hard line?” These fixes cost us a fortune, we’re not writing them for our health. And that goes for all other components too, not just DFSR. It’s an issue intrinsic to all software. DFSR is not less reliable than many other Windows components – after all, NTFS is considered an extremely reliable file system but that hasn’t stopped it from having 168 hotfixes in its lifetime; DFSR just has a passionate group of Support Engineers and developers here at MS that want you to have the best experience.

• Turn off RDC on fast connections with mostly smaller files - later testing (not addressed in the chart below) showed 3-4 times faster replication when using LAN-speed networks (i.e. 1GBb or faster) on Win2008 R2. This is because it was faster to send files in their totality than send deltas, when the files were smaller and more dynamic and the network was incredibly fast. The performance improvements were roughly twice as fast on Win2008 non-R2. This should absolutely not be done on WAN networks under 100 Mbit though as it will likely have a very negative affect.

• Consider and test anti-virus exclusions – Most anti-virus software has no concept of the data types that make up DFSR’s working files and database. Additionally, those file types are not executables and are therefore very unlikely to contain a useful malicious payload. If you are seeing slow performance within DFSR, test the following anti-virus file exclusions; if DFSR performs considerably better, contact your AV vendor for an updated version of their software and an explanation around the performance gap.

<drive>:\system volume information\DFSR\

   $db_normal$
   FileIDTable_* 
   SimilarityTable_*

<drive>:\system volume information\DFSR\database_<guid>\

   $db_dirty$
   Dfsr.db
   Fsr.chk
   *.log
   Fsr*.jrs
   Tmp.edb

<drive>:\system volume information\DFSR\config\

   *.xml

<drive>:\<replicated folder>\dfsrprivate\staging\*

   *.frx

This should be validated carefully; many anti-virus products allow exclusions to be set but then do not actually abide by them. For maximum performance, you would exclude scanning any replicated files at all, but this is obviously unfeasible for most customers.

• Pre-seed the data when setting up a new replicated folder- Pre-seeding - often referred to as "pre-staging" - data on servers can lead to huge performance gains during initial sync. This is especially useful when creating new branch office servers; if they are being built in the home office, they can be quickly pre-seeded with data then sent out to the field for replication of the change delta. See the following article for pre-seeding recommendations.

Going back to those same tests I showed earlier with 32 spokes replicating back to a single hub, note the average performance behavior when the data was perfectly pre-seeded:

Test	Spokes	Hubs	Topology	GB/node	Unique	RG	Tuned	Staging	Net	Time to sync
C9	32	1	Collect	1	Yes	32	Y	4GB	1Gbps	0:49:21
C11	32	1	Collect	1	Yes	32	Y	4GB	512Kbps	0:46:34
C12	32	1	Collect	1	Yes	32	Y	4GB	256Kbps	0:46:08
C13	32	1	Collect	1	Yes	32	Y	4GB	64Kbps	0:48:29

Even the 64Kbps frame relay connection was nearly as fast as the LAN! This is because no files had to be sent, only file hashes.

Note: do not use this table to predict replication times. It is designed to show behavior trends only.

• Go native Windows Server 2008 R2 – Not to beat a dead horse but the highest performance gains - including registry tuning and the greatly improved Credit Manager code - will be realized by using Windows Server 2008 R2. Win2003 R2 was first generation DFSR, Win2008 was second generation, and Win2008 R2 is third generation; if you are serious about performance you must get to 2008 R2.

Hardware tuning

• Use 64-bit OS with as much RAM as possible on hubs - DFSR can become bound by RAM availability on busy hub servers, especially when using the registry performance values above. There is absolutely no reason to run a 32-bit file server in this day and age, and with the coming of Windows Server 2008 R2, it’s no longer possible. For spoke servers that tend to have far less load, you can cut more corners of course; the ten-user sales team in Hicksville doesn’t need 16GB of RAM in their file server.

As a side note, customers periodically open cases to report “memory leaks” in DFSR. What we discuss is that DFSR intentionally caches as much RAM as it can get its hands on – really though, it’s the ESE (Jet) database doing this. So the idler other processes on a DFSR server are, the more memory a DFSR process will be able to gobble up. You can see the same behavior in LSASS’s database on DC’s.

• Use the fastest disk subsystem you can afford on hubs - Much of DFSR will be disk bound - especially in staging and RDC operations - so high disk throughput will dramatically lower bottlenecks; this is especially true on hub servers. As always, a disk queue length greater than 2 in PerfMon is in indication of an over-used or under-powered disk subsystem. Talk to your hardware vendors about the performance and cost differences of SATA, SCSI, and FC. Don’t forget about reliability too – I have a job here for life thanks to all the customers that use the least expensive, off-brand, no warranty, low parity, practically consumer-grade iSCSI products they can find. You get what you pay for and ultimately your users do not care about anything but their data. The OS is just a thing to make applications access files so that the business can make money. Someday the Linux desktop folks will figure this out and get some applications; then we may actually be in trouble here.

If using iSCSI, make sure you have redundant network paths to the disks, using multiple switches and NIC’s. We have had quite a few cases lately of no fault tolerance iSCSI configs that would go down for hours in the middle of DFSR updating the database and transaction logs, and the results were obviously not pretty.

• Use reliable networks - They don't necessarily have to be fast, but they do need to stay up. Many DFSR performance issues are caused by using old network card drivers, using malfunctioning "Scalable Network" (TCP offload, RSS, etc.) settings, or using defective WANs. Network card vendors release frequent driver updates to increase performance and resolve problems; just like Windows service packs, the drivers should be installed to improve reliability and performance. Companies often deploy cost saving WAN solutions (with VPN tunnels, frame relay circuits, etc.) that in the end cost the company more in lost productivity than they ever saved in monthly expense. DFSR - like all RPC applications - is sensitive to constant network instability.
• Review our performance tuning guides – For much more detail on squeezing performance out of your hardware, including network, storage, and the rest, review:
  
  
  • Performance Tuning Guidelines for Windows Server 2008 R2
  • Performance Tuning Guidelines for Windows Server 2008

And that’s it.

- Ned “fork” Pyle

It’s your guest writer Herbert Mauerer again.

A very common AD disaster is an unexpected deletion or modification of objects. Unlike a bad football match or family meeting, you can prepare for that and make the crisis more bearable. In this blog, I will discuss best practices of Windows Server 2003 and 2008 forest level backup and restore. I will not discuss Windows 2000 as it’s at the end of its lifetime, and also not Windows 2008 R2 because we have a pretty good solution for object deletion without the need for backups: AD Recycle Bin.

AD Object deletions and unwanted modifications are often due to human error. Sometimes a bad script does that or a provisioning system, but then these are also created by humans. The common factor is the data loss. Now there is quite some complexity in the current KB article on AD object restores:

840001 How to restore deleted user accounts and their group memberships in Active Directory
http://support.microsoft.com/default.aspx?scid=kb;EN-US;840001

We basically follow method 1 in the article 840001, with a few tweaks and preparations:

1. Windows Server 2003 Service Pack 2 or newer and Windows Server 2003 Forest Functional level. This allows the restoration of links using LDIF files written by NTDSUTIL.EXE.
2. Preparing for the restore by converting the LEGACY group memberships to Link-Value Replication (LVR) group memberships.

Step 2 requires that you remove and re-add all group members. This is relatively easy using the Windows Server 2003 command line tools for querying and changing AD objects:

Dsget group CN=GroupX,OU=Groups,DC=contoso,DC=com /members | dsmod group CN=GroupX,OU=Groups,DC=contoso,DC=com /chmbr

This command gets the members of the group and replaces the members with the output, thus adding them as LVR members. This may mean some replication traffic if you apply this to many groups within a short time. However, since the removal and re-adding of members happened in a very short time, you should not see the member go away, as all changes should be replicated in the same replication job.

There are two problems with this approach:

1. The version of DSGET.EXE in Windows Server 2003 gets a maximum of 1500 members, and the command above would discard the rest. In one of my test environments, DSGET does not show any members. I have not seen both problems with the Windows Server 2008 or later version of DSGET.
2. If a group has lots of members, the execution of DSMOD.EXE can overflow the AD database version store, and adding the members would fail. So although the change of the membership type is a one-time action, it’s certainly a good idea to first export the member list to files first. I describe that below.
3. Use updated operating system file versions, details also below.

How to Convert Groups for LVR

You can determine the state of linked attribute values like group memberships from the replication meta-data on the object. When you run "repadmin.exe /showobjmeta", the link "Type" says LEGACY:

Type     Attribute       Last Mod Time   Originating DC  Loc.USN Org.USN     Ver  Distinguished Name
=======  ========        =============   =============== ========= ========= ===  ==========================================
LEGACY  member                                                                  CN=test-user1,OU=Test-OU,DC=contoso,DC=com
PRESENT   member   2008-09-16 18:22:29   HQ\contoso-DC1  175379684 175379684   1  CN=test-user2,OU=Test-OU,DC=contoso,DC=com

The line for LEGACY does not have data on the latest change all these values share that with the attribute meta-data listed in the first part of the output.

Hint: I use the "for" command in CMD for loops for direct execution on the command line. When you use CMD scripts, you need to change loop variables like "%f" to "%%f".

The steps are:

1. Freeze the group memberships, stop all group member management.

2. Export the groups in your domain:

Dsquery group dc=contoso,dc=com /limit 0 > grouplist.txt

Review the list and remove all built-in groups. You cannot remove all members there, and these groups usually don’t have many members to begin with. Also, there’s hope nobody deletes the memberships for these.

3. Get all group members on a Windows Vista member with RSAT or Windows Server 2008 computer:

For "delims=" %f in (grouplist.txt) do DSGET %f /members > groupmembers\%f

Important: The DNs of the groups must not contain characters ":\/*?". If you need to rename a group or an OU, it’s sufficient to change the DN.

4. Now we count the members:

In object counting, I use an older version of the MKS toolkit WC.EXE which counts lines, words and characters. You need a similar tool that counts lines in text files. You may have a favorite tool for that, or use a text editor that provides a line count, and use that to get the line count for the bigger output files.

For "delims=" %f in (grouplist.txt) do wc groupmembers \%f >> membercount.txt

The file membercount.txt has the count of members in the first column. Groups with more than 5000 members require special treatment. I suggest moving their member list into a folder "big-groups1".

5. For all other groups, execute:

Dir /b groupmembers >grouplist-small.txt
For /f "delims=" %f in (grouplist-small.txt) do dsmod group "%f" /chmbr < " groupmembers\%f"

Hint: "%f" is in double quotes here as "dir /b" does not print quotes, and we want to handle names with spaces properly.

So after this, the small groups are off our radar.

6. Now for the big groups:

My suggestion is to split the membership lists into multiple text files of approximately 5000 lines and put these sections into separate folders with the same file names (the DN of the group):

Dir /b big-groups1> big-groups1.txt
For /f "delims=" %f in (big-groups1.txt) do dsmod group "%f" /chmbr < " big-groups1\%f"

Dir /b big-groups2> big-groups2.txt
For /f "delims=" %f in (big-groups2.txt) do dsmod group "%f" /chmbr < " big-groups2\%f"

7. Verify the change by getting the group meta-data:

For "delims=" %f in (grouplist.txt) do repadmin /showobjmeta . %f > group-meta\%f

The output files in “group-meta” should have "LEGACY" only for other attributes.

Well, this was quite some work. The good news is that we don’t have to worry about the primary group… :-)

I suggest you test-drive this with test groups, for these tests it does not matter that the members are in LVR mode already. Then expand the work to smaller OU trees until you tackle the rest of the domain. One idea would be to split out parts of the group-list.txt so you impact a few groups only.

Suggested Fixes

After Windows Server 2003 Service Pack 2 was released, we had two problems affecting replication after a (authoritative) restore:

937855 After you restore deleted objects by performing an authoritative restoration on a Windows Server 2003-based domain controller, the linked attributes of some objects are not replicated to the other domain controllers
http://support.microsoft.com/default.aspx?scid=kb;EN-US;937855

943576 Active Directory objects may not be replicated from the restored server after an authoritative restore on a Windows Server 2003-based domain controller
http://support.microsoft.com/default.aspx?scid=kb;EN-US;943576

When you install Fix 943576 on the DCs you backup on a regular basis, you should not see any issues. In the long run, you should get this fix on all DCs in the enterprise. This problem is fixed in Windows 2008.

On top of that, there are also problem with the tool we use to restore the objects, NTDSUTIL.

Problems with object names containing extended characters:
886689 The Ntdsutil authoritative restore operation is not successful if the distinguished name path contains extended characters in Windows Server 2003 and in Windows 2000
http://support.microsoft.com/default.aspx?scid=kb;EN-US;886689

910823 Error message when you try to import .ldf files on a computer that is running Windows Server 2003 with Service Pack 1: "Add error on line LineNumber: No such object"
http://support.microsoft.com/default.aspx?scid=kb;EN-US;910823

Problems with excessive links in the LDIF files:

951320 The ntdsutil.exe utility in Windows Server 2003 writes out too many links to .ldf files during an authoritative restore process
http://support.microsoft.com/default.aspx?scid=kb;EN-US;951320

The good news is that having fix 951320 on the DCs you backup often also takes care of problem 910823. Fix 951320 is included in Windows Server 2008 Service Pack 2.

There is a new problem with authoritative restores where objects are not treated correctly in replication. We have a corrective fix in NTDSUTIL for that:

974803 The domain controller runs slower or stops responding when the garbage collection process runs
http://support.microsoft.com/default.aspx?scid=kb;EN-US;974803

This last problem is not fixed in Windows Server 2008 yet. The problem cannot happen on Windows Server 2008 R2.

How to Avoid Accidental Deletion

Most of the AD objects recovery cases we see in our support work are because of accidental deletions. Typically, there are whole OUs with lots of user, groups and computer accounts affected. These objects have attributes that can’t be recovered by re-creation as you can’t get the same Sid again.

For this problem, Microsoft has implemented an option in the Windows Server 2008 Admin Tools, also available for Windows Vista using RSAT (Remote Server Administration Toolkit). This version has a check box "Protect object from accidental deletion” when “advanced view” is enabled. This flag is set automatically on all OUs you create with this admin tool.

When this is used, the OU itself and its parent get ACEs (Access Control Entries) that deny "Everyone" the permission to delete the object itself and child objects. This works with Domains hosted on Windows Server 2003. You can add the ACEs this way using your own tools and scripts and you will get the same effect.

Thus such a mass deletion should not happen by accident anymore, as you need to step through clearing the check box. Yes, the deletion might still happen from a script, but then it’s written to remove these ACEs and brings the problem to the next level, the script programmer. At this point, you can’t call it accidental anymore.

A Solution

For the object deletion scenario, Windows Server 2008 R2 offers the AD Recycle Bin. The forest needs to run in the Windows Server 2008 R2 forest mode and the feature needs to be enabled separately.

All objects that are deleted will have no attributes removed anymore, and linked attributes will have a third state to signal they are inactive pointing to a deleted object. When the object is reactivated, the link state will be switched to active again.

The graphical user interface for this facility is not in this release, but the groundwork is done, PowerShell is available, and there are scripts to help you perform the recovery.

Until you can enable this feature, the task is to be prepared for object deletions. This article will still be useful for unwanted object attribute modifications.

Herbert “by way of Germany” Mauerer

Hiya folks. The mail sack was a no-show last week since I was out of town; I hope you can find it in your heart to forgive me. If not… well, you get what you pay for. To make up for it, this one is longer than usual. Here are some interesting issues from the past two weeks (both from the Internet and internally too).

• DFSR and temporary files
• ADUC defaults
• Inter-CA communication
• Third party software recommendations
• Compatibility mode

Question

I’m replicating files with DFSR. I’ve found a large number of files that have had the TEMPORARY attribute set on them. I know that DFSR won’t replicate those files, so I am going to remove that attribute. What happens after I do that?

Answer

Short answer: they will replicate. :-)

Long answer: DFSR treats temporary files like they do not exist. This means if a file was normal and replicated between servers, then someone sets the temporary attribute on that file, it is no longer there as far as DFSR is concerned. When a file no longer exists, we treat it as  - wait for – deleted! So the downstream servers from the originating change are going to delete that file.

Here is the debug log from the server where the temporary attribute was just set:

20100325 17:48:46.067 1836 USNC  2881 UsnConsumer::TombstoneOrDelete LDB Updating ID Record: <— Hmm, looks like it’s being treated as a deletion to me.
+    fid                            0x700000000EB52 <— Note the File ID
+    usn                             0xfa1ead8
+    uidVisible                      1
+    filtered                        0
+    journalWrapped                  0
+    slowRecoverCheck                0
+    pendingTombstone                0
+    internalUpdate                  0
+    dirtyShutdownMismatch           0
+    meetInstallUpdate               0
+    meetReanimated                  0
+    recUpdateTime                   20100325 21:47:46.739 GMT
+    present                         0
+    nameConflict                    0
+    attributes                      0x20
+    ghostedHeader                   0
+    data                            0
+    gvsn                            {16EC84D2-ECEE-4E0D-B6FA-0C4137F65EE4}-v277 <— Note the version
+    uid                             {F5C8EE4D-3C55-42A4-BB62-63F72E7EEBED}-v50
+    parent                          {780F3375-CDB0-4583-A3AE-85845086E884}-v1
+    fence                           Default (3)
+    clockDecrementedInDirtyShutdown 0
+    clock                           20100325 21:48:46.067 GMT (0x1cacc64f1e63a10)
+    createTime                      20100325 21:47:29.386 GMT
+    csId                            {780F3375-CDB0-4583-A3AE-85845086E884}
+    hash                            E30F7F47-58F6FE00-B9846C04-43B82E2A
+    similarity                      00000000-00000000-00000000-00000000
+    name                            ohnoes!.txt <— Note the name
+   
20100325 17:48:46.067 1836 USNC  2887 UsnConsumer::TombstoneOrDelete ID record tombstoned from USN_RECORD:
+    USN_RECORD:
+    RecordLength:        96
+    MajorVersion:        2
+    MinorVersion:        0
+    FileRefNumber:       0x700000000EB52 <— Same FID
+    ParentFileRefNumber: 0x1AD000000008E29
+    USN:                 0xfa1ead8
+    TimeStamp:           20100325 17:48:46.067 Eastern Standard Time
+    Reason:              Basic Info Change Close
+    SourceInfo:          0x0
+    SecurityId:          0x0
+    FileAttributes:      0x120 <— This is 0x20 && 0x100, which is a file with the archive bit and the temporary attribute set. Testify!
+    FileNameLength:      30
+    FileNameOffset:      60
+    FileName:            ohnoes!.txt <— Same name

20100325 17:48:46.114 2304 JOIN  1244 Join::SubmitUpdate Sent: uid:{F5C8EE4D-3C55-42A4-BB62-63F72E7EEBED}-v50 gvsn:{16EC84D2-ECEE-4E0D-B6FA-0C4137F65EE4}-v277 name:ohnoes!.txt connId:{2F726EBA-3970-4AA5-AD23-4C07600CE427} csId:{780F3375-CDB0-4583-A3AE-85845086E884} csName:condelrf <— Same name, same version being sent to the server’s partner. Since the last operation was delete, guess what we’re telling the other server to do?

Then later, I remove the temporary attribute:

20100325 17:49:07.523 1836 USNC  2703 UsnConsumer::CreateNewRecord LDB Inserting ID Record: <— Well hello there! Just like a new file got added. It will replicate out momentarily…
+    fid                             0x700000000EB52
+    usn                             0xfa1eb98
+    uidVisible                      0
+    filtered                        0
+    journalWrapped                  0
+    slowRecoverCheck                0
+    pendingTombstone                0
+    internalUpdate                  0
+    dirtyShutdownMismatch           0
+    meetInstallUpdate               0
+    meetReanimated                  0
+    recUpdateTime                   16010101 00:00:00.000 GMT
+    present                         1
+    nameConflict                    0
+    attributes                      0x20
+    ghostedHeader                   0
+    data                            0
+    gvsn                            {16EC84D2-ECEE-4E0D-B6FA-0C4137F65EE4}-v278
+    uid                             {16EC84D2-ECEE-4E0D-B6FA-0C4137F65EE4}-v278
+    parent                          {780F3375-CDB0-4583-A3AE-85845086E884}-v1
+    fence                           Default (3)
+    clockDecrementedInDirtyShutdown 0
+    clock                           20100325 21:49:07.523 GMT (0x1cacc64feb0277e)
+    createTime                      20100325 21:49:07.523 GMT
+    csId                            {780F3375-CDB0-4583-A3AE-85845086E884}
+    hash                            00000000-00000000-00000000-00000000
+    similarity                      00000000-00000000-00000000-00000000
+    name                            ohnoes!.txt

Craig Landis has written all kinds of interesting stuff about the temporary file attribute and DFSR in this old blog post here and now in the new TechNet Wiki here. Maybe someone will update the Wiki with this new tidbit? Hmmm…. could be you!

Question

Is there a way to make the “AD Users and Computers” snap-in (DSA.MSC) always open in “Advanced Features View” by default? I can’t find any command-line switches or registry settings that handle this scenario.

Answer

Oh MMC, how you taunt us. ADUC stores all of its configuration info inside of a special (read: not well documented and highly proprietary) cache file in your profile. You can see it by opening this folder:

%appdata%\microsoft\mmc

Good luck reading the contents of the DSA file though. The setting is stored in there as binary goo. I suppose you could copy your cache file around from place to place and use that, but seems like it would be a rather risky operation with little gain over the the 2 seconds it takes to click:

Perhaps a better idea is to have an "Admin” Terminal Server that has all your favorite applications configured the way you like ‘em. Maybe even using RemoteApp.

Question

I am looking for documentation on how the Root CA communicates with the Sub CAs in a Microsoft PKI. Once the self-signed certificate has been put on the Root CA, is it enrolled or replicated to the Sub CAs? I guess the question is this: once the self signed certificate has been put on the Root CA is it enrolled or replicated to the Sub CAs? We have NPS installed and are using it as a Radius server for the AP's and Wireless LAN Controller. We have a 802.1x policy configured and have the NPS Server validating the wireless client. The client is getting a Wi-Fi policy and a PKI-Policy through a Domain Group Policy. In this environment the wireless clients are getting a certificate with a PKI Policy but when you look at the certificate on the laptop once the system has been logged on the certificate could be issued from any of the Sub CAs. Is this the normal process?

Answer

Root CAs and subordinate CAs don't "communicate" with each other in quite that fashion. In fact, the only time any user or computer (sub CAs included) would need to contact the Root CA, outside of management tasks, is if they wanted to request a certificate. Within the sphere of the Windows CA itself, the answer to your question is no, the root CA certificate is not automatically propagated to subordinate CAs. This task must be done via another process.

If the Windows root CA has been installed on a domain server then the CA will automatically publish its root CA into Active Directory. Every Windows workstation or server will automatically retrieve the root certificate once its presence is detected and thus trust the new CA. If the Windows root CA was installed on a non-domain server then the root certificate must be published to Active Directory manually. Since you mentioned that enrollment is occurring successfully, I don't believe that trusting the root CA is an issue for you.

The next thing to address is how Windows client select the CA from which they request a certificate. The following discussion assumes that CEP/CES has not been installed and that CAs are installed in Enterprise Mode. This process applies to any version of Windows from Windows XP onwards. First, there are two contexts in which certificates can be requested -- the user context or the local computer context. Users can request certificates manually via the Certificates MMC snap-in or the Web Enrollment pages, or certificates may be assigned by an Administrator and distributed by Autoenrollment. Computer certificates can also be requested manually by a local Administrator using the Certificates MMC snap-in, or also by Autoenrollment. Regardless of the context, or how enrollment occurs, the process is the same.

1. The Windows Certificate Client (WCC) queries Active Directory for a list of certificates for which the account (user or computer) has Enroll (or Autoenroll) permissions. This information may be cached in the registry, but is periodically refreshed.
2. The WCC then queries Active Directory for a list of available CAs in the forest. Each CA publishes an Enrollment Services object that holds the FQDN of the CA server and the list of templates available on that CA.
3. The list of templates and the list of CAs (and their configured templates) are combined to build a list of those templates for which the account has permission to request a certificate AND which are available on some trusted CA in the forest.
4. In the case of the Certificates MMC snap-in, this combined list is presented in the UI as a list from which the user can select. In the case of Autoenrollment, this list is compared against those templates assigned to the account via Group Policy.
5. Once a template and the CA have been selected, the public/private key pair is created, the request is generated, and the request is submitted to the CA. Depending on the template settings, the request may be fulfilled immediately or it may be pended so that the request can be manually approved.

What this means for you is that you need to determine the following:

1. Is the Root CA in Enterprise or Standalone mode? If the latter, it will not issue certificates based on templates which eliminates the ability to request certificates via the MMC, or through Autoenrollment.
2. Do you have a certificate template that you assign specifically through your PKI-Policy? If so, on which CAs is that template configured? Check the Certificate Templates folder in the Certificate Services snap-in. (HINT: If you don't have one, the CA is in Standalone mode.)

So, if your root CA isn't configured to issue certificates based on your PKI-Policy template, or is in Standalone mode, but your subordinate CAs *are* configured to issued certificates based on that template, then it's no surprise all the certificates you've looked at were issued from one of the subordinate CAs. Funnily, this is actually a good thing. The purpose of a root CA is to be a Trust Anchor for your entire PKI. It issues certificates to subordinate CAs, which in turn issue certificates to users and computers on your network.

If you want an old but still decent introduction to PKI concepts check out Microsoft Windows 2000 Public Key Infrastructure (http://msdn.microsoft.com/en-us/library/ms995346.aspx). If you want the latest and greatest info and a lot more depth, you should pick up the book "Windows Server 2008 PKI and Certificate Security" by Brian Komar (http://www.microsoft.com/learning/en/us/book.aspx?ID=9549&locale=en-us)

[If you can’t tell, this came from Jonathan, the man who never uses a sentence when a paragraph will do – Ned]

Question

Hey, there are a bunch of different companies that make software to do X. Can you recommend one?

Answer

Nope! If I recommend one over a bunch of others, the others can get really mad. And in case you haven’t noticed, we, ummm, get sued a lot. Even if the other ones don’t sue and only complain loudly over the Internet, and shaking that bush is a real no-no here in Support. So forgive us if we take a pretty hard line and recommend nothing.

I will say this: any good vendor will be more than happy to let you run a fully functional time-bombed trial edition and give things a whirl. An excellent one will let you access their online knowledgebase and support forums before you buy. I am always suspicious of those who do not want you to see any dirty laundry until the check has cleared. Everyone has a competitor; there are no real monopolies in the software world. Use that to your advantage when evaluating new products.

Question

On a Windows Server 2008 R2 computer I can only get back to Vista for compatibility mode (this computer is running some old stuff via remote desktop services). Why is this and is there a way to at least get back to XP mode?

Answer

You tricked yourself – it depends on the application architecture you are running:

The one on the left is a 32-bit program. The one on the right is a 64-bit program. Both are on the same computer. Win2008 R2 runs 64-bit only and when it comes to 64-bit programs, can only pretend back as far as Vista 64-bit RTM. If you need more than this you will need to pay a visit to the Application Compatibility Toolkit. Or, you know, upgrade your crummy old app… ^_^

Hey, sometimes the questions aren’t about DS necessarily. They just need to be interesting enough to make me care. In this case I had no idea, had to figure it out.

Until next time.

- Ned “get to da choppah!” Pyle

Hi Everyone, Mike here. Today I am discussing the default processing behavior for Group Policy scripts. Microsoft changed the default behavior of Group Policy startup and logon scripts processing from synchronous to asynchronous starting with Windows Vista and Windows Server 2008. This behavior is the same in Windows 7 and Windows Server 2008 R2. I’ve recently read some confusion regarding this policy setting.

Computer Startup Scripts:

The default processing behavior for computer startup scripts is asynchronous. Asynchronous scripts processing is when computer startup scripts no longer wait for the previous script to complete before starting the next startup script. Versions of Windows prior to Windows Vista defaulted to synchronous processing. This behavior can be changed by reading from the following locations, in the following order.

Computer Preference

Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
ValueName: RunStartupScriptSync

Computer Policy

Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System
ValueName: RunStartupScriptSync

User Logon Scripts:

The default processing behavior for Group Policy logon scripts is asynchronous. However, this behavior can be changed by reading from the following locations, in order.

User Preference

Key: HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
ValueName: RunLogonScriptSync

Computer Preferences

Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
ValueName: RunLogonScriptSync

User Policy

Key: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\System
ValueName: RunLogonScriptSync

Computer Policy

Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System
ValueName: RunLogonScriptSync

User Logoff/Computer Shutdown:

User Logoff and Computer Shutdown Group Policy Scripts always process synchronously.

Customers are noticing timing issues with their computer startup scripts when using Windows 7 and Windows Server 2008 R2. Yes, this is true because the default behavior changed to run these scripts asynchronously. If you relied on the previous behavior, then you need to configure computer startup scripts to run synchronously. Unfortunately, this is where some confusion surfaces.

You’ll soon notice that Windows does not include a policy setting to enable computer startup scripts to run synchronously, or does it? Yes, it does; however, not so intuitive. I’ve had some GP administrators break out the nice search feature in the Group Policy Management Editor and search for the keyword synchronous—this is not going to return the information you are seeking. The policy setting to enable this does not include the word synchronous in the explain text or in any other portion of the policy. You need to configure the Run startup scripts asynchronously policy setting to disabled to force computer startup scripts to process synchronously, which is the default behavior prior to Windows 7 and Windows Vista.

Hopefully, this will clear up any confusion with how Windows 7 process computer startup scripts and explain how to restore the older behavior.

- Mike “Tips and Tricks is Ned’s kryptonite” Stephens

KB

Blogs

Hey everyone, Mike here again to discuss an interesting feature I learned about in Windows 7. Many Microsoft customers deploy Roaming User Profiles. In fact, many combine Roaming User profiles and Folder Redirection to get the best experience possible. However, one of the drawbacks with Roaming User profiles is the user must logoff before their settings are uploaded to the server. Folder redirection solves this problem for any of the known folders within the user profile namespace, such as Documents, Music, or Downloads—the data is highly available without requiring a user logoff. Now, if this could only occur with user’s registry settings.

Windows 7 solves this problem by allowing the User Profile service to upload the user’s registry settings of a Roaming User profile while the user remains logged on to the computer, or terminal services session (provided the session is hosted on Windows Server 2008 R2).

You enabled this feature using Group Policy. The policy setting Background upload of a roaming user profile’s registry file while user is logged on, is located under Computer Configuration\Policies\Administrative Templates\System\User Profiles. The policy setting offers two configuration settings: scheduled or interval. The scheduled method allows you to configure a time of day (represented on 24 hour time) at which to upload the user’s registry settings. The interval method allow you to choose a specific interval (represented in hours) at which to upload the user’s registry settings. This method accepts an interval range between 1 and 720 inclusively. Both settings include a random delay that does not exceed one hour.

Figure 1 Background upload policy setting in Windows 7

Background uploading only occurs with Roaming User profile. Also, background uploading does not alter uploading the entire profile when the user logs off. It is important to remember that background uploading only uploads the user’s registry settings (ntuser.dat).

Now those subtle changes made in the registry can be uploaded to the server while the user remains logged on. Pretty cool.

Mike “the other Mike Stephens” Stephens

In addition to the normal list of KBs and blog posts, here is a look at the topics related to Directory Services that have been created on the TechNet Wiki recently.

TechNet Wiki Topics

KB

Blogs

Hi folks, Ned here again. This week we hunt down some documentation gremlins and give them a well-deserved smack.

Also, things will be a bit slow next week as I will be out in Redmond teaching this rotation of Microsoft Certified Masters. Never heard of it? If you’re at the IT career tipping point, this may be just what the doctor ordered. No really, it is, and I will be there!

• Dcdiag /fix
• DFSR RPC improvements in Win2008+
• USMT /SF

Question

What exactly does the dcdiag.exe /fix command do? According to this it fixes the SPNs on the DC machine account. But according to this it ensures that SRV records are appropriately registered (I thought the NetLogon service did this?!). And what exactly does the netdiag.exe /fix command do? This article says it "fixes minor problems", whatever that means.

Answer

1. Dcdiag /fix writes back the computers account’s AD replication SPN (DRSUAPI with an index value of “E3514235-4B06-11D1-AB04-00C04FC2DCD2”) entry only. More info on this SPN here:

http://msdn.microsoft.com/en-us/library/dd207876(PROT.13).aspx
http://msdn.microsoft.com/en-us/library/ee791539(PROT.10).aspx

If someone (else!) has destroyed all the other SPN’s, you will need to recreate them or restart whichever service recreates them. For example if the DFSR SPN goes missing, you restart the DFSR service and it will get put back.

2. Netdiag /fix reads the %systemroot%\system32\config\Netlogon.dns file and attempts to register all records in DNS.

I confirmed both in source code, regardless of what old TechNet goo states. :-)

Question

In Win2008 DFSR has been improved regarding the asynchronous RPC connections and 16 concurrent connections for upload and download. Do you have any further info on how improved the performance will be from Win2003 R2 to Win2008/2008 R2? Are there any other factors that would drive me to start rolling out the later OS versions?

Answer

I will be posting posted some new info about performance improvements in 2008/2008 R2 as well as registry tuning options in the coming weeks. But we don’t have any specific case studies that I am aware of yet – I’ll see if I can find them, and if you do, feel free to comment. We do have some rather unspecific ones, if you’re interested.

From testing and customer experience though, we see anywhere from a 4 to 20 times performance improvement of 2008 over 2003 R2, depending on a variety of factors that are often very customer specific (network speed, bandwidth, latency, loss rates, errors, overall uptime + memory + CPU + disk subsystem + drivers). Not only did DFSR improve, but the OS got improvements and it makes better use of newer hardware. Besides the RPC and other changes, Win2008 tweaks the DFSR credit manager, and 2008 R2 really improves it – much more evenly-distributed replication with greatly lowered chance of servers being starved by updates.

Other factors:

1. Win2003 enters extended support on July 13 2010. This means no further hotfixes that improve reliability or performance, and 5 years of crossed fingers until end of life.
2. You would now have the option on DC’s to switch to DFSR-enabled SYSVOL and no longer use FRS there.
3. If deploying 2008 R2, you would also gain read-only and cluster support, which is unavailable in 2003/2008.

Question

I am using your old blog post on making custom registry changes and…

Answer

Ewwwww… The only reason to use that old document is if you are still running Windows 2000 somewhere. Otherwise you should be busting out Group Policy Preferences and wowing your friends and family.

Oh, and really? You’re running Win2000? That’s very uncool of you…

Question

I am doing USMT migrations with /SF. What is that switch and why are my migrations absolutely busted to heck?

Answer

This one came in late last week and was so gnarly that it ended generating a whole blog post. Read more here. Sometimes your questions to us generate more than a Friday reply.

Good work, Internets!

– Ned “3 important rules” Pyle