Blog - Title

March, 2010

  • Tuning replication performance in DFSR (especially on Win2008 R2)

    Hi all, Ned here again. There are a number of ways that DFSR can be tuned for better performance. This article will go through these configurations and explain the caveats. Even if you cannot deploy Windows Server 2008 R2 - for the absolute best performance - you can at least remove common bottlenecks from your older environments. If you are really serious about performance in higher node count DFSR environments though, Win2008 R2’s 3rd generation DFSR is the answer.

    If you’ve been following DFSR for the past few years, you already know about some improvements that were made to performance and scalability starting in Windows Server 2008:

    Windows Server 2003 R2

    Windows Server 2008

    Multiple RPC calls

    RPC Async Pipes (when replicating with other servers running Windows Server 2008)

    Synchronous inputs/outputs (I/Os)

    Asynchronous I/Os

    Buffered I/Os

    Unbuffered I/Os

    Normal Priority I/Os

    Low Priority I/Os (this reduces the load on the system as a result of replication)

    4 concurrent file downloads

    16 concurrent file downloads

    But there’s more you can do, especially in 2008 R2.

    Registry tuning

    All registry values are REG_DWORD (and in the explanations below, are always in decimal). All registry tuning for DFSR in Win2008 and Win2008 R2 is made here:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DFSR\Parameters\Settings

    A restart of the DFSR service is required for the settings to take effect, but a reboot is not required. The list below is not complete, but instead covers the important values for performance. Do not assume that setting a value to the max will make it faster; some settings have a practical limitation before other bottlenecks make higher values irrelevant.

    Important Note: None of these registry settings apply to Windows Server 2003 R2.

    AsyncIoMaxBufferSizeBytes
    Default value: 2097152
    Possible values: 1048576, 2097152, 4194304, 8388608
    Tested high performance value: 8388608
    Set on: All DFSR nodes

    RpcFileBufferSize
    Default value: 262144
    Possible values: 262144, 524288
    Tested high performance value: 524288
    Set on: All DFSR nodes

    StagingThreadCount
    Default value: 6
    (Win2008 R2 only; cannot be changed on Win2008)
    Possible values: 4-16
    Tested high performance value: 8
    Set on: All DFSR nodes. Setting to 16 may generate too much disk IO to be useful.

    TotalCreditsMaxCount
    Default value: 1024
    Possible values: 256-4096
    Tested high performance value: 4096
    Set on: All DFSR nodes that are generally inbound replicating (so hubs if doing data collection, branches if doing data distribution, all servers if using no specific replication flow)

    UpdateWorkerThreadCount
    Default value: 16
    Possible values (Win2008): 4-32
    Possible values (Win2008 R2): 4-63*
    Tested high performance value: 32

    Set on: All DFSR nodes that are generally inbound replicating (so hubs if doing data collection, branches if doing data distribution, all servers if using no specific replication flow. The number being raised here is only valuable when replicating in from more servers than the value. I.e. if replicating in 32 servers, set to 32. If replicating in 45 servers set to 45.

    *Important note: The actual top limit is 64. We have found that under certain circumstances though, setting to 64 can cause a deadlock that prevents DFSR replication altogether. If you exceed the maximum tested value of 32, set to 63 or lower. Do not set to 64 ever. The 32 max limit is recommended because we tested it carefully, and higher values were not rigorously tested. If you set this value to 64, periodically replication will stop working, the dfsrdiag replstate command hangs and does not return results, and the dfsrdiag backlog command hangs and does not return results.

     

    When using all the above registry tuning on Windows Server 2008 R2, testing revealed that initial sync replication time was sometimes twice as fast compared to no registry settings in place. This was using 32 servers replicating a "data collection" topology to a single hub over thirty-two non-LAN networks with 32 RG's containing unique branch office data. The slower the network, the better the relative performance averaged:

    Test

    Spokes

    Hubs

    Topology

    GB/node

    Unique

    RG

    Tuned

    Network

    Time to sync

    C1

    32

    1

    Collect

    1

    Yes

    32

    N

    1Gbps

    0:57:27

    C2

    32

    1

    Collect

    1

    Yes

    32

    Y

    1Gbps

    0:53:09

    C3

    32

    1

    Collect

    1

    Yes

    32

    N

    1.5Mbps

    3:31:36

    C4

    32

    1

    Collect

    1

    Yes

    32

    Y

    1.5Mbps

    2:24:09

    C5

    32

    1

    Collect

    1

    Yes

    32

    N

    512Kbps

    10:56:42

    C6

    32

    1

    Collect

    1

    Yes

    32

    Y

    512Kbps

    5:57:09

    C7

    32

    1

    Collect

    1

    Yes

    32

    N

    256Kbps

    21:43:02

    C8

    32

    1

    Collect

    1

    Yes

    32

    Y

    256Kbps

    10:46:46

    On Windows Server 2008 the same registry values showed considerably less performance improvement; this is partly due to additional service improvements made to DFSR in Win2008 R2, especially around the Credit Manager. Just like your phone, “3G” DFSR is going to work better than older models…

    Note: do not use this table to predict replication times. It is designed to show behavior trends only!

    Topology tuning

    Even if you are not using Windows Server 2008 R2 there are plenty of other factors to fast replication. Some of these I’ve talked about before, some are new. All are important:

    • Minimize mixing of Win2003 and Win2008/Win2008 R2 - Windows Server 2008 introduced significant DFSR changes for RPC, inbound and outbound threading, and other aspects. However, if a Win2008 server is partnered with a Win2003 server for DFSR, most of those improvements are disabled for backwards compatibility. An ideal environment is 100% Windows Server 2008 R2, but a Win2008-only is still a huge improvement. Windows Server 2003 should be phased out of use as quickly as possible as it has numerous "1G" design issues that were improved on with experience in later OS's. Windows Server 2008 R2 credit manager and update worker improvements are most efficient when all operating systems are homogenous. If you are replacing Win2003 servers with newer OS, do the hub servers first as the increased number of files will provide some benefits even when talking to 2003 spokes.
    • Consider multiple hubs - If using a large number of branch servers in a hub-and-spoke topology, adding “subsidiary hub” servers will help reduce load on the main hubs.

      So for example, this configuration would cause more bottlenecking:

    image

    And this configuration would cause less bottlenecking:

    image

    • Increase staging quota - The larger the replicated folder staging quotas are for each server, the less often files must be restaged when replicating inbound changes. In a perfect world, staging quota would be configured to match the size of the data being replicated. Since this is typically impossible, it should be made as large as reasonably possible. It must always be configured to be at least as large as the combined size of the count of the files controlled by UpdateWorkerThreadCount+16 on Win2008 and Win2008 R2. Why 16? Because that is the number of outbound files that could be replicated simultaneously.

    This means that by default on Win2008/Win2008 R2, quota must be as large as the 32 largest files. If UpdateWorkerThreadCount is increased to 32, it must be as large as the 48 largest files (32+16). If any smaller then staging can become blocked when all 32 files are being replicated inbound and 16 outbound, preventing further replication until that queue is cleared. Frequent 4202 and 4204 staging events are indications of an inappropriately configured staging quota, especially if no longer in the initial sync phase of setting up DFSR for the first time.

    Source : DFSR
    Catagory : None
    Event ID : 4202
    Type : Warning
    Description :
    The DFS Replication service has detected that the staging space in use for the replicated folder at local path c:\foo is above the high watermark. The service will attempt to delete the oldest staging files. Performance may be affected.   

    Source : DFSR
    Catagory : None
    Event ID : 4204
    Type : Information
    Description :
    The DFS Replication service has successfully deleted old staging files for the replicated folder at local path c:\foo. The staging space is now below the high watermark.

    If you get 4206 staging events you have really not correctly sized your staging, as you are now blocking replication behind large files.

    Event Type: Warning
    Event Source: DFSR
    Event Category: None
    Event ID: 4206
    Date: 4/4/2009
    Time: 3:57:21 PM
    User: N/A
    Computer: SRV
    Description:
    The DFS Replication service failed to clean up old staging files for the replicated folder at local path c:\foo. The service might fail to replicate some large files and the replicated folder might get out of sync. The service will automatically retry staging space cleanup in 1 minutes. The service may start cleanup earlier if it detects some staging files have been unlocked.

    If still using Win2003 R2, staging quota would need to be as large as the 9 largest files. And if using read-only replication on Windows Server 2008 R2, at least 16 or the size specified in UpdateWorkerThreadCount – after all, a read-only replicated folder has no outbound replication.

    So to recap the staging quota minimum recommendations:

    - Windows Server 2003 R2: 9 largest files
    - Windows Server 2008: 32 largest files (default registry)
    - Windows Server 2008 R2: 32 largest files (default registry)
    - Windows Server 2008 R2 Read-Only: 16 largest files

    If you want to find the 32 largest files in a replicated folder, here’s a sample PowerShell command:

    Get-ChildItem <replicatedfolderpath> -recurse | Sort-Object length -descending | select-object -first 32 | ft name,length -wrap –auto

    • Consider read-only - Deploy Windows Server 2008 R2 read-only replication when possible. If users are not supposed to change data, mark those replicated folders as read-only. A read-only server cannot originate data and will prevent unwanted replication or change orders from occurring outbound to other servers. Unwanted changes generate load and lead to data overwrites – which to fix you will need to replicate back out from backups, consuming time and replication resources.
    • Latest QFE and SP - Always run the latest service pack for that OS, and the latest DFSR.EXE/DFSRS.EXE for that OS. There are also updates for NTFS and other components that DFSR relies on. Hotfixes have been released that remove performance bugs or make DFSR more reliable; a more reliable DFSR is naturally faster too. These are documented in KB968429 and KB958802 but the articles aren’t always perfectly up to date, so here’s a trick: If you want to find the latest DFSR service updates, use these three searches and look for the highest KB number in the results:

    Win2008 R2: http://www.bing.com/search?q=%22windows+server+2008+r2%22+%22dfsrs.exe%22+kbqfe+site%3Asupport.microsoft.com&go=&form=QBRE

    Win2008: http://www.bing.com/search?q=%22windows+server+2008%22+%22dfsrs.exe%22+kbqfe+site%3Asupport.microsoft.com&form=QBRE&qs=n

    Win2003 R2: http://www.bing.com/search?q=%22windows+server+2003+r2%22+%22dfsr.exe%22+kbqfe+site%3Asupport.microsoft.com&form=QBRE&qs=n

    Remember, Win2003 mainstream support ends July 13, 2010. That’s the end of non-security updates for that OS.

    People ask me all the time why I take such a hard line on DFSR hotfixes. I ask in return “Why don’t you take such a hard line?” These fixes cost us a fortune, we’re not writing them for our health. And that goes for all other components too, not just DFSR. It’s an issue intrinsic to all software. DFSR is not less reliable than many other Windows components – after all, NTFS is considered an extremely reliable file system but that hasn’t stopped it from having 168 hotfixes in its lifetime; DFSR just has a passionate group of Support Engineers and developers here at MS that want you to have the best experience.

    • Turn off RDC on fast connections with mostly smaller files - later testing (not addressed in the chart below) showed 3-4 times faster replication when using LAN-speed networks (i.e. 1GBb or faster) on Win2008 R2. This is because it was faster to send files in their totality than send deltas, when the files were smaller and more dynamic and the network was incredibly fast. The performance improvements were roughly twice as fast on Win2008 non-R2. This should absolutely not be done on WAN networks under 100 Mbit though as it will likely have a very negative affect.
    • Consider and test anti-virus exclusions – Most anti-virus software has no concept of the data types that make up DFSR’s working files and database. Additionally, those file types are not executables and are therefore very unlikely to contain a useful malicious payload. If you are seeing slow performance within DFSR, test the following anti-virus file exclusions; if DFSR performs considerably better, contact your AV vendor for an updated version of their software and an explanation around the performance gap.

    <drive>:\system volume information\DFSR\

       $db_normal$
       FileIDTable_* 
       SimilarityTable_*

    <drive>:\system volume information\DFSR\database_<guid>\

       $db_dirty$
       Dfsr.db
       Fsr.chk
       *.log
       Fsr*.jrs
       Tmp.edb

    <drive>:\system volume information\DFSR\config\

       *.xml

    <drive>:\<replicated folder>\dfsrprivate\staging\*

       *.frx

    This should be validated carefully; many anti-virus products allow exclusions to be set but then do not actually abide by them. For maximum performance, you would exclude scanning any replicated files at all, but this is obviously unfeasible for most customers.

    • Pre-seed the data when setting up a new replicated folder- Pre-seeding - often referred to as "pre-staging" - data on servers can lead to huge performance gains during initial sync. This is especially useful when creating new branch office servers; if they are being built in the home office, they can be quickly pre-seeded with data then sent out to the field for replication of the change delta. See the following article for pre-seeding recommendations.

    Going back to those same tests I showed earlier with 32 spokes replicating back to a single hub, note the average performance behavior when the data was perfectly pre-seeded:

    Test

    Spokes

    Hubs

    Topology

    GB/node

    Unique

    RG

    Tuned

    Staging

    Net

    Time to sync

    C9

    32

    1

    Collect

    1

    Yes

    32

    Y

    4GB

    1Gbps

    0:49:21

    C11

    32

    1

    Collect

    1

    Yes

    32

    Y

    4GB

    512Kbps

    0:46:34

    C12

    32

    1

    Collect

    1

    Yes

    32

    Y

    4GB

    256Kbps

    0:46:08

    C13

    32

    1

    Collect

    1

    Yes

    32

    Y

    4GB

    64Kbps

    0:48:29

    Even the 64Kbps frame relay connection was nearly as fast as the LAN! This is because no files had to be sent, only file hashes.

    Note: do not use this table to predict replication times. It is designed to show behavior trends only.

    • Go native Windows Server 2008 R2 – Not to beat a dead horse but the highest performance gains - including registry tuning and the greatly improved Credit Manager code - will be realized by using Windows Server 2008 R2. Win2003 R2 was first generation DFSR, Win2008 was second generation, and Win2008 R2 is third generation; if you are serious about performance you must get to 2008 R2.

    Hardware tuning

    • Use 64-bit OS with as much RAM as possible on hubs - DFSR can become bound by RAM availability on busy hub servers, especially when using the registry performance values above. There is absolutely no reason to run a 32-bit file server in this day and age, and with the coming of Windows Server 2008 R2, it’s no longer possible. For spoke servers that tend to have far less load, you can cut more corners of course; the ten-user sales team in Hicksville doesn’t need 16GB of RAM in their file server.

    As a side note, customers periodically open cases to report “memory leaks” in DFSR. What we discuss is that DFSR intentionally caches as much RAM as it can get its hands on – really though, it’s the ESE (Jet) database doing this. So the idler other processes on a DFSR server are, the more memory a DFSR process will be able to gobble up. You can see the same behavior in LSASS’s database on DC’s.

    • Use the fastest disk subsystem you can afford on hubs - Much of DFSR will be disk bound - especially in staging and RDC operations - so high disk throughput will dramatically lower bottlenecks; this is especially true on hub servers. As always, a disk queue length greater than 2 in PerfMon is in indication of an over-used or under-powered disk subsystem. Talk to your hardware vendors about the performance and cost differences of SATA, SCSI, and FC. Don’t forget about reliability too – I have a job here for life thanks to all the customers that use the least expensive, off-brand, no warranty, low parity, practically consumer-grade iSCSI products they can find. You get what you pay for and ultimately your users do not care about anything but their data. The OS is just a thing to make applications access files so that the business can make money. Someday the Linux desktop folks will figure this out and get some applications; then we may actually be in trouble here.

    If using iSCSI, make sure you have redundant network paths to the disks, using multiple switches and NIC’s. We have had quite a few cases lately of no fault tolerance iSCSI configs that would go down for hours in the middle of DFSR updating the database and transaction logs, and the results were obviously not pretty.

    • Use reliable networks - They don't necessarily have to be fast, but they do need to stay up. Many DFSR performance issues are caused by using old network card drivers, using malfunctioning "Scalable Network" (TCP offload, RSS, etc.) settings, or using defective WANs. Network card vendors release frequent driver updates to increase performance and resolve problems; just like Windows service packs, the drivers should be installed to improve reliability and performance. Companies often deploy cost saving WAN solutions (with VPN tunnels, frame relay circuits, etc.) that in the end cost the company more in lost productivity than they ever saved in monthly expense. DFSR - like all RPC applications - is sensitive to constant network instability.
    • Review our performance tuning guides – For much more detail on squeezing performance out of your hardware, including network, storage, and the rest, review:

    And that’s it.

    - Ned “fork” Pyle

  • Group Policy Script Processing Behavior

    Hi Everyone, Mike here. Today I am discussing the default processing behavior for Group Policy scripts. Microsoft changed the default behavior of Group Policy startup and logon scripts processing from synchronous to asynchronous starting with Windows Vista and Windows Server 2008. This behavior is the same in Windows 7 and Windows Server 2008 R2. I’ve recently read some confusion regarding this policy setting.

    Computer Startup Scripts:

    The default processing behavior for computer startup scripts is asynchronous. Asynchronous scripts processing is when computer startup scripts no longer wait for the previous script to complete before starting the next startup script. Versions of Windows prior to Windows Vista defaulted to synchronous processing. This behavior can be changed by reading from the following locations, in the following order.

    Computer Preference

    Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
    ValueName: RunStartupScriptSync

    Computer Policy

    Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System
    ValueName: RunStartupScriptSync

    User Logon Scripts:

    The default processing behavior for Group Policy logon scripts is asynchronous. However, this behavior can be changed by reading from the following locations, in order.

    User Preference

    Key: HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
    ValueName: RunLogonScriptSync

    Computer Preferences

    Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
    ValueName: RunLogonScriptSync

    User Policy

    Key: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\System
    ValueName: RunLogonScriptSync

    Computer Policy

    Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System
    ValueName: RunLogonScriptSync

    User Logoff/Computer Shutdown:

    User Logoff and Computer Shutdown Group Policy Scripts always process synchronously.

    Customers are noticing timing issues with their computer startup scripts when using Windows 7 and Windows Server 2008 R2. Yes, this is true because the default behavior changed to run these scripts asynchronously. If you relied on the previous behavior, then you need to configure computer startup scripts to run synchronously. Unfortunately, this is where some confusion surfaces.

    You’ll soon notice that Windows does not include a policy setting to enable computer startup scripts to run synchronously, or does it? Yes, it does; however, not so intuitive. I’ve had some GP administrators break out the nice search feature in the Group Policy Management Editor and search for the keyword synchronous—this is not going to return the information you are seeking. The policy setting to enable this does not include the word synchronous in the explain text or in any other portion of the policy. You need to configure the Run startup scripts asynchronously policy setting to disabled to force computer startup scripts to process synchronously, which is the default behavior prior to Windows 7 and Windows Vista.

    Hopefully, this will clear up any confusion with how Windows 7 process computer startup scripts and explain how to restore the older behavior.

    - Mike “Tips and Tricks is Ned’s kryptonite” Stephens

  • Best practices around Active Directory Authoritative Restores in Windows Server 2003 and 2008

    It’s your guest writer Herbert Mauerer again.

    A very common AD disaster is an unexpected deletion or modification of objects. Unlike a bad football match or family meeting, you can prepare for that and make the crisis more bearable. In this blog, I will discuss best practices of Windows Server 2003 and 2008 forest level backup and restore. I will not discuss Windows 2000 as it’s at the end of its lifetime, and also not Windows 2008 R2 because we have a pretty good solution for object deletion without the need for backups: AD Recycle Bin.

    AD Object deletions and unwanted modifications are often due to human error. Sometimes a bad script does that or a provisioning system, but then these are also created by humans. The common factor is the data loss. Now there is quite some complexity in the current KB article on AD object restores:

    840001 How to restore deleted user accounts and their group memberships in Active Directory
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;840001

    We basically follow method 1 in the article 840001, with a few tweaks and preparations:

    1. Windows Server 2003 Service Pack 2 or newer and Windows Server 2003 Forest Functional level. This allows the restoration of links using LDIF files written by NTDSUTIL.EXE.
    2. Preparing for the restore by converting the LEGACY group memberships to Link-Value Replication (LVR) group memberships.

    Step 2 requires that you remove and re-add all group members. This is relatively easy using the Windows Server 2003 command line tools for querying and changing AD objects:

    Dsget group CN=GroupX,OU=Groups,DC=contoso,DC=com /members | dsmod group CN=GroupX,OU=Groups,DC=contoso,DC=com /chmbr

    This command gets the members of the group and replaces the members with the output, thus adding them as LVR members. This may mean some replication traffic if you apply this to many groups within a short time. However, since the removal and re-adding of members happened in a very short time, you should not see the member go away, as all changes should be replicated in the same replication job.

    There are two problems with this approach:

    1. The version of DSGET.EXE in Windows Server 2003 gets a maximum of 1500 members, and the command above would discard the rest. In one of my test environments, DSGET does not show any members. I have not seen both problems with the Windows Server 2008 or later version of DSGET.
    2. If a group has lots of members, the execution of DSMOD.EXE can overflow the AD database version store, and adding the members would fail. So although the change of the membership type is a one-time action, it’s certainly a good idea to first export the member list to files first. I describe that below.
    3. Use updated operating system file versions, details also below.

    How to Convert Groups for LVR

    You can determine the state of linked attribute values like group memberships from the replication meta-data on the object. When you run "repadmin.exe /showobjmeta", the link "Type" says LEGACY:

    Type     Attribute       Last Mod Time   Originating DC  Loc.USN Org.USN     Ver  Distinguished Name
    =======  ========        =============   =============== ========= ========= ===  ==========================================
    LEGACY  member                                                                  CN=test-user1,OU=Test-OU,DC=contoso,DC=com
    PRESENT   member   2008-09-16 18:22:29   HQ\contoso-DC1  175379684 175379684   1  CN=test-user2,OU=Test-OU,DC=contoso,DC=com

    The line for LEGACY does not have data on the latest change all these values share that with the attribute meta-data listed in the first part of the output.

    Hint: I use the "for" command in CMD for loops for direct execution on the command line. When you use CMD scripts, you need to change loop variables like "%f" to "%%f".

    The steps are:

    1. Freeze the group memberships, stop all group member management.

    2. Export the groups in your domain:

    Dsquery group dc=contoso,dc=com /limit 0 > grouplist.txt

    Review the list and remove all built-in groups. You cannot remove all members there, and these groups usually don’t have many members to begin with. Also, there’s hope nobody deletes the memberships for these.

    3. Get all group members on a Windows Vista member with RSAT or Windows Server 2008 computer:

    For "delims=" %f in (grouplist.txt) do DSGET %f /members > groupmembers\%f

    Important: The DNs of the groups must not contain characters ":\/*?". If you need to rename a group or an OU, it’s sufficient to change the DN.

    4. Now we count the members:

    In object counting, I use an older version of the MKS toolkit WC.EXE which counts lines, words and characters. You need a similar tool that counts lines in text files. You may have a favorite tool for that, or use a text editor that provides a line count, and use that to get the line count for the bigger output files.

    For "delims=" %f in (grouplist.txt) do wc groupmembers \%f >> membercount.txt

    The file membercount.txt has the count of members in the first column. Groups with more than 5000 members require special treatment. I suggest moving their member list into a folder "big-groups1".

    5. For all other groups, execute:

    Dir /b groupmembers >grouplist-small.txt
    For /f "delims=" %f in (grouplist-small.txt) do dsmod group "%f" /chmbr < " groupmembers\%f"

    Hint: "%f" is in double quotes here as "dir /b" does not print quotes, and we want to handle names with spaces properly.

    So after this, the small groups are off our radar.

    6. Now for the big groups:

    My suggestion is to split the membership lists into multiple text files of approximately 5000 lines and put these sections into separate folders with the same file names (the DN of the group):

    Dir /b big-groups1> big-groups1.txt
    For /f "delims=" %f in (big-groups1.txt) do dsmod group "%f" /chmbr < " big-groups1\%f"

    Dir /b big-groups2> big-groups2.txt
    For /f "delims=" %f in (big-groups2.txt) do dsmod group "%f" /chmbr < " big-groups2\%f"

    7. Verify the change by getting the group meta-data:

    For "delims=" %f in (grouplist.txt) do repadmin /showobjmeta . %f > group-meta\%f

    The output files in “group-meta” should have "LEGACY" only for other attributes.

    Well, this was quite some work. The good news is that we don’t have to worry about the primary group… :-)

    I suggest you test-drive this with test groups, for these tests it does not matter that the members are in LVR mode already. Then expand the work to smaller OU trees until you tackle the rest of the domain. One idea would be to split out parts of the group-list.txt so you impact a few groups only.

    Suggested Fixes

    After Windows Server 2003 Service Pack 2 was released, we had two problems affecting replication after a (authoritative) restore:

    937855 After you restore deleted objects by performing an authoritative restoration on a Windows Server 2003-based domain controller, the linked attributes of some objects are not replicated to the other domain controllers
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;937855

    943576 Active Directory objects may not be replicated from the restored server after an authoritative restore on a Windows Server 2003-based domain controller
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;943576

    When you install Fix 943576 on the DCs you backup on a regular basis, you should not see any issues. In the long run, you should get this fix on all DCs in the enterprise. This problem is fixed in Windows 2008.

    On top of that, there are also problem with the tool we use to restore the objects, NTDSUTIL.

    Problems with object names containing extended characters:
    886689 The Ntdsutil authoritative restore operation is not successful if the distinguished name path contains extended characters in Windows Server 2003 and in Windows 2000
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;886689

    910823 Error message when you try to import .ldf files on a computer that is running Windows Server 2003 with Service Pack 1: "Add error on line LineNumber: No such object"
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;910823

    Problems with excessive links in the LDIF files:

    951320 The ntdsutil.exe utility in Windows Server 2003 writes out too many links to .ldf files during an authoritative restore process
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;951320

    The good news is that having fix 951320 on the DCs you backup often also takes care of problem 910823. Fix 951320 is included in Windows Server 2008 Service Pack 2.

    There is a new problem with authoritative restores where objects are not treated correctly in replication. We have a corrective fix in NTDSUTIL for that:

    974803 The domain controller runs slower or stops responding when the garbage collection process runs
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;974803

    This last problem is not fixed in Windows Server 2008 yet. The problem cannot happen on Windows Server 2008 R2.

    How to Avoid Accidental Deletion

    Most of the AD objects recovery cases we see in our support work are because of accidental deletions. Typically, there are whole OUs with lots of user, groups and computer accounts affected. These objects have attributes that can’t be recovered by re-creation as you can’t get the same Sid again.

    For this problem, Microsoft has implemented an option in the Windows Server 2008 Admin Tools, also available for Windows Vista using RSAT (Remote Server Administration Toolkit). This version has a check box "Protect object from accidental deletion” when “advanced view” is enabled. This flag is set automatically on all OUs you create with this admin tool.

    When this is used, the OU itself and its parent get ACEs (Access Control Entries) that deny "Everyone" the permission to delete the object itself and child objects. This works with Domains hosted on Windows Server 2003. You can add the ACEs this way using your own tools and scripts and you will get the same effect.

    Thus such a mass deletion should not happen by accident anymore, as you need to step through clearing the check box. Yes, the deletion might still happen from a script, but then it’s written to remove these ACEs and brings the problem to the next level, the script programmer. At this point, you can’t call it accidental anymore.

    A Solution

    For the object deletion scenario, Windows Server 2008 R2 offers the AD Recycle Bin. The forest needs to run in the Windows Server 2008 R2 forest mode and the feature needs to be enabled separately.

    All objects that are deleted will have no attributes removed anymore, and linked attributes will have a third state to signal they are inactive pointing to a deleted object. When the object is reactivated, the link state will be switched to active again.

    The graphical user interface for this facility is not in this release, but the groundwork is done, PowerShell is available, and there are scripts to help you perform the recovery.

    Until you can enable this feature, the task is to be prepared for object deletions. This article will still be useful for unwanted object attribute modifications.

    Herbert “by way of Germany” Mauerer

  • Read-Only Replication in R2

    Ned here again. I’ve been involved with DFSR since its late beta days in 2005. Through good luck and a previous interest in FRS, I decided to take on DFSR as a focus area; when the support cases started rolling in I dealt with most of the issues. My expertise in DFSR is pretty much happenstance. :-)

    Since that first week, I have had hundreds of customers ask me about configuring DFSR “one-way” replication. I’ve also had many customers try to invent their own read-only configurations for DFSR, with results ranging from pretty decent – using share permissions - to “OMG cleanse it with fire!!!” topology hacking.

    Thankfully, Windows Server 2008 R2 takes care of this now in a more sophisticated way. I am going to discuss the architecture of read-only replication in Win 2008 R2, cover some best practices, and describe a few troubleshooting techniques. If you are looking for introductory information about why you would use read-only DFSR or how to configure it step by step, I highly recommend the DFSR development teams blog posts and help file:

    On with the show.

    Architecture

    Read-only (RO) replication refers to the concept of “one-way” replication, where a downstream RO replicated folder is incapable of sending files outbound. It also prevents the creation, modification, or deletion of files in the RO content set locally. RO replicated folders sets can exist on a server that also contains other Read-Write (RW) replicated folders - the boundary of RW or RO is the replicated folder itself on a given server. This new system is ideal for static data that end users are never supposed to modify, such as application installation points, corporate procedural files, centralized backup locations etc.

    RO replication relies on the new mini filter file system driver DFSRRO.SYS. This IO-blocking driver is at an altitude below most common filters, meaning that other drivers will have to abide by DFSRRO’s decisions – including anti-virus software. DFSRRO.SYS is implemented at the “FSFilter Content Screener” level:

    image

    Because the filter driver blocks write IOs, any applications or even many kernel-mode drivers that attempt to save data in the RO folder will receive "Access is denied" (0x5 ERROR_ACCESS_DENIED). This also means that if an application opens files with a WRITE disposition unnecessarily - even if the application only intends to read data – DFSR will return access denied. You can read more about CreateFile dispositions here and file system drivers here.

    A Replication Group containing an RO replicated folder must still contain both inbound and outbound connections. Since a replication group could potentially maintain a mix of RO/RW, and because an RO can change to an RW dynamically, both connections must exist at all times. RO replicated folders can only directly replicate from RW replicated folders. An RO replicated folder cannot source from another RO replicated folder. Also, an RW cannot transitively replicate through an RO.

    So these configurations are good:

    image                     image                 image

    And these configurations are bad:

     

    image                                 image

    Note: my arrows indicate direction of replication, not a single DFSR connection object! Don’t make me come over there.

    An RW replicated folder can be converted to an RO replicated folder (and back) “on the fly”. Converting will cause a non-authoritative sync to occur on the replicated folder for the server being altered. This is done to ensure that the contents are up to date with the partner before data can flow inbound/outbound. This gives you DFSR event log messages 4620 or 4622:

    Log Name:      DFS Replication
    Source:        DFSR
    Date:          2/22/2010 4:27:29 PM
    Event ID:      4620
    Task Category: None
    Level:         Information
    Keywords:      Classic
    User:          N/A
    Computer:      2008r2-f-01.contoso.com
    Description:
    The DFS Replication service has detected that ken, which used to be a read-write replicated folder has now been configured to be a read-only replicated folder. The DFS Replication service will not allow any changes to be made to the contents of this replicated folder. Any updates occurring on other read-write replicated folders will be replicated in and applied to the contents of this read-only replicated folder.
    Additional Information:
    Replicated Folder Name: ken
    Replicated Folder Root: c:\ken
    Replicated Folder ID: 171E6A7E-6182-496D-8277-1797FF18C05C
    Replication Group Name: clu2
    Replication Group ID: 8A645529-FE74-4430-9ECD-D1BDC0BA800A
    Member ID: 0EAD46B4-A442-48EA-97F6-109714968E40

    ==========================

    Log Name:      DFS Replication
    Source:        DFSR
    Date:          2/22/2010 4:31:31 PM
    Event ID:      4622
    Task Category: None
    Level:         Information
    Keywords:      Classic
    User:          N/A
    Computer:      2008r2-f-01.contoso.com
    Description:
    The DFS Replication service has detected that ken, which used to be a read-only replicated folder has now been configured to be a read-write replicated folder. The DFS Replication service will now allow any changes to be made to the contents of this replicated folder. Any updates occurring on this read-write replicated folder will be replicated out and applied to the contents of other replicated folders on other members.
    Additional Information:
    Replicated Folder Name: ken
    Replicated Folder Root: c:\ken
    Replicated Folder ID: 171E6A7E-6182-496D-8277-1797FF18C05C
    Replication Group Name: clu2
    Replication Group ID: 8A645529-FE74-4430-9ECD-D1BDC0BA800A
    Member ID: 0EAD46B4-A442-48EA-97F6-109714968E40

    These events would be followed by 4102 and 4104. There is also a new event that will write when someone tries to restore over a read-only RF:

    Log Name: DFS Replication
    Source: DFSR
    Date: 11/21/2008 11:29:14 PM
    Event ID: 1112
    Task Category: None
    Level: Error
    Keywords: Classic
    User: N/A
    Computer: WIN7-6946-02.southridgevideo.com
    Description:
    The DFS Replication service failed a restore request. This could happen if an
    attempt was made to restore the contents of a read-only replicated folder
    authoritatively. Read-only replicated folders should only be restored
    non-authoritatively. Authoritative restores should be performed only on read-write
    replicated folders.

    Additional Information:
    Replicated Folder Name: foo
    Replicated Folder ID: 02436A8B-D620-4D83-980A-657E2603FBA0

    Finally, there are also new debug entries for RO, under the module ID’s BFLT, FREP, RFLT ad well as updated in VLMG and CCTX. If that makes no sense, review this insomnia cure.

    Best Practices and Answers to Common Questions

    Note: some of these will be repeats from this and other blog posts. I do that because no matter how many times I talk about certain best practices, people continue to ignore me. :-)

    1. Before deploying read-only folders you must test! Your antivirus software, your backup software, and especially your line of business applications that open files from the replicated folder may all react badly to RO. There are no known MS applications that have any problems.

    2. Converting a replicated folder from RW to RO (or the reverse) will cause a non-authoritative sync to occur on the replicated folder. Plan accordingly.

    3. DFSR read-only is not a panacea! If users can access files on the read-write folders, RO will not prevent anything. If you are trying to stop administrators from changing data, get better administrators.

    4. Tell your users! Unless you want to have lots of help desk tickets about “I get access denied errors”, make sure your users understand that you are marking previously writable data as read-only. Not that they will read the email where you told them this, but at least you tried.

    5. You cannot authoritatively restore data to an RO folder. Duh. Getting backups from there is fine and highly recommended.

    6. Don’t bother trying to circumvent RO. Even when running as SYSTEM, you will not be able to write files into an RO-protected folder. If you don’t want RO protection, don’t run RO. Make your changes on RW folders.

    7. RO can replicate inbound from Windows Server 2003 R2 and Windows 2008. You will need to have extended the AD schema to at least Windows Server 2008 though.

    8. An RO replicated folder can have more than one RW partner replicating inbound, as long as those RW servers have more than just the transitive RO connection.

    Troubleshooting

    There aren’t many new steps to troubleshooting read-only replication compared to classic read-write – make sure you have two connections between every server, in both directions, just like RW. Make sure you have the latest hotfixes. Review everything I described above about topology – most issues in RO have been admin-generated so far. :-)

    The one thing you can add to your test environment troubleshooting is FLTMC.EXE. This tool allows you to dynamically load and unload the DFSRRO.SYS filter driver for testing purposes, using the LOAD and UNLOAD commands. This way if you are testing an application that mysteriously fails when talking to a read-only folder, you have a way to temporarily allow writing to the folder and confirm the application is trying to write data without the need to completely remove RO. There is a lazy sync worker thread that removes changes and replicates down changes from the RW server after the driver is reloaded, so that the file system does not get irrevocably out of sync. Remember to load the driver back up when done testing!

    Wrap Up

    It’s one more good reason to ditch FRS and deploy DFSR.

    And a funny story about that previous interest in FRS I mentioned: most people in MS support really dislike FRS for all of its inadequacies, but I owe it a debt of gratitude. When I was in my hiring interview years ago, one of the questions I got asked was “Can you describe in as much detail as possible how FRS processes files?” The panel thought that was a real stumper. So I drew this on the whiteboard from memory and explained it:

    image

    And I got the job. So thanks FRS, I owe you this gig.

    Until next time.

    Ned “r” Pyle

  • Background uploading of User Registry Settings

    Hey everyone, Mike here again to discuss an interesting feature I learned about in Windows 7. Many Microsoft customers deploy Roaming User Profiles. In fact, many combine Roaming User profiles and Folder Redirection to get the best experience possible. However, one of the drawbacks with Roaming User profiles is the user must logoff before their settings are uploaded to the server. Folder redirection solves this problem for any of the known folders within the user profile namespace, such as Documents, Music, or Downloads—the data is highly available without requiring a user logoff. Now, if this could only occur with user’s registry settings.

    Windows 7 solves this problem by allowing the User Profile service to upload the user’s registry settings of a Roaming User profile while the user remains logged on to the computer, or terminal services session (provided the session is hosted on Windows Server 2008 R2).

    You enabled this feature using Group Policy. The policy setting Background upload of a roaming user profile’s registry file while user is logged on, is located under Computer Configuration\Policies\Administrative Templates\System\User Profiles. The policy setting offers two configuration settings: scheduled or interval. The scheduled method allows you to configure a time of day (represented on 24 hour time) at which to upload the user’s registry settings. The interval method allow you to choose a specific interval (represented in hours) at which to upload the user’s registry settings. This method accepts an interval range between 1 and 720 inclusively. Both settings include a random delay that does not exceed one hour.

    image

    Figure 1 Background upload policy setting in Windows 7

    Background uploading only occurs with Roaming User profile. Also, background uploading does not alter uploading the entire profile when the user logs off. It is important to remember that background uploading only uploads the user’s registry settings (ntuser.dat).

    Now those subtle changes made in the registry can be uploaded to the server while the user remains logged on. Pretty cool.

    Mike “the other Mike Stephens” Stephens

  • USMT 4.0: Cryptic Messages with Easy Fixes

    Ned here again. As you begin the process of moving off your older operating systems to Windows 7 you may end up using USMT 4.0. With its powerful capabilities and intended audience of IT professionals, the tool can be unforgiving and often gives inscrutable messages. Today I’ll cover a few of the most common but unclear errors people see when they first run the tool.

    Return code 27: Invalid store path

    You ran a scanstate and now you’re trying to restore the data to a new target computer. You run:

    Loadstate.exe c:\store\usmt /auto

    And…you get a store error. Examining the loadstate log shows:

    [0x000000] USMT Started at 2010/01/31:15:53:58.796
    [0x000000] Command line: loadstate c:\store\usmt /auto
    [0x000000] Activity: 'MIGACTIVITY_COMMAND_LINE_PROCESSING'
    [0x000000] Failed.[gle=0x00000002]
    [0x000000]   An error occurred processing the command line.
      Invalid store path; check the store parameter and/or file system permissions[gle=0x00000002]
    [0x000000] USMT Completed at 2010/01/31:15:53:58.812[gle=0x00000002]
    [0x000000] Entering MigShutdown method
    [0x000000] Leaving MigShutdown method

    But you look and that path definitely exists with a valid store MIG file inside it.

    This is an easy mistake to make; when you specify a store path during scanstate, USMT automatically makes a sub-folder called “USMT” that contains your data. Admins often assume they need to specify the full path - you don’t. Just provide a path to the same folder level that was used in the scanstate. So in the above case:

    Loadstate.exe c:\store /auto

    Return code 28: Unable to find a script file specified by /i

    You are running a scan or load with a perfectly valid command-line, but every time you start it:

    image

    You get error 28. You examine the directory where scanstate.exe is located and the XML files are definitely there. You examine the scanstate.log and see:

    [0x000000] USMT Started at 2010/01/31:15:59:28.118
    [0x000000] Command line: c:\temp\usmt\scanstate.exe c:\store /v:5 /i:migdocs.xml /i:migapp.xml /o
    [0x000000] Activity: 'MIGACTIVITY_COMMAND_LINE_PROCESSING'
    [0x000000] Script file is not present: 'C:\users\std\migdocs.xml'[gle=0x00000002]
    [0x000000] Failed.[gle=0x00000002]
    [0x000000]   An error occurred processing the command line.
      Unable to find a script file specified by /i[gle=0x00000002]
    [0x000000] USMT Completed at 2010/01/31:15:59:28.149[gle=0x00000002]
    [0x000000] Entering MigShutdown method
    [0x000000] Leaving MigShutdown method

    This error is returned because both scanstate.exe and loadstate.exe assume that the working folder is the current directory -- where the command is execute -- and not the directory wherein the executable is located. Since the XML files do not exist in my profile -- the current directory -- the command fails. To fix the issue, first switch to the directory containing your USMT files using the CD command, or specify full paths to the XML files with each /i.

    Return code 26: Software malfunction or Unknown exception

    Hooboy, that is a really helpful error. You examine the scanstate log and see:

    [0x000000] USMT Started at 2010/01/31:16:11:54.383
    [0x000000] Command line: scanstate c:\store /o /i:miguser.xml /i:migapp.xml /v:5
    [0x000000] Activity: 'MIGACTIVITY_COMMAND_LINE_PROCESSING'
    <snip>
    [0x000000] Entering MigStartupOnline method
    [0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
    [0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
    [0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
    [0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
    [0x080405] EnablePrivilege: AdjustTokenPrivileges failed (Error:0x514)
    [0x0803ac] Initializing online WinNT platform (Read/Write)
    [0x080000] Platform is not using admin privileges

    Aha! As you hopefully know from your reading, USMT requires you to be an Administrator to run the scanstate with full functionality or to run loadstate at all. The problem here is that while you are logged on with an account that is a member of the Administrators group, User Account Control is turned on and this CMD prompt is not elevated. That means that your user account shows up as an Administrator but can’t do anything. If you were to lookup that 0x514 error you’d see it means “Not all privileges or groups referenced are assigned to the caller”. Indeed, they are not.

    Elevate your CMD prompt, use an account that bypasses UAC, or turn off UAC – the issue will go away.

    User running the scanstate is the only profile ever migrated

    Why. Do. You. Ignore. My. Command. Line?!? ARRRRGGGHHHH!!!

    This one is just unfair. As I mentioned above, you must be an administrator to run loadstate – it gives you a very specific error otherwise:

    image

    But if you run scanstate as a user that is not a member of the Administrators group, you get no errors. Goo gets gathered up; everything seems fine.

    image

    Except that nothing you put on the command-line for /ui is considered at all, and if you were to closely examine the store data, you’d find tons of operating system settings missing. I have two different accounts on this computer that contain “ned” in some way, that examination should have gotten them both.

    Yes, this is by design -- more information is tucked away here. There is almost no good reason to run any USMT command as a standard user; just get into the habit of using an (elevated) administrator.

    Conclusion

    Migrating your computers to Windows 7 can be a very interesting exercise. Hopefully this blog post makes things a bit easier.

    Oh - and why did I only say “may end up using USMT 4.0” at the start of this post? Because you have other options:

    Until next time.

    - Ned “The Riddler” Pyle

  • Friday Mail Sack – Mogwai Edition

    Hi folks, Ned here again. This week we hunt down some documentation gremlins and give them a well-deserved smack.

    Also, things will be a bit slow next week as I will be out in Redmond teaching this rotation of Microsoft Certified Masters. Never heard of it? If you’re at the IT career tipping point, this may be just what the doctor ordered. No really, it is, and I will be there!

    Question

    What exactly does the dcdiag.exe /fix command do? According to this it fixes the SPNs on the DC machine account. But according to this it ensures that SRV records are appropriately registered (I thought the NetLogon service did this?!). And what exactly does the netdiag.exe /fix command do? This article says it "fixes minor problems", whatever that means.

    Answer

    1. Dcdiag /fix writes back the computers account’s AD replication SPN (DRSUAPI with an index value of “E3514235-4B06-11D1-AB04-00C04FC2DCD2”) entry only. More info on this SPN here:

    http://msdn.microsoft.com/en-us/library/dd207876(PROT.13).aspx
    http://msdn.microsoft.com/en-us/library/ee791539(PROT.10).aspx

    If someone (else!) has destroyed all the other SPN’s, you will need to recreate them or restart whichever service recreates them. For example if the DFSR SPN goes missing, you restart the DFSR service and it will get put back.

    image

    image

    2. Netdiag /fix reads the %systemroot%\system32\config\Netlogon.dns file and attempts to register all records in DNS.

    I confirmed both in source code, regardless of what old TechNet goo states. :-)

    Question

    In Win2008 DFSR has been improved regarding the asynchronous RPC connections and 16 concurrent connections for upload and download. Do you have any further info on how improved the performance will be from Win2003 R2 to Win2008/2008 R2? Are there any other factors that would drive me to start rolling out the later OS versions?

    Answer

    I will be posting posted some new info about performance improvements in 2008/2008 R2 as well as registry tuning options in the coming weeks. But we don’t have any specific case studies that I am aware of yet – I’ll see if I can find them, and if you do, feel free to comment. We do have some rather unspecific ones, if you’re interested.

    From testing and customer experience though, we see anywhere from a 4 to 20 times performance improvement of 2008 over 2003 R2, depending on a variety of factors that are often very customer specific (network speed, bandwidth, latency, loss rates, errors, overall uptime + memory + CPU + disk subsystem + drivers). Not only did DFSR improve, but the OS got improvements and it makes better use of newer hardware. Besides the RPC and other changes, Win2008 tweaks the DFSR credit manager, and 2008 R2 really improves it – much more evenly-distributed replication with greatly lowered chance of servers being starved by updates.

    Other factors:

    1. Win2003 enters extended support on July 13 2010. This means no further hotfixes that improve reliability or performance, and 5 years of crossed fingers until end of life.
    2. You would now have the option on DC’s to switch to DFSR-enabled SYSVOL and no longer use FRS there.
    3. If deploying 2008 R2, you would also gain read-only and cluster support, which is unavailable in 2003/2008.

    Question

    I am using your old blog post on making custom registry changes and…

    Answer

    Ewwwww… The only reason to use that old document is if you are still running Windows 2000 somewhere. Otherwise you should be busting out Group Policy Preferences and wowing your friends and family.

    Oh, and really? You’re running Win2000? That’s very uncool of you…

    Question

    I am doing USMT migrations with /SF. What is that switch and why are my migrations absolutely busted to heck?

    Answer

    This one came in late last week and was so gnarly that it ended generating a whole blog post. Read more here. Sometimes your questions to us generate more than a Friday reply.

    Good work, Internets!

    – Ned “3 important rules” Pyle

  • Friday Mail Sack – Missed Week Edition

    Hiya folks. The mail sack was a no-show last week since I was out of town; I hope you can find it in your heart to forgive me. If not… well, you get what you pay for. To make up for it, this one is longer than usual. Here are some interesting issues from the past two weeks (both from the Internet and internally too).

    Question

    I’m replicating files with DFSR. I’ve found a large number of files that have had the TEMPORARY attribute set on them. I know that DFSR won’t replicate those files, so I am going to remove that attribute. What happens after I do that?

    Answer

    Short answer: they will replicate. :-)

    Long answer: DFSR treats temporary files like they do not exist. This means if a file was normal and replicated between servers, then someone sets the temporary attribute on that file, it is no longer there as far as DFSR is concerned. When a file no longer exists, we treat it as  - wait for – deleted! So the downstream servers from the originating change are going to delete that file.

    Here is the debug log from the server where the temporary attribute was just set:

    20100325 17:48:46.067 1836 USNC  2881 UsnConsumer::TombstoneOrDelete LDB Updating ID Record: <— Hmm, looks like it’s being treated as a deletion to me.
    +    fid                            0x700000000EB52 <— Note the File ID
    +    usn                             0xfa1ead8
    +    uidVisible                      1
    +    filtered                        0
    +    journalWrapped                  0
    +    slowRecoverCheck                0
    +    pendingTombstone                0
    +    internalUpdate                  0
    +    dirtyShutdownMismatch           0
    +    meetInstallUpdate               0
    +    meetReanimated                  0
    +    recUpdateTime                   20100325 21:47:46.739 GMT
    +    present                         0
    +    nameConflict                    0
    +    attributes                      0x20
    +    ghostedHeader                   0
    +    data                            0
    +    gvsn                            {16EC84D2-ECEE-4E0D-B6FA-0C4137F65EE4}-v277 <— Note the version
    +    uid                             {F5C8EE4D-3C55-42A4-BB62-63F72E7EEBED}-v50
    +    parent                          {780F3375-CDB0-4583-A3AE-85845086E884}-v1
    +    fence                           Default (3)
    +    clockDecrementedInDirtyShutdown 0
    +    clock                           20100325 21:48:46.067 GMT (0x1cacc64f1e63a10)
    +    createTime                      20100325 21:47:29.386 GMT
    +    csId                            {780F3375-CDB0-4583-A3AE-85845086E884}
    +    hash                            E30F7F47-58F6FE00-B9846C04-43B82E2A
    +    similarity                      00000000-00000000-00000000-00000000
    +    name                            ohnoes!.txt <— Note the name
    +   
    20100325 17:48:46.067 1836 USNC  2887 UsnConsumer::TombstoneOrDelete ID record tombstoned from USN_RECORD:
    +    USN_RECORD:
    +    RecordLength:        96
    +    MajorVersion:        2
    +    MinorVersion:        0
    +    FileRefNumber:       0x700000000EB52 <— Same FID
    +    ParentFileRefNumber: 0x1AD000000008E29
    +    USN:                 0xfa1ead8
    +    TimeStamp:           20100325 17:48:46.067 Eastern Standard Time
    +    Reason:              Basic Info Change Close
    +    SourceInfo:          0x0
    +    SecurityId:          0x0
    +    FileAttributes:      0x120 <— This is 0x20 && 0x100, which is a file with the archive bit and the temporary attribute set.
    Testify!
    +    FileNameLength:      30
    +    FileNameOffset:      60
    +    FileName:            ohnoes!.txt <— Same name

    20100325 17:48:46.114 2304 JOIN  1244 Join::SubmitUpdate Sent: uid:{F5C8EE4D-3C55-42A4-BB62-63F72E7EEBED}-v50 gvsn:{16EC84D2-ECEE-4E0D-B6FA-0C4137F65EE4}-v277 name:ohnoes!.txt connId:{2F726EBA-3970-4AA5-AD23-4C07600CE427} csId:{780F3375-CDB0-4583-A3AE-85845086E884} csName:condelrf <— Same name, same version being sent to the server’s partner. Since the last operation was delete, guess what we’re telling the other server to do?

    Then later, I remove the temporary attribute:

    20100325 17:49:07.523 1836 USNC  2703 UsnConsumer::CreateNewRecord LDB Inserting ID Record: <— Well hello there! Just like a new file got added. It will replicate out momentarily…
    +    fid                             0x700000000EB52
    +    usn                             0xfa1eb98
    +    uidVisible                      0
    +    filtered                        0
    +    journalWrapped                  0
    +    slowRecoverCheck                0
    +    pendingTombstone                0
    +    internalUpdate                  0
    +    dirtyShutdownMismatch           0
    +    meetInstallUpdate               0
    +    meetReanimated                  0
    +    recUpdateTime                   16010101 00:00:00.000 GMT
    +    present                         1
    +    nameConflict                    0
    +    attributes                      0x20
    +    ghostedHeader                   0
    +    data                            0
    +    gvsn                            {16EC84D2-ECEE-4E0D-B6FA-0C4137F65EE4}-v278
    +    uid                             {16EC84D2-ECEE-4E0D-B6FA-0C4137F65EE4}-v278
    +    parent                          {780F3375-CDB0-4583-A3AE-85845086E884}-v1
    +    fence                           Default (3)
    +    clockDecrementedInDirtyShutdown 0
    +    clock                           20100325 21:49:07.523 GMT (0x1cacc64feb0277e)
    +    createTime                      20100325 21:49:07.523 GMT
    +    csId                            {780F3375-CDB0-4583-A3AE-85845086E884}
    +    hash                            00000000-00000000-00000000-00000000
    +    similarity                      00000000-00000000-00000000-00000000
    +    name                            ohnoes!.txt

    Craig Landis has written all kinds of interesting stuff about the temporary file attribute and DFSR in this old blog post here and now in the new TechNet Wiki here. Maybe someone will update the Wiki with this new tidbit? Hmmm…. could be you!

    Question

    Is there a way to make the “AD Users and Computers” snap-in (DSA.MSC) always open in “Advanced Features View” by default? I can’t find any command-line switches or registry settings that handle this scenario.

    Answer

    Oh MMC, how you taunt us. ADUC stores all of its configuration info inside of a special (read: not well documented and highly proprietary) cache file in your profile. You can see it by opening this folder:

    %appdata%\microsoft\mmc

    Good luck reading the contents of the DSA file though. The setting is stored in there as binary goo. I suppose you could copy your cache file around from place to place and use that, but seems like it would be a rather risky operation with little gain over the the 2 seconds it takes to click:

    image

    Perhaps a better idea is to have an "Admin” Terminal Server that has all your favorite applications configured the way you like ‘em. Maybe even using RemoteApp.

    Question

    I am looking for documentation on how the Root CA communicates with the Sub CAs in a Microsoft PKI. Once the self-signed certificate has been put on the Root CA, is it enrolled or replicated to the Sub CAs? I guess the question is this: once the self signed certificate has been put on the Root CA is it enrolled or replicated to the Sub CAs? We have NPS installed and are using it as a Radius server for the AP's and Wireless LAN Controller. We have a 802.1x policy configured and have the NPS Server validating the wireless client. The client is getting a Wi-Fi policy and a PKI-Policy through a Domain Group Policy. In this environment the wireless clients are getting a certificate with a PKI Policy but when you look at the certificate on the laptop once the system has been logged on the certificate could be issued from any of the Sub CAs. Is this the normal process?

    Answer

    Root CAs and subordinate CAs don't "communicate" with each other in quite that fashion. In fact, the only time any user or computer (sub CAs included) would need to contact the Root CA, outside of management tasks, is if they wanted to request a certificate. Within the sphere of the Windows CA itself, the answer to your question is no, the root CA certificate is not automatically propagated to subordinate CAs. This task must be done via another process.

    If the Windows root CA has been installed on a domain server then the CA will automatically publish its root CA into Active Directory. Every Windows workstation or server will automatically retrieve the root certificate once its presence is detected and thus trust the new CA. If the Windows root CA was installed on a non-domain server then the root certificate must be published to Active Directory manually. Since you mentioned that enrollment is occurring successfully, I don't believe that trusting the root CA is an issue for you.

    The next thing to address is how Windows client select the CA from which they request a certificate. The following discussion assumes that CEP/CES has not been installed and that CAs are installed in Enterprise Mode. This process applies to any version of Windows from Windows XP onwards. First, there are two contexts in which certificates can be requested -- the user context or the local computer context. Users can request certificates manually via the Certificates MMC snap-in or the Web Enrollment pages, or certificates may be assigned by an Administrator and distributed by Autoenrollment. Computer certificates can also be requested manually by a local Administrator using the Certificates MMC snap-in, or also by Autoenrollment. Regardless of the context, or how enrollment occurs, the process is the same.

    1. The Windows Certificate Client (WCC) queries Active Directory for a list of certificates for which the account (user or computer) has Enroll (or Autoenroll) permissions. This information may be cached in the registry, but is periodically refreshed.
    2. The WCC then queries Active Directory for a list of available CAs in the forest. Each CA publishes an Enrollment Services object that holds the FQDN of the CA server and the list of templates available on that CA.
    3. The list of templates and the list of CAs (and their configured templates) are combined to build a list of those templates for which the account has permission to request a certificate AND which are available on some trusted CA in the forest.
    4. In the case of the Certificates MMC snap-in, this combined list is presented in the UI as a list from which the user can select. In the case of Autoenrollment, this list is compared against those templates assigned to the account via Group Policy.
    5. Once a template and the CA have been selected, the public/private key pair is created, the request is generated, and the request is submitted to the CA. Depending on the template settings, the request may be fulfilled immediately or it may be pended so that the request can be manually approved.

    What this means for you is that you need to determine the following:

    1. Is the Root CA in Enterprise or Standalone mode? If the latter, it will not issue certificates based on templates which eliminates the ability to request certificates via the MMC, or through Autoenrollment.
    2. Do you have a certificate template that you assign specifically through your PKI-Policy? If so, on which CAs is that template configured? Check the Certificate Templates folder in the Certificate Services snap-in. (HINT: If you don't have one, the CA is in Standalone mode.)

    So, if your root CA isn't configured to issue certificates based on your PKI-Policy template, or is in Standalone mode, but your subordinate CAs *are* configured to issued certificates based on that template, then it's no surprise all the certificates you've looked at were issued from one of the subordinate CAs. Funnily, this is actually a good thing. The purpose of a root CA is to be a Trust Anchor for your entire PKI. It issues certificates to subordinate CAs, which in turn issue certificates to users and computers on your network.

    If you want an old but still decent introduction to PKI concepts check out Microsoft Windows 2000 Public Key Infrastructure (http://msdn.microsoft.com/en-us/library/ms995346.aspx). If you want the latest and greatest info and a lot more depth, you should pick up the book "Windows Server 2008 PKI and Certificate Security" by Brian Komar (http://www.microsoft.com/learning/en/us/book.aspx?ID=9549&locale=en-us)

    [If you can’t tell, this came from Jonathan, the man who never uses a sentence when a paragraph will do – Ned]

    Question

    Hey, there are a bunch of different companies that make software to do X. Can you recommend one?

    Answer

    Nope! If I recommend one over a bunch of others, the others can get really mad. And in case you haven’t noticed, we, ummm, get sued a lot. Even if the other ones don’t sue and only complain loudly over the Internet, and shaking that bush is a real no-no here in Support. So forgive us if we take a pretty hard line and recommend nothing.

    I will say this: any good vendor will be more than happy to let you run a fully functional time-bombed trial edition and give things a whirl. An excellent one will let you access their online knowledgebase and support forums before you buy. I am always suspicious of those who do not want you to see any dirty laundry until the check has cleared. Everyone has a competitor; there are no real monopolies in the software world. Use that to your advantage when evaluating new products.

    Question

    On a Windows Server 2008 R2 computer I can only get back to Vista for compatibility mode (this computer is running some old stuff via remote desktop services). Why is this and is there a way to at least get back to XP mode?

    Answer

    You tricked yourself – it depends on the application architecture you are running:

    clip_image001 clip_image001[5]

    The one on the left is a 32-bit program. The one on the right is a 64-bit program. Both are on the same computer. Win2008 R2 runs 64-bit only and when it comes to 64-bit programs, can only pretend back as far as Vista 64-bit RTM. If you need more than this you will need to pay a visit to the Application Compatibility Toolkit. Or, you know, upgrade your crummy old app… ^_^

    Hey, sometimes the questions aren’t about DS necessarily. They just need to be interesting enough to make me care. In this case I had no idea, had to figure it out.

    Until next time.

    - Ned “get to da choppah!” Pyle