Blog - Title

July, 2011

  • Friday Mail Sack: Anchors Aweigh Edition

    Hiya folks, Ned here again. I finally have an editor that allows anchors on all the questions, so I am adding a quasi “table of contents” for these posts that allow easier navigation and linking. I’ll retrofit all the old mail sack articles too… eventually. This week we discuss – eh - let’s have the bullets do the talking:


    We are trying to move away from NTLM in our Active Directory environment. I read your previous post on NTLM Auditing for later blocking. However, the blog posting does not differentiate between the two versions of NTLM. What would be the best way to audit for only NTLMv1 or LM? Also, will Microsoft ever publish those two TechNet articles?


    I still suggest you give up on this, unless you want to spend six months not succeeding. :) If you want to try though, add security event logging on your Win2008 R2 servers/DCs for 4624 Logon events:

    977519  Description of security events in Windows 7 and in Windows Server 2008 R2;EN-US;977519

    Those will capture the Package Name type. For example:


    Best I can tell, those two TechNet articles are never going to be published. Jonathan is trying yet again as I write this. Maybe Win8...? We'll see...


    I am now 100% Windows Server 2008 R2 in my domains and am ready to move my Domain and Forest Functional Levels to 2008 R2. What does that and my new schema buy me, and are there any steps I should do in special order?


    Nothing has to happen in any special order. Some of your new AD-related options include:

    1. AD Recycle Bin ( )
    2. DFSR for SYSVOL ( )
    3. V2 DFS Namespaces ( ) and migrate existing V1 namespaces ( )
    4. Last Interactive Logon ( )
    5. Fine Grain Password Policies – ( )
    6. Virtual Desktops ( )
    7. Managed Service Accounts with automatic SPN management ( )
    8. Other things we recommend at the end of the upgrade (  )

    With your awesome Win2008 R2 servers, you can also:


    Our USMT scanstate log shows error:

    Error  [0x08081e] Failed to load manifest at C:\USMT\x86\dlmanifests\ XmlException:  hResult = 0x0, Line = 18, Position = 31; A string literal was expected, but no opening quote character was found.

    But nothing bad seems to happen and our migration has no issues we can detect. It looks like the quotation marks in the XML are incorrect. If I correct that, it runs without error, but am I making something worse and is this supported?


    Right you are. Note the quotation marks – looks like some developer copied them out of a rich text editor at some point:


    But no matter – you can change it or delete that MAN file, it makes no difference. That manifest file does not have a USMT scope set, so it is never used even when syntactically correct. In order for USMT to pick up a manifest file during scanstate and loadstate, it must have this set:

      <migration scope="Upgrade,MigWiz,USMT">

    If not present, the manifest is skipped with message “filtered out because it does not match the scope USMT”:


    Roughly two thirds of the manifests included with USMT are not used at all for this very same reason.


    I am looking for a full list of Event IDs for FRS. KB 308406 only seems to include them for Windows 2000 – is that list accurate for later operating systems like WIndows Server 2003 or 2008 R2?


    That KB article has a few issues, I’ll get that ironed out. In the meantime:

    Windows Server 2003 added events:

    Event ID: 13569

    Severity: Error

    The File Replication Service has skipped one or more files and/or directories during primary load of the following replica set. The skipped files will not replicate to other members of the replica set.

    Replica set name is    : "%1"

    A list of all the files skipped can be found at the following location. If a directory is skipped then all files under the directory are also skipped.

    Skipped file list      : "%2"

    Files are skipped during primary load if FRS is not able to open the file. Check if these files are open. These files will replicate the next time they are modified.


    Event ID: 13570

    Event Type: Error

    The File Replication Service has detected that the volume hosting the path %1 is low on disk space. Files may not replicate until disk space is made available on this volume.

    The available space on the volume can be found by typing

    "dir /a %1".

    For more information about managing space on a volume type "copy /?", "rename /?", "del /?", "rmdir /?", and "dir /?".


    Event ID: 13571

    Event Type: Error

    The File Replication Service has detected that one or more volumes on this computer have the same Volume Serial Number. File Replication Service does not support this configuration. Files may not replicate until this conflict is resolved.

    Volume Serial Number : %1

    List of volumes that have this Volume Serial Number: %2

    The output of "dir" command displays the Volume Serial Number before listing the contents of the folder.


    Event ID: 13572

    Event Type: Error

    The File Replication Service was unable to create the directory "%1" to store debug log files.

    If this directory does not exist then FRS will be unable to write debug logs. Missing debug logs make it difficult, if not impossible, to diagnose FRS problems.

    Windows Server 2008 added no events.

    Windows Server 2008 R2 added events:

    Event ID: 13574

    Event Type: Error

    The File Replication Service has detected that this server is not a domain controller. Use of the File Replication Service for replication of non-SYSVOL content sets has been deprecated and therefore, the service has been stopped. The DFS Replication service is recommended for replication of folders, the SYSVOL share on domain controllers and DFS link targets.


    Event ID: 13575

    Event Type: Error

    This domain controller has migrated to using the DFS Replication service to replicate the SYSVOL share. Use of the File Replication Service for replication of non-SYSVOL content sets has been deprecated and therefore, the service has been stopped. The DFS Replication service is recommended for replication of folders, the SYSVOL share on domain controllers and DFS link targets.


    Event ID: 13576

    Event Type: Error

    Replication of the content set "%1" has been blocked because use of the File Replication Service for replication of non-SYSVOL content sets has been deprecated. The DFS Replication service is recommended for replication of folders, the SYSVOL share on domain controllers and DFS link targets.

    All  OSes included event:

    Event ID: 13573

    Event Type: Warning

    File Replication Service has been repeatedly prevented from updating:

    File Name : "%1"

    File GUID : "%2"

    due to consistent sharing violations encountered on the file. Sharing violations occur when another user or application holds a file open, blocking FRS from updating it. Blockage caused by sharing violations can result in out-of-date replicated content. FRS will continue to retry this update, but will be blocked until the sharing violations are eliminated.  For more information on troubleshooting please refer to

    Win2008 should have had those 13574-13576 events as they are just as applicable, but $&% happens.


    Why isn’t it possible to grant local admin rights to a domain controller without added them to the built-in Administrators or Domain Admins groups? It can be done on RODCs, after all.


    It’s with good intentions – if I am a local administrator on a DC, I own that whole domain. I can directly edit the AD database, or even replace it with my own copy. I can install a filter driver that intercepts all password communications between LSASS and the database. I can turn off all auditing and group policy. I can add a service that runs as SYSTEM and therefore, runs as the DC itself – then impersonate the DC. I can install a keyboard logger that captures the “real” domain admins as they logon. My power is almost limitless.

    The reasons we added the functionality for non-domain admin administrators on RODC are:

    1. RODCs are not authoritative for anything and cannot originate any data out to any other DC or RODC. So the likelihood of damage or compromise is lower - although theoretically, not removed.
    2. RODCs are for branch offices that don’t have dedicated IT staff and which may not even be reliably network connected to the main IT location – so having a “local admin” makes sense for management.


    You have talked about how to track individual DFSR file replication using its built-in “enable audit” setting. Does this impact server performance?


    Yes and no. The additional DFSR logging impact is negligibly low on any OS. The object access auditing impact ranges from medium (by default) to high (if you have added many custom SACLs). You have to enable the object access auditing to use the DFSR logging on Win2003 though, so the net result there is medium to high impact when compared to other auditing categories.

    It’s worth noting that overall, auditing impact in Win2008+ is lower, as the audit system was redesigned for greater scalability and performance. You also have a much less disruptive security audit option, which is to enable only the subcategory:

    Category: Object Access
    Subcategory: Application Generated


    That way you don’t have to enable the completely ridiculous set of Object Access auditing in order to track only DFSR file changes. And the impact is greatly lowered.


    And besides, to run Win2008+, you need much faster hardware anyway. ^_^


    Can Netware volumes be DFSN Link Targets?


    Good grief, someone still has Netware servers?

    Yes, with caveats:

    824729  Novell 6 CIFS pass-through authentication failures;EN-US;824729

    Novell also created a DFS service, to act as a root instead of simply a link target like above:

    Using DFS to Create Junctions

    Generally speaking, if a target can provide SMB/CIFS shares, they can be a link target. To connect to a DFS target, your OS needs a DFS client:

    Can Apple, Linux, and other non-MS operating systems connect to DFS Namespaces?

    Bring on the Banyan Vines questions!


    There is no later version of the Group Policy Best Practices Analyzer tool and no updates when it starts. Is it going to be updated for Windows Server 2008 or later? The tool was even mentioned by Tim on this very blog years ago, but since then, nothing.

    [This “question” came from a continued conversation about a specific aspect of the tool – Ned]


    • This tool has no updates or development team and is effectively abandoned. It was not created by the Group Policy Windows developer group nor is it maintained by them – it doesn’t have a dev team at all. It probably should have released in CodePlex instead of the download center. The genie cannot be put back in the bottle now though, as people will just grab copies from elsewhere on the internet, likely packed with malware payloads.
    • This tool is not supported – it’s provided as-is. When Tim talked about it, the tool had a bright future. Now it is gooey dirt.
    • This tool’s results and criteria are questionable, bordering on dangerous. It gives a very false sense of security if you pass, because it checks very little. It also incorrectly flags issues that do not exist – for example, it states that the Enterprise Domain Controllers group does not have Apply GP permissions to the Default Domain Controllers policy, and this is an error. The DCs are all members of Authenticated Users though, and that’s how they get Apply permissions. And why doesn’t it raise the same flag for the default domain policy? Who knows! The developers were not correct in this design or assumptions. The tool recommends you add more RPC ports for invalid reasons, which is silly. It talks about a few security settings, ignoring hundreds of others and giving no warning that changing these can break your entire environment. Gah!

    If you are looking for security-related best practice recommendations for group policy, you should be using the Security Compliance Manager tool:

    Microsoft Security Compliance Manager v1 (release)

    Microsoft Security Compliance Manager v2 (beta)

    That tool at least has best effort support and a living dev team that is providing vetted recommendations.

    More Comic-Con Cosplay

    As you know, I spent last week at San Diego Comic-Con and even showed some pictures I snagged. Here is more amazing cosplay, courtesy of the rad (click thumbnails to make with the bigness). And check out the eyes on Scorpion.






    1927997-img_5915_super1927975-img_5938_super – go there now, unless you hate awesomeness

    Until next time.

    Ned “I should go as a Keebler Elf next year” Pyle

  • Troubleshooting SID translation failures from the obvious to the not so obvious

    Hi guys, Joji Oshima here with my first post. A common problem we see is SID translation failure. The problem usually occurs when you add users or groups from a trusted domain into your domain local groups.

    What you hope to see is the friendly names of the users, and their domain:


    Unfortunately, you only see the Security Identifiers (SIDs): A security identifier (SID) is a unique value used to identify a security principal or group in a Windows operating system


    So what goes on in the background that allows SID translation to occur? Some people assume this translation process takes place using LDAP queries, but in Windows there are APIs that will translate the SID to the friendly name. Let’s take a quick look at the sequence of events that take place during SID translation with the commonly used LsarLookupSids3 function.

    SID/Name Translation:

    • System will bind to EPM* (port 135) of the target domain controller
    • System will poll EPM for a dynamic port to connect into LSARpc(LDAT/LSAD)
    • System will call LSAT* LsarLookupSids3 for each SID, and receive the name associated with it

    The End Point Mapper (EPM) points clients to a dynamic RPC port that a service is listening on. LSAT is a protocol that allows clients to translate security identifiers (SIDs) into human readable names and vice-versa.


    Something along the process is blocking the request from completing. Before going too deep into the troubleshooting process, check to see if the necessary ports are open between your system and the domain controller. PortQryUI is a great tool to check if ports are open between two systems.


    Assuming the ports are open, there is some other piece blocking the translation. Most commonly, we will see this when there is a one way trust involved and anonymous translations are blocked. You can easily allow anonymous SID/Name translation in Group Policy. This policy is only applied to Domain controllers since they are the servers that will actually process the translation request. You will find this policy in Computer Configuration->Windows Settings->Security Settings->Local Policies->Security Options. It should be noted that by enabling this policy, domain controllers will allow translations to occur even if the user is anonymous or sends bad credentials. It is possible that this can be exploited by a malicious user to gain usernames for administrative accounts. No supported version of Windows needs this setting enabled, so it would only be a troubleshooting step for applications not included with the OS.


    Network access: Allow anonymous SID/Name translation

    Once this policy is enabled, be sure to allow time for SYSVOL replication to occur and group policy to refresh. You can check to see if the policy is enabled on your target DC by running GPRESULT /h gpresult.html. There are some instances where translation is still not occurring. At this point, the best way to troubleshoot is with a network capture from both sides. You can take a network trace using Microsoft Network Monitor 3.4.

    Microsoft Network Monitor 3.4


    I’ve pulled some select lines out of a trace from a case that I was troubleshooting that went beyond the policy setting:

    2723 1:05:12 PM 6/13/2011 49.1983596 MSRPC MSRPC:c/o Bind: EPT(EPMP) UUID{E1AF8308-5D1F-11C9-91A4-08002B14A0FA} Call=0x3 Assoc Grp=0xE5B51 Xmit=0x16D0 Recv=0x16D0 {MSRPC:393, TCP:391, ESP:44, IPv4:39}

    2724 1:05:12 PM 6/13/2011 49.2383211 MSRPC MSRPC:c/o Bind Nack: Call=0x3 Reject Reason: REASON_NOT_SPECIFIED {MSRPC:393, TCP:391, ESP:44, IPv4:39}

    Notice that the bind attempt to the EPM (end point mapper) is getting a Bind Nack for REASON_NOT_SPECIFIED. In this case the customer had an end point protection software suite that was blocking the connection. One thing to note is simply disabling most security software is not enough to fully stop its inspection behaviors. The filter drivers are still loaded and will continue to manipulate these connections. A full uninstall is the only way to ensure it is nullified. After removing the software package on both sides, I took another trace to see if it gets past this part.

    434 9:44:01 AM 6/14/2011 10.8427423 MSRPC MSRPC:c/o Bind: EPT(EPMP) UUID{E1AF8308-5D1F-11C9-91A4-08002B14A0FA} Call=0x2 Assoc Grp=0x0 Xmit=0x16D0 Recv=0x16D0 {MSRPC:62, TCP:61, ESP:52, IPv4:51}

    435 9:44:01 AM 6/14/2011 10.8811941 MSRPC MSRPC:c/o Bind Ack: Call=0x2 Assoc Grp=0xA179 Xmit=0x16D0 Recv=0x16D0 {MSRPC:62, TCP:61, ESP:52, IPv4:51}

    436 9:44:01 AM 6/14/2011 10.8813273 MSRPC MSRPC:c/o Auth3: Call=0x2 {MSRPC:62, TCP:61, ESP:52, IPv4:51}

    437 9:44:01 AM 6/14/2011 10.8813768 EPM EPM:Request: ept_map: NDR, LSARpc(LSAT/LSAD) {12345778-1234-ABCD-EF00-0123456789AB} v0.0, RPC v5, (0x87) [DCE endpoint resolution(135)] {MSRPC:62, TCP:61, ESP:52, IPv4:51}

    439 9:44:01 AM 6/14/2011 10.9205440 MSRPC MSRPC:c/o Fault: Call=0x2 Context=0x0 Status=0x5 Cancels=0x0 {MSRPC:62, TCP:61, ESP:52, IPv4:51}

    Notice that the bind attempt to the EPM (end point mapper) is now successful with the Bind Ack. After this, the system makes an EPM request for LSARpc(LSAT/LSAD), but the system gets a MSRPC c/o fault with a status of 0x5. 0x5 typically stands for ERROR_ACCESS_DENIED.

    At this point it looks like something is still blocking this request from completing. We already removed the end point protection and tried enabling anonymous SID/Name translation.

    Gathering a GPRESULT /H from both sides, there appeared to be several potential blockages. I also had the user export the following registry keys to review:

    • HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer
    • HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation
    • HKLM\SYSTEM\CurrentControlSet\Control\LSA
    • HKLM\SYSTEM\CurrentControlSet\Services\Netlogon

    Ultimately, there were several pieces that blocked the translation from occurring, which I have listed below. If you have a similar case, review some of these settings and see if RPC is being blocked in one of these ways. Keep in mind that there are legitimate reasons to have these security settings configured to block anonymous connections. I would suggest evaluating the benefits of allowing anonymous connections vs. blocking them.

    This configuration item restricts anonymous access to the RPC interfaces. Keep in mind this can block incoming and outgoing anonymous RPC connections.

    This configuration item controls which named pipes can be accessed anonymously

    Similar to the Null Session Shares, this configuration item can control which named pipes can be accessed anonymously

    This configuration item configures if anonymous clients can connect to translate SIDs into names

    This configuration item can block anonymous users from connecting to a computer remotely and can effect SID to name translation

    Matched LM Compatibility Levels (set to 2 on both sides)

    Since we have an external domain trust, NTLM authentication is used. It is possible that this configuration mismatch will block SID to name translation.

    When resolved, the trace should look similar to this:


    6 5:20:34 PM 6/17/2011 8.9047907 MSRPC MSRPC:c/o Bind: EPT(EPMP) UUID{E1AF8308-5D1F-11C9-91A4-08002B14A0FA} Call=0x1 Assoc Grp=0x0 Xmit=0x16D0 Recv=0x16D0 {MSRPC:3, TCP:2, IPv4:1}

    7 5:20:34 PM 6/17/2011 8.9047907 MSRPC MSRPC:c/o Bind Ack: Call=0x1 Assoc Grp=0x53B3 Xmit=0x16D0 Recv=0x16D0 {MSRPC:3, TCP:2, IPv4:1}

    8 5:20:34 PM 6/17/2011 8.9047907 EPM EPM:Request: ept_map: NDR, LSARpc(LSAT/LSAD) {12345778-1234-ABCD-EF00-0123456789AB} v0.0, RPC v5, (0x87) [DCE endpoint resolution(135)] {MSRPC:3, TCP:2, IPv4:1}

    9 5:20:34 PM 6/17/2011 8.9047907 EPM EPM:Response: ept_map: NDR, LSARpc(LSAT/LSAD) {12345778-1234-ABCD-EF00-0123456789AB} v0.0, RPC v5, (0xC003) [49155] {MSRPC:3, TCP:2, IPv4:1}

    13 5:20:34 PM 6/17/2011 8.9047907 MSRPC MSRPC:c/o Bind: LSARpc(LSAT/LSAD) UUID{12345778-1234-ABCD-EF00-0123456789AB} Call=0x1 Assoc Grp=0x0 Xmit=0x16D0 Recv=0x16D0 {MSRPC:5, TCP:4, IPv4:1}

    15 5:20:34 PM 6/17/2011 9.0610407 MSRPC MSRPC:c/o Bind Ack: Call=0x1 Assoc Grp=0x424B Xmit=0x16D0 Recv=0x16D0 {MSRPC:5, TCP:4, IPv4:1}

    16 5:20:34 PM 6/17/2011 9.0610407 LSAT LSAT:LsarLookupSids3 Request, *Encrypted* {MSRPC:5, TCP:4, IPv4:1}

    17 5:20:34 PM 6/17/2011 9.0766657 LSAT LSAT:LsarLookupSids3 Response, *Encrypted* {MSRPC:5, TCP:4, IPv4:1}


    I hope this makes things clearer and you have some places to look if you are not translating. If you are having this issue, remember the sequence of events, look at a network trace of the failure, and find where it is failing.

    Have a great week.

    - Joji “SID Caesar” Oshima

  • Not IT-related (but geeks will enjoy, regardless)

    Ned here. As you can tell, I’ve been gone for a bit – hang tight, I’m replying to all these emails. In the meantime, here’s what I was up to:

    San Diego Comic-Con 2011

    Oh sure, now you’re interested. :) A sampling of what my Windows Phone 7 camera and I came up with during the four-day awesome-o-thon. These are thumbnails,  click to see them in their enlarged glory.

    Not pictured: a huge number of PG-13 or better worse pictures and about a million more storm troopers…

    People with too much time, not enough shame

    clip_image002clip_image005 clip_image004



     clip_image008 caclip_image009

    Lifesize (or bigger) stuff that must have really sucked to transport and assemble

    clip_image010 clip_image011 clip_image012

    clip_image013 clip_image014clip_image015

    Good grief, the crowds :(


    74 (3)98

    Amazing collectibles (for those with a few thousand bucks)




    The 70’s sci-fi hat trick!


    Even the pedi-cab guys were into it


    The stupidest hotel sign in the whole world – thanks lawyers!

    Finally, the stalking of Andrew WK… no really, he liked me!


    Was anyone else there?

    Ned “waiting in line is fun!” Pyle

  • Friday Mail Sack: Peevish Nediquette Edition

    Hi folks, Ned here again. This week I talk about Vista’s hidden AD schema, SYSVOL migration mission control, kick-starting cached logon performance, USMT c'est la vie, foul-mouthed NetBIOS, DFSR do-over, and the usual random goo. 


    If someone has run the 2008 beta schema that was included the Vista RTM DVD - taking the AD Schema to wacky version 39 - are they hosed or can they install Windows Server 2008 or 2008 R2’s schema upgrades without issues?


    You can upgrade your schema to 44 (Win2008) or 47 (Win2008 R2) without issue. And if this a production domain, do so immediately. The 39 schema was never supported!





    If you compare the sch32.ldf-sch39.ldf schema files that came with the Vista RTM media and the ones on Windows Server 2008 R2 SP1 media, the only difference is sch34.ldf. That contained one extra attribute called ms-DS-Seniority-Index that was not shipped with Windows Server 2008:

    dn: CN=ms-DS-Seniority-Index,CN=Schema,CN=Configuration,DC=X
    changetype: ntdsSchemaAdd
    objectClass: attributeSchema
    ldapDisplayName: msDS-SeniorityIndex
    adminDisplayName: ms-DS-Seniority-Index
    adminDescription: Contains the seniority index as applied by the organization where the person works.
    attributeId: 1.2.840.113556.1.4.1947
    omSyntax: 2
    isSingleValued: TRUE
    systemOnly: FALSE
    searchFlags: 1
    schemaIdGuid:: 1KV7zpDwQUeswxXiyVVV2g==
    attributeSecurityGuid:: VAGN5Pi80RGHAgDAT7lgUA==
    mapiID: 35996
    showInAdvancedViewOnly: TRUE
    systemFlags: 16

    Fortunately, it used a proper registered MS OID so there is no extra likelihood of conflict in the future. If someone argues that you are using a non-standard schema that may someday cause adprep to fail and require you to migrate forests, I say “wait until that day comes and don’t worry about it for now.” It’s been years and no one has complained yet. Besides, your schema contains hundreds of attributes that were included and never used. Heck, even by Win2008, when we were on our third iteration and knew better, we still included dozens of new attributes that went nowhere.


    Should I wait to migrate my SYSVOL from FRS to DFSR after deploying all Win2008 R2 DCs, or just go for it now that I have Windows Server 2008 DCs and 2008 domain functional level?


    I recommend you go for it now - the tendency is to have more DCs over time. Fewer DCs to migrate cuts down on the time, simplifies your life, and makes a smooth migration more likely. And it’s one less thing hanging over your head in your Win2008 R2 upgrade cycle.

    Hotfixes that will matter to Win2008 SP2:

    KB978994 Error message when you try to migrate the SYSVOL share from the FRS to the DFSR service in a Windows Server 2008 domain: "The parameter is incorrect" -;EN-US;978994

    KB2286715 A SYSVOL share migration from FRS to the DFS Replication service stops responding at the Start state in Windows Server 2008 -;en-US;2286715

    KB972105 All files are conflicted on all domain controllers except the PDC Emulator when a DFSR migration of the SYSVOL share reaches the Redirected state in Windows Server 2008 or in Windows Server 2008 R2 -

    If you decide to wait to R2, you still need KB972105 unless you install SP1. If you deploy Win2008 R2 RODCs without SP1, make sure you also install KB2401600 and KB976655.

    Or just use our handy master KB for all of the above.


    I know Windows caches user logon credentials so the user can log on when the domain isn't available (like laptops that employees take home each night). Is there a way to speed up the time it takes Windows before cached logon credentials kick in, so the users can logon faster when not connected to the corporate network?


    Make sure the computer has no network connection at all. If there’s any network (even some crummy home network) the client will try for a short while to get to a DC, as he has no way to know that this network is nonsense. You can try this out yourself by unplugging a desktop or shutting the WIFI antenna off on a laptop, then restarting and trying to logon – it will be lightning fast.


    If using USMT to migrate from XP to Windows 7, are different installed MUI (language packs) supported?


    As long as the localized versions of the OS are the same, you’re supported– i.e. we don’t care if you go from EN-US to EN-US, with one running the French MUI and the other the Italian MUI. Heck, we even migrate the regional settings as part of config.xml*. But if one really was French localized and the other Italian, that’s not supported.

    * See:

    <component displayname="Date, Time, Language and Region" migrate="yes" ID="date_time_language_and_region">

    <component displayname="Regional Language Options" migrate="yes" ID="date_time_language_and_region\regional_language_options">

    <component displayname="Microsoft-Windows-TextServicesFramework-Migration-DL" migrate="yes" ID=""/>

    <component displayname="Microsoft-Windows-MUI-Settings-DL" migrate="yes" ID=""/>

    <component displayname="Microsoft-Windows-International-Core-DL" migrate="yes" ID=""/>


    As far as how well we do this, I have no experience to impart. I have never heard of any issues and found no support cases, but this would be a seriously rare operation…


    If Win2008+ DFS Replication tries to replicate open files and 16 of them are locked at once, would DFSR cease to replicate any further until at least one unlocked?


    No, those files are skipped and retried. DFSR will retry after one second, two seconds, four seconds, up to twelve times with a maximum timeout of 5 minutes between each retry. DFSR will then stop retrying that file for 10 minutes and re-enter the same retry cycle infinitely. The DFSR events are throttled and the events doesn’t match the actual number of retries.
    Locked files definitely have a negative effect – all these retries aren’t free – but you will not completely block behind 16 files through bad luck.


    I was reading KB2491951, which states that you cannot install Exchange 2010 SP1 if “the NetBIOS domain name of a domain controller contains an Ampersand (&) character.”

    Is it possible for a DC to have an ampersand in its name? Or am I confusing the issue?


    The KB terminology is misleading (that’s being fixed). It refers to Exchange 2010 SP1 installation failure if the domain’s NetBIOS name contains an ampersand. Not the DC itself.

    Since Windows 2000 SP1, you have not been able to name a computer with an ampersand. Trying to will give you variations on this error:


    Prior to that, you could use many wacky characters in the name, as WINS and NetBIOS name resolution don’t care. If you managed to do that and then started using DNS, we changed the characters to hyphens (dashes) when registering A records in order to comply with RFC 952.

    For what it’s worth, NetBIOS domain names can contain unicode characters, numbers, and the symbols:

    ! @ # $ % ^ & ' ) ( - _ { } ~

    They cannot contain symbols:

    \ / : * ? " < > | .

    Check out my drunken good-for-nothin’ domain, ya’ll!

    The FQDN is NSFW

    If you examine the workaround, Exchange would have this problem with a user samAccountName that contains an ampersand too - and this is valid:


    That’s why Exchange has to write that KB. Windows NT has been doing this since 1992 (and LAN Manager since the 80s) . Nah nah nah, we were here first! Not that you’d want to use these weird characters anyways.

    Other Random Goo

    How responsible is your facial hair? I’m currently Unsavory.

    Via superfan Mark Morowczynski, courtesy of genius Matt McInerney and Pixelspread. Click it twice! 

    Looks like VMware went and shot itself in the foot.

    Attack of the Show got a tour of Microsoft that’s worth watching. I mainly clicked it to stare at Jessica Chobot, but then there were some crazy things shared – like Skinput (at 2:00) and another video showing new Kinect tech. They both illustrate a bit of campus paradise too, for those interested.

    We began enabling TechNet and MSDN Profile achievements yesterday. You can see these by hovering over a user’s ID on a webpage. Turns out they will be retroactive to 2002, so it will take at least two weeks to grovel the (billions of!!!) data points. Pretty cool, check yourself out:

    It might encourage better pictures too, Artem…


    There are a million “email efficiency” lists on the internoob. They say things like “only check email twice a day” or “use lots of folders to organize your inbox.” Never mind that if you're only checking email twice a day, you're ignoring your primary communication channel for 7 hours out of 8. Or that Outlook and Gmail proved years ago that email folders are inferior to a good search routine.

    Most of these lists are reheated leftovers. But the latest so-called tip I see is the poorly-defined “don’t send ‘thank you’ emails” rule. This is supposed to make you more efficient, but what it really says is that you are an ungrateful clod, who's too important to spend five seconds on someone who took time to help you. Would you walk up to a colleague, ask them a question, listen to their answer, then turn and walk away without saying a word? Of course not, you aren’t a freak. Why then is it ok in email, where someone answered your question with much more effort than talking? They stopped, read your mail, considered your question, wrote a response, made sure it was not egregiously misspelled, then sent it. They're left wondering if you got the mail, if you accidentally deleted it, or if you didn’t understand it and are too embarrassed to admit it. Maybe to the detriment of a customer or your company.

    Naturally, I am not saying “reply to every single email with a thank you”; you don’t need to ACK every SYN. But if you ask someone for info, the least you can do is tell them it resolved your issue and give them a “thnx”. Who knows, you might even accidentally brighten someone’s lousy day. And wouldn’t that be civilized…

    Until next time.

    Ned “Miss Manners with a machinegun” Pyle

  • How to Determine the Minimum Staging Area DFSR Needs for a Replicated Folder

    Warren here again. This is a quick reference guide on how to calculate the minimum staging area needed for DFSR to function properly. Values lower than these may cause replication to go slowly or stop altogether. Keep in mind these are minimums only. When considering staging area size, remember this: the bigger the staging area the better, up to the size of the Replicated Folder. See the section “How to determine if you have a staging area problem” and the blog posts linked at the end of this article for more details on why it is important to have a properly sized staging area.

    Update: Warren is very persuasive! We now have a hotfix to help you with calculating staging sizes.

    Rules of thumb

    Windows Server 2003 R2 – The staging area quota must be as large as the 9 largest files in the Replicated Folder

    Windows Server 2008 and 2008 R2 – The staging area quota must be as large as the 32 largest files in the Replicated Folder

    Initial Replication will make much more use of the staging area than day-to-day replication. Setting the staging area higher than the minimum during initial replication is strongly encouraged if you have the drive space available

    Where do I get PowerShell?

    PowerShell is included on Windows 2008 and higher. You must install PowerShell on Windows Server 2003. You can download PowerShell for Windows 2003 here.

    How do you find these X largest files?

    Use a PowerShell script to find the 32 or 9 largest files and determine how many gigabytes they add up to (thanks to Ned Pyle for the PowerShell commands). I am actually going to present you with three PowerShell scripts. Each is useful on its own; however, number 3 is the most useful.

    1. Run:

    Get-ChildItem c:\temp -recurse | Sort-Object length -descending | select-object -first 32 | ft name,length -wrap –auto

    This command will return the file names and the size of the files in bytes. Useful if you want to know what 32 files are the largest in the Replicated Folder so you can “visit” their owners.

    2. Run:

    Get-ChildItem c:\temp -recurse | Sort-Object length -descending | select-object -first 32 | measure-object -property length –sum

    This command will return the total number of bytes of the 32 largest files in the folder without listing the file names.

    3. Run:

    $big32 = Get-ChildItem c:\temp -recurse | Sort-Object length -descending | select-object -first 32 | measure-object -property length –sum

    $big32.sum /1gb

    This command will get the total number of bytes of 32 largest files in the folder and do the math to convert bytes to gigabytes for you. This command is two separate lines. You can paste both them into the PowerShell command shell at once or run them back to back.

    Manual Walkthrough

    To demonstrate the process and hopefully increase understanding of what we are doing, I am going to manually step through each part.

    Running command 1 will return results similar to the output below. This example only uses 16 files for brevity. Always use 32 for Windows 2008 and later operating systems and 9 for Windows 2003 R2

    Example Data returned by PowerShell


    How to use this data to determine the minimum staging area size:

    • Name = Name of the file.
    • Length = bytes
    • One Gigabyte = 1073741824 Bytes

    First, you need to sum the total number of bytes. Next divide the total by 1073741824. I suggest using Excel or your spreadsheet of choice to do the math.


    From the example above the total number of bytes = 75241684992. To get the minimum staging area quota needed I need to divide 75241684992 by 1073741824.

    75241684992 / 1073741824 = 70.07 GB

    Based on this data I would set my staging area to 71 GB if I round up to the nearest whole number.

    Real World Scenario:

    While a manual walkthrough is interesting it is likely not the best use of your time to do the math yourself. To automate the process, use command 3 from the examples above. The results will look like this


    Using the example command 3 without any extra effort except for rounding to the nearest whole number, I can determine that I need a 6 GB staging area quota for d:\docs.

    Do I Need to Reboot or Restart the Service for the Changes to be Picked Up?

    Changes to the staging area quota do not require a reboot or restart of the service to take effect. You will need to wait on AD replication and DFSR’s AD polling cycle for the changes to be applied.

    How to determine if you have a staging area problem

    You detect staging area problems by monitoring for specific events IDs on your DFSR servers. The list of events is 4202, 4204, 4206, 4208 and 4212. The texts of these events are listed below. It is important to distinguish between 4202 and 4204 and the other events. It is possible to log a high number of 4202 and 4204 events under normal operating conditions. Think of 4202 and 4204 events as being analogous to taking your pulse whereas 4206, 4208 and 4212 are like chest pains. I explain below how to interpret your 4202 and 4204 events below.

    Staging Area Events

    Event ID: 4202
    Severity: Warning

    The DFS Replication service has detected that the staging space in use for the replicated folder at local path (path) is above the high watermark. The service will attempt to delete the oldest staging files. Performance may be affected.

    Event ID: 4204
    Severity: Informational

    The DFS Replication service has successfully deleted old staging files for the replicated folder at local path (path). The staging space is now below the high watermark.

    Event ID: 4206
    Severity: Warning

    The DFS Replication service failed to clean up old staging files for the replicated folder at local path (path). The service might fail to replicate some large files and the replicated folder might get out of sync. The service will automatically retry staging space cleanup in (x) minutes. The service may start cleanup earlier if it detects some staging files have been unlocked.

    Event ID: 4208
    Severity: Warning

    The DFS Replication service detected that the staging space usage is above the staging quota for the replicated folder at local path (path). The service might fail to replicate some large files and the replicated folder might get out of sync. The service will attempt to clean up staging space automatically.

    Event ID: 4212
    Severity: Error

    The DFS Replication service could not replicate the replicated folder at local path (path) because the staging path is invalid or inaccessible.

    What is the difference between 4202 and 4208?

    Events 4202 and 4208 have similar text; i.e. DFSR detected the staging area usage exceeds the high watermark. The difference is that 4208 is logged after staging area cleanup has run and the staging quota is still exceeded. 4202 is a normal and expected event whereas 4208 is abnormal and requires intervention.

    How many 4202, 4204 events are too many?

    There is no single answer to this question. Unlike 4206, 4208 or 4212 events, which are always bad and indicate action is needed, 4202 and 4204 events occur under normal operating conditions. Seeing many 4202 and 4204 events may indicate a problem. Things to consider:

    1. Is the Replicated Folder (RF) logging 4202 performing initial replication? If so, it is normal to log 4202 and 4204 events. You will want to keep these to down to as few as possible during Initial Replication by providing as much staging area as possible
    2. Simply checking the total number of 4202 events is not sufficient. You have to know how many were logged per RF. If you log twenty 4202 events for one RF in a 24 hour period that is high. However if you have 20 Replicated Folders and there is one event per folder, you are doing well.
    3. You should examine several days of data to establish trends.

    I usually counsel customers to allow no more than one 4202 event per Replicated Folder per day under normal operating conditions. “Normal” meaning no Initial Replication is occurring. I base this on the reasoning that:

    1. Time spent cleaning up the staging area is time spent not replicating files. Replication is paused while the staging area is cleared.
    2. DFSR benefits from a full staging area using it for RDC and cross-file RDC or replicating the same files to other members
    3. The more 4202 and 4204 events you log the greater the odds you will run into the condition where DFSR cannot clean up the staging area or will have to prematurely purge files from the staging area.
    4. 4206, 4208 and 4212 events are, in my experience, always preceded and followed by a high number of 4202 and 4204 events.

    While allowing for only one 4202 event per RF per day is conservative it greatly decreases your odds of running into staging area problems and better utilizes your DFSR server’s resources for the intended purpose of replicating files.

    More Information

    Warren “way over my Oud quota” Williams

  • Windows Server 2008 SP1 and Windows Vista SP1 now UNSUPPORTED

    Ned here with the last word on this: Windows Server 2008 SP1 and Windows Vista SP1 are no longer supported. You must install Service Pack 2 to get updates and support from Microsoft going forward with those OSes. But me no buts.

    I made a helpful diagram of how to resolve this - or indeed any - Windows issue.

    yay computer click

    Ned “clipart is overrated” Pyle

  • I’ll take NDES in the DMZ, for 1000 Alex

    Hello. Jim here yet again to talk to you about deploying Windows Server 2008 R2 with the Network Device Enrollment Services (NDES) role in a secure perimeter network. Let's consider the scenario.

    You have an internal PKI hierarchy consisting of an Offline Root Certificate Authority (CA), a policy CA, and an issuing CA. You want to deploy the server holding the NDES role to the secure perimeter network. You do not want to open any ports to facilitate communication between the internal network and the perimeter network. You want to enroll unmanaged non-Microsoft devices like iPads and iPhones using the NDES server in the perimeter network from the Internet.

    Your preliminary idea looks like this:


    Looks good on paper. The problem is that without any connectivity between the perimeter network and the internal AD forest, certificate enrollment via the NDES server will be impossible. There are a few options for you to consider. The first is to insist that the necessary ports opened between the perimeter network and the internal LAN, which is an uphill fight at best. Another option would be to configure the issuing CA to listen for RPC traffic on a single port, and then open that single port in the firewall. You can then use firewall rules allow traffic only between the NDES server and the CA. This requires opening a port to the internal network. Another option is to bring up a new AD forest within the perimeter network. This will require at least one domain controller, an issuing Certificate Authority, a Windows 2008 R2 server to handle the NDES role and finally a server to act as an Internet-facing HTTP CRL Distribution Point (CDP) location for all of your CAs when you first set up the PKI. Your perimeter hosts will use that to check revocation status of certificates issued by the internal and perimeter network PKI. HTTP CRL distribution point locations are ideal for providing accessible CRL locations for clients that are not running the Windows operating system.

    "Why not setup a forest trust?" you may ask. Well, you would then need to punch even more holes in the firewall to establish and maintain the trust, which is sure to make your security team gnash their teeth even more than when you asked them to open up ports for the NDES server.

    Good news! A forest trust is unnecessary. All we need to do is follow some specific steps concerning setting up the additional issuing CA in the perimeter network after we have created the new forest. This can be deployed quite effectively using Hyper-V or some other more costly virtualization solutions that will remain nameless.

    Our new setup will look like this:



    Configuration decisions for the perimeter network must be made including but not limited to IP settings, subnets, DNS, backup planning and management. For the sake of this blog post, I will leave those details to you and move on.

    Here is a picture of how the PKI hierarchy will look on both sides of the firewall:


    I am keeping it simple with one site, one subnet and the DC in the perimeter network will serve DNS only for the resources in the perimeter network. The first thing we do is bring up a new domain controller in the perimeter network and it will be the first DC for a brand new forest. This DC will also be a DNS server. Once this is accomplished, we build and join a Windows Server 2008 R2 server to the new perimeter forest. This server will be our issuing CA. We install the Certificate Services role on this member server and make it an Enterprise CA so we can leverage certificate templates.

    During the configuration of the issuing CA, you have to save the certificate request to file and manually transport it to the parent CA because we do not have connectivity to the internal Root CA from the perimeter network.


    Now we complete our install of the issuing CA in the perimeter network by getting the certificate back from the Root CA in the internal network and making our issuing CA operational.

    You must also export the Offline Root CA certificate and publish it in the perimeter forest using the following certutil.exe command:

    certutil.exe -dspublish -f filename.CER RootCA

    Since this is a Windows Server 2008 Active Directory forest we must enable Certificate Services Client – Auto-Enrollment. This can be accomplished by creating a new Group Policy object or configured within the default domain policy as explained here –

    This will propagate the Root certificate down to the trusted root store on all domain joined computers via Group Policy. This will establish your internal Root CA as a trust anchor in the perimeter forest.

    Next, bring a server online in the DMZ to be the repository for all the CRLs for every CA that is involved. This server has the IIS role installed and will be configured as a CDP on all the certificates issued by the perimeter forests issuing CA.

    To specify the Web Server as the CDP you will have to perform the following tasks on the internal ROOT CA and on the issuing CA server in the perimeter forest in that order. This will enable both to publish their CRLs to this location as well as including that CDP path on the certificate the ROOT CA issues to the issuing CA in the DMZ forest:

    Load Certification Authority management console.

    Right-click your CA, select Properties, and then click the Extensions tab.

    With CRL Distribution Point (CDP) selected


    Click Add.


    In the Add Location dialog box, type the following and then click OK:


    For example, if your Web server is called and the virtual directory you created in IIS was called CRL, you would type:<CaName><CRLNameSuffix><DeltaCRLAllowed>.crl


    The following options will be selected for this new entry and are highlighted in the above screen capture:

    1. Check Include in CRLs. Clients use this to find Delta CRL locations.
    2. Check Include in the CDP extension of issued certificates
    3. Click OK.
    4. If you are prompted to restart Active Directory Certificate Services, click Yes.

    It will be necessary to copy the CRLs for the internal PKI hierarchy to the HTTP location in the perimeter forest. This can be accomplished via script or scheduled task leveraging robocopy and a service account in the perimeter forest.

    You can then proceed with the installation of the NDES role on the Windows Server 2008 R2 member server in the perimeter forest. Steps for configuration are illustrated in detail here.

    The CA in the DMZ should probably have a Hardware Security Module HSM as this network isn't safe.

    After installing the NDES role, you may install the certificate authority web enrollment pages, as these are not installed by default when you install the NDES role nor are they needed. The ISAPI .DLL will intercept any requests for mscep_admin, or mscep so you don’t need an actual certsrv/mscep_admin virtual directory. However, if it makes you happy to open up IIS management and see the virtual directories there, have at it!


    Something else to remember is if you install both Web Enrollment and NDES on the same server, depending on the order, anonymous access to the mscep virtual directory may be turned off. Anonymous access must be enabled on the mscep virtual directory so that NDES clients, without accounts, can submit requests.

    Now you are cruising through the NDES install and get to the end. Much to your chagrin, you are confronted with the following error:

    Element not found. 0x80070490 (WIN32: 1168)


    You verify the permissions on the following templates:

    • CEP Encryption
    • Exchange Enrollment Agent (Offline Request)
    • IPSec (Offline Request)

    Authenticated Users – Read and Enroll
    NDES service account – Read and Enroll
    Domain Admins – Read, Write and Enroll
    Enterprise Admins – Read, Write and Enroll

    You test accessibility to the templates by logging in to the NDES server with an Enterprise Administrator account and then running the Computer Certificates MMC and attempting to enroll against the aforementioned templates. When the available templates window opens in the Computer Certificates MMC, nothing is listed.

    You then take a network trace of the Computer Certificates MMC attempt at manual enrollment and observe the following:

    618 LDAP searchRequest(60) "CN=CA Policy CA,CN=AIA,CN=Public Key Services,CN=Services,CN=Configuration,DC=Contoso,DC=COM" baseObject

    619 LDAP searchResDone(60) operationsError (00000000: LdapErr: DSID-0C090627, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, vece) [0 results]

    Frame 618 indicates that we are attempting to complete the chaining to a Policy CA certificate as part of the verification of the issuing CA's certificate. Remember the CA hierarchy from our internal forest in the diagram above.

    We cannot access that LDAP path in the other forest from the perimeter network. The query fails as reported in frame 619 and so does the ability to enroll against any templates published by the subordinate CA in the perimeter forest.

    Now we must export the Policy CA certificate from the internal forest and copy it to a domain joined computer in the perimeter forest and run the following command:

    certutil.exe -f -dspublish policyca.cer SubCA

    Autoenrollment, which is configured and in place will download the Policy CA certificate to the intermediate store on any domain joined computer.


    Computer Autoenrollment runs when the computer is started and every 8 hours thereafter. You can force Computer Autoenrollment to run by running:

    certutil.exe -pulse

    Now the install of the NDES server will complete without error.

    I deliberately skipped the Policy CA certificate -dspublish step to show you the kind of cryptic error you will receive during the install in this particular instance.

    After you have accomplished all of the aforementioned please install the following hotfix as well:

    2483564 Renewal request for an SCEP certificate fails in Windows Server 2008 R2 if the certificate is managed by using NDES -

    Now you have a relatively secure environment where you can leverage NDES for device certificate enrollment and renewal without opening up your internal LAN to the DMZ. There is understandably a lot of overhead in this design as we are working around the limitations placed on access to the internal network through the firewall. For information on non-Windows device enrollment, please refer back to this spectacular blog post written by my colleague Robert Greene.

    Until next time.

    Jim "Daily Double" Tierney

  • Last Week Before Vista and Win2008 SP1 Support Ends

    Last chance folks. Windows Vista and Windows Server 2008 Service Pack 1 support ends on July 12. As in, one week from now. This means computers running SP1 won’t get security updates after the next Patch Tuesday.

    The bomb represents malware. Or your next review…

    For those not running WSUS or SCCM, grab SP2 here:

    Not sure which computers are running lean? Check out these inventory techniques you can use to find SP1 computers in the domain using AD PowerShell; all you need is one Win7 computer or Win2008 R2 DC. Only have older DCs? That’s ok, use the AD Management Gateway. Think PowerShell is t3h sux 4 n00b$? That’s ok, Amish IT, use CSVDE.EXE to get a list back from your DCs that you can examine in Excel. Foir example, all the Vista non-SP2 computers:

    csvde -f c:\sp.csv -p subtree –d “dc=contoso,dc=com” -r "(&(operatingsystem=windows vista*)(!operatingsystemservicepack=Service Pack 2))" -l operatingsystem,operatingsystemservicepack -u

    No complaints that this finds stale computers and doesn’t tell you IP addresses – AD PowerShell does all that and bakes delicious pies.

    Windows Server 2008 shipped with SP1 built-in as it were, so there is no way for it to have no service pack at all. Whatever you do, it’s closing time for SP1. You don’t have to go home but you can’t stay here.

    Ned “I recently upgraded to Windows 2000 – it’s pretty slick!” Pyle