Blog - Title

February, 2011

  • RTM RSAT and SP1 Win7 (shot, over)

    Hi all, Ned here.

    Update 4/7/11: The Remote Server Administration Toolkit update to support Windows 7 Service Pack 1 has released. Come and get it:

    http://www.microsoft.com/downloads/en/details.aspx?FamilyID=7d2f6ad7-656b-4313-a005-4e344e43997d

     Yesterday we announced Windows 7 SP1 availability info. For those that use RSAT to admin from their clients, I am launching a preemptive strike before the comments flow:

    If you have SP1 installed and attempt to install the RTM version of the Remote Server Administration Tools, downloaded from here:

    http://www.microsoft.com/downloads/en/details.aspx?FamilyID=7d2f6ad7-656b-4313-a005-4e344e43997d&displaylang=en

    You will receive error:

    The update is not applicable to your computer

    This is expected behavior and noted in the SP1 deployment guide and on the download page. The RTM version of Win7 RSAT has internal checks to prevent install on later operating systems and service packs. All is not lost though.

    Workaround

    Install RTM RSAT before installing SP1. When already installed, SP1 will update the RSAT component files to SP1 versions. Obviously, this will not work on computers being built from SP1-slipstreamed install media. For example, I had RSAT installed on my own computer when I installed SP1. My Hyper-v Manager snap-in updated automagically:

    Long view

    Install the Win7 SP1 version of the RSAT package once we release it. When it is ready I will update this article as well as post a new article for the RSS crowd.

    The SP1 version of RSAT is planned for April. Telling me how important it is will not move things faster – we know it’s important. Don’t bother trying to override with DISM /Add-Package or PKGMGMR /IP, that will not work either; the top level package in the cab cannot be bypassed.

    Until next time.

    Ned “splash, out” Pyle

  • So glad that I moved…

    Hot enough for ya, Mark?

    Update - way more glad now that I saw this:

    image

    Ned “drove in with his windows down today” Pyle

  • KCC Offline Bridgehead Behaviors

    This is a guest post from our friend Keith Brewer, a Premier Field Engineer that recently spent some time with us here in support as part of a “foreign exchange student” program. As you can see, we pay him by the screenshot… :-P

    Hi all, Keith here. Recently I answered a forum question on KCC “topology review” frequency. You can read that here.

    There were some interesting follow up questions that came from that conversation:

    1. How exactly does the KCC behave when a bridgehead goes offline?
    2. What is the impact if the bridgehead is the ISTG or if the ISTG goes offline at the same time as one of the domain controllers serving as the bridgehead?
    3. Do manually created connection objects change the behavior?

    So I thought the easiest way to explain is to walk through it…

    The setup is below (don’t worry about Branch1 and the RODC). For the purposes of this example, we will concentrate on the Hub Site HQ, the Branch Site Branch2, and the Backup Hub Site BackupHub.

    image

    • FAB-DC3 & FAB-DC4 are Windows Server 2008 R2
    • FAB-DC1 & FAB-DC2 are Windows Server 2008 SP2
    • Forest & Domain Functional Level is Windows Server 2003

    Under normal operation the ISTG builds an automatically-generated connection object to a DC (or DCs) in the HQ Site. Similar to what we see below for the BackupHub site and HQ because of the connectivity described on the HQ-BUHUB Site Link.

    image

    I have created a manual connection object between Branch2 DC (FAB-DC3) and the HQ site with FAB-DC2 to speak to question 3 above.

    image

    Additionally here are the HQ connections that have both FAB-DC1 & FAB-DC2 acting as bridgehead domain controllers.

    image

    image

    Here is the current (truncated) replication information.

    image

    image

    @ 14:44 FAB-DC2 goes offline

    @ 14:49 FAB-DC3 shows 1st failure from FAB-DC2

    image

    @ 14:54 DC4 follows suit and shows 1st failure from FAB-DC2

    image

    Now we wait for the 2 hour default window. While we wait let’s look at the ISTG election information:

    Conveniently the ISTG for HQ Site is FAB-DC2 who as we all know has tragically gone offline @ 14:44

    image

    So we know that FAB-DC1 will review its information (contained in the UpToDateness Vector Table) on the validity of DC2 as the ISTG. Seen here:

    image

    So then at some point between 16:43 & 16:58 we should see DC1 take over the HQ sites ISTG Role.

    JACKPOT!

    image

    Looking at the Replication Metadata we can get a clear picture of when the election took place and who wrote the change.

    image

    A new ISTG is elected @ 16:44 2 hours from the last successful replication of the old ISTG.

    So now we see what the KCC did once we met both criteria

    • # of Failures
    • Duration of time since last success

    We can see on FAB-DC3 a new automatically created connection was created.

    image

    Note the creation time of 4:39 or 16:39 (Which is 2:05 from the last successful Replication which occurred at 14:34 or 2:34.

    Now taking a look at FAB-DC4 (similar behavior):

    image

    FAB-DC4 created a connection @ 4:45 or 16:45 (Which was 2:02 from the last successful replication which occurred at 14:43 or 2:43

    Last but not least we see how the Hub Site Behavior and resulting connections are handled once the new ISTG is elected.

    image

    And now connection from Branch2 (FAB-DC3) has been created by the KCC from FAB-DC3 to FAB-DC1 at 4:44pm seconds after the ISTG election took place @ 4:44:37 in response to the # of failures and amount of time since FAB-DC2 last replicated from Branch3.

    Note About the use of manual connection objects:

    While the question posed involves manual connection objects and the explanation of the behavior includes manual connection objects that is by no means an endorsement of their use.

    Careful planning should be invested into designing the Active Directory site & site link configuration.

    In most cases it is preferred to allow the KCC to utilize Active Directory configuration information to build and manage all replication connections. Adding manual connection’s adds administrative overhead and limits the KCC’s ability to build and manage the replication topology.

    Now how the KCC cleans up the connections on DC4 for DC2 on DC3 for DC2 and in the Hub on DC2 from DC3 is a story for another thread….

    -Keith “What’s your vector, Victor?” Brewer

  • I moved my PDCE role and accounts started locking out!

    Hi, David here. We’ve seen a few cases on this now, so I wanted to put the word out and explain why it happens and how you can (very easily) prevent it from happening to you.

    The scenario:

    Imagine an ordinary domain admin. Let’s call him Fred. Fred has finally gotten the ok from his management to start deploying shiny new Windows Server 2008 R2 domain controllers (and the new hardware he wanted to do this). Fred brings up a DC and spends several weeks making sure that everything works. Then, confident in the stability of his new DC, he transfers the PDC Emulator role to it. Nothing explodes and everything appears to be good. Mission accomplished, Fred goes home and enjoys the rest of his weekend.

    On Monday morning Fred gets to work and finds out that the help desk is swamped with calls from users whose accounts are locked out. Unlocking the accounts only seems to fix them temporarily, and then they get locked out again. Fred’s manager tells him to undo the change he made over the weekend, which he does. Desperate to figure out why his new DC betrayed him so horribly, Fred opens up a case with Microsoft for support and gets someone on our team.

    Troubleshooting an account lockout:

    Obviously this is a bad situation for Fred, but unfortunately it’s kind of hard to troubleshoot an account lockout without logs from while the problem was happening.

    As an aside here, if you haven’t examined the Security Compliance Manager tool and its included docs, you should probably take a look. It lays out our recommendations around account lockout policies.

    There are multiple tools for troubleshooting account lockouts, but sometimes it pays to go old-school: What we want for this are the Netlogon debug logs, which every domain administrator should be familiar with. Netlogon debug logging can show you all kinds of very useful information for troubleshooting authentication issues, particularly with NTLM authentication. In this situation, it shows us something very interesting if we take a look at the log from the domain controllers of the domain while the accounts are being locked out:

    [LOGON] SamLogon: Transitive Network logon of Domain\User from Computer successfully handled on DC (UseHub is FALSE).

    I should mention here that the netlogon debug logging is NOT turned on by default, which is really just a holdover from the days when your server processor speeds were measured in mhz. It can be a highly useful troubleshooting tool and everyone should know how to turn it on – documented here.

    Here in DS, we spend a lot of time looking at Netlogon debug logs, and when we first saw the above line in the log, we were stumped as to where it was coming from. It’s not something that we normally see at all, and none of us could remember ever seeing it in a case before.

    It turns out that this output only happens in the log under a very specific set of circumstances when the authenticating domain controller decides that it needs to bypass validating the password with the PDC if a bad password is received. When it makes this decision, it sets a parameter called UseHub to FALSE instead of the default of TRUE. Thankfully, it writes this in the netlogon log for us to see; otherwise we’d never have had any clue what it was doing.

    Unfortunately the log didn’t tell us why it was happening - only that it did happen. But, after some snooping in source code and a few dozen emails, we discovered that this decision occurs when the PDC of the domain will not allow us to pass the client’s credentials because the client is using a Lan Manager Authentication Method that is not supported on the PDC.

    Or, in normal language, what it means is that your LMCompatibility settings don’t match.

    Why would this happen just by moving the PDC Emulator role?

    So, like every new operating system, we ship Windows 2008 R2 with enhanced security when compared to its predecessors. Sometimes this security is accomplished by changing the way that OS works to make it harder to attack, or turning off unnecessary services until they are needed. At other times, we simply change default settings on features that were present in previous OS versions, because the majority of the world can now support those higher settings. LMCompatibility is one of those settings. The default for Windows 2008 R2 is a setting of 3: Send NTLM v2 response only/ Allow LM and NTLM.

    In Fred’s case, it turned out that his XP clients all had a setting of 1: Send LM and NTLM responses, while his new PDC emulator had a setting of 5: Send NTLM v2 response only/Refuse LM and NTLM. It’s worth noting that these setting aren’t the default – someone had to choose to put them there. The clients couldn’t use NTLMv2 session security, which is why we couldn’t pass the user’s credentials to the 2008 R2 PDC Emulator for evaluation. The 2003 DCs on the other hand, had a setting of 2: Send LM and NTLM, Use NTLMv2 if negotiated. So when the PDC was running Windows 2003, we didn’t have this problem. So the new Win2008 R2 OS was not specifically an issue – the same issue would have happened to any version of Windows running the PDCE.

    For normal Kerberos logons, we don’t care about LM Compatibility, but there are plenty of applications out there that will default to NTLM – and most applications will retry logons multiple times on your behalf without ever telling you that they’re doing it. In Fred’s environment, all it took was his Outlook clients, connecting to his Exchange CAS servers over http and using NTLM to try and authenticate that connection. The users had changed their passwords that morning and the local DCs didn’t have the new password – so, the password that Outlook used looked “bad” to the local DC. Because of the LM Compatibility mismatch, we couldn’t talk to the PDC, and thus we ended up locking the account out.

    Solving the problem – the right way

    So, Fred’s first inclination upon hearing from support about this might have been to reduce the security setting on the PDC emulator to make everything magically start working. And while this would have been effective, it would not have been the best solution from a security perspective. There are plenty of good reasons why you should want to use the strongest encryption and security algorithms on network communications, especially ones where your users passwords are being handed back and forth between computers for validation.

    The right solution here is that Fred should be centrally managing his settings in a way that fits his network and enforces the best possible level of security. Fortunately there’s a group policy setting that enables him to do just that:

    clip_image001

    This setting is located in Computer Configuration ->Windows Settings -> Security Settings -> Local Policies -> Security Options. Notice the very helpful text on the Explain tab that outlines the default settings.

    So, if Fred was confident that all of the computers on his network supported NTLMv2, he could go ahead and use the policy to enforce the highest level of security on his entire network (Send NTLMv2 response only\refuse LM and NTLM). Or, if he suspected that there might be a few applications (or more likely, ancient operating systems) out there that haven’t quite been retired yet that might have problems with NTLMv2, he could use the fourth option instead and just refuse LM connections. As a note here, every supported Windows OS version supports NTLM v2 – so the situations where you can’t use it should be very rare and only happen with specific, third-party applications or OS platforms.

    David “Fred Herring” Beach

  • Friday Mail Sack: No Redesign Edition

    Hello folks, Ned here again. Today we talk PDCs, DFSN, DFSR, AGPM, authentication, PowerShell, Kerberos, event logs, and other random goo. Let’s get to it.

    Question

    Is the PDC Emulator required for user authentication? How long can a domain operate without a server that is running the PDC Emulator role?

    Answer

    It’s not required for direct user authentication unless you are using (unsupported) NT and older operating systems or some Samba flavors. I’ve had customers who didn’t notice their PDCE was offline for weeks or months. Plenty of non-fully routed networks exist where many users have no direct access to that server at all.

    However!

    It is used for a great many other things:

    • With the PDCE offline, users who have recently changed their passwords are more likely to get logon or access errors. They will also be more likely to stay locked out if using Account Lockout policies.
    • Time can more easily get out of sync, leading to Kerberos authentication errors down the road.
    • The PDCE being offline will also prevent the creation of certain well-known security groups and users when you are upgrading forests and domains.
    • The AdminSDHolder process will not occur when the PDCE is offline.
    • You will not be able to administer DFS Namespaces.
    • It is where group policies are edited (by default).
    • Finally - and not documented by us - I have seen various non-MS applications over the years that were written for NT and which would stop working if there is no PDCE. There’s no way to know which they might be – a great many were home-made application written by the customer themselves – so you will have to determine this through testing.

    But don’t just trust me; I am a major plagiarizer!

    How Operations Masters Work (see section “Primary Domain Controller (PDC) Emulator”)
    http://technet.microsoft.com/en-us/library/cc780487(WS.10).aspx

    Question

    The DFSR help file recommends a full mesh topology only when there are 10 or fewer members. Could you kindly let me know reasons why? We feel that a full mesh will mean more redundancy.

    Answer

    It’s just trying to prevent a file server administrator from creating an unnecessarily complex or redundant topology, especially since the vast majority of file server deployments do not follow this physical network topology. The help file also makes certain presumptions about the experience level of the reader.

    It’s perfectly ok – from a technical perspective - to make as many connections as you like if using Windows Server 2008 or later. This is not the case with Win2003 R2 (see this old post that applies only to that OS). The main downsides to a lot of connections are:

    • It may lead to replication along slower, non-optimal networks that are already served by other DFSR connections; DFSR does not sense bandwidth or use any site/connection costing. This may itself lead to the networks becoming somewhat slower overall.
    • It will generate slightly more memory and CPU usage on each individual member server (keeping track of all this extra topology is not free).
    • It’s more work to administer. And it’s more complex. And more work + more complex usually = less fun.

    Question

    I'm trying setup delegation for Kerberos but I can't configure it for user or computer accounts using AD Users and Computers (DSA.MSC). I’m logged as a domain administrator. Every time when I'm trying activate delegation I get error:

    The following Active Directory error occurred: Access is denied.

    Answer

    It’s possible that someone has removed the user right for your account to delegate. Check your applied domain security policy (using RSOP or GPRESULT or whatever) to see if this has been monkeyed up:

    Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment
    "Enable computer and user accounts to be trusted for delegation"

    The Default Domain Controllers policy will have the built-in Administrators group set for that user right assignment once you create a domain. The privilege serves no purpose being set on servers other than DCs, they don’t care. Changing the defaults for this assignment isn’t necessary or recommended, for reasons that should now be self-evident.

    Question

    I want to clear all of my event logs at once on Windows Vista/2008 or later computers. Back in XP/2003 this was pretty easy as there were only 6 logs, but now there are a zillion.

    Answer

    Your auditors must love you :). Paste this into a batch file and run in an elevated CMD prompt as an administrator:

    Wevtutil el > %temp%\eventlistmsft.txt
    For /f "delims=;" %%i in (%temp%\eventlistmsft.txt) do wevtutil cl "%%i"

    If you run these two commands manually, remember to remove the double percent signs and make them singles; those are being escaped for running in a batch file. I hope you have a systemstate backup, this is forever!

    Question

    Can AGPM be installed on any DC? Should it be on all DCs? The PDCE?

    Answer

    [Answer from AGPM guru Sean Wright]

    You can install it on any server as long as it’s part of the domain  - so a DC, PDCE, or a regular member server. Just needs to be on one computer.

    Question

    Is it possible to use Authentication Mechanism Assurance that is available in Windows Server 2008 R2 with a non-Microsoft PKI implementation? Is it possible to use Authentication Mechanism Assurance with any of Service Administration groups Domain Admins or Enterprise Admins? If that is possible what would be the consequences for built-in administrator account, would this account be exempt from Authentication Mechanism Assurance? So that administrators would have a route to fix issues that occurred in the environment, i.e. a get out of jail.

    Answer

    [Answer from security guru Rob Greene]

    First, some background:

    1. This only works with Smart Card logon. 
    2. This works because the Issuance Policy OID is “added to” msDS-OIDToGroupLink on the OID object in the configuration partition.  There is a msDS-OIDToGroupLinkBl (back link) attribute on the group and on the OID object.
    3. The attribute msDS-OIDToGroupLink attribute on the OID object (in the configuration partition)stores the DN of the group that is going to use it.
    4. Not sure why, but the script expects the groups that are used in this configuration to be Universal groups.  So the question about Administrative groups, none of these are Universal groups except for “Enterprise Admins”.

    So here are the answers:

    Is it possible to use Authentication Mechanism Assurance that is available in Windows Server 2008 R2 with a non-Microsoft PKI implementation?

    Yes, however, you will need to create the Issuance Policies that you plan to use by adding them through the Certificate Template properties as described in the TechNet article.

    Is it possible to use Authentication Mechanism Assurance with any of Service Administration groups Domain Admins or Enterprise Admins?

    This implementation requires that the group be a universal group in order for it to be used.  So the only group of those listed above that is universal is “Enterprise Admins”.  In theory this would work, however in practice it might not be such a great idea.

    If that is possible what would be the consequences for built-in administrator account, would this account be exempt from Authentication Mechanism Assurance?

    In most cases the built-in Administrator account is special cased to allow access to certain things even if their access has somehow been limited.  However, this isn’t the best way to design your security of administrative accounts if you are concerned about not being able to get back into the domain.  You would have similar issues if you made these administrative accounts require Smart Cards for logon, then for some reason the CA hierarchy did not publish a new CRL and the CA required a domain based admin to be able to logon interactively then you would be effectively locked out of your domain also.

    Question

    I find references on TechNet to a “rename-computer” PowerShell cmdlet added in Windows 7. But it doesn’t seem to exist.

    Answer

    Oops. Yeah, it was cut very late but still lives on in some documentation. If you need to rename a computer using PowerShell, the approach I use is:

    (get-wmiobject Win32_ComputerSystem).rename("myputer")

    That keeps it all on one line without need to specify an instance first or mess around with variables. You need to be in an elevated CMD prompt logged in as an administrator, naturally.

    Then you can run restart-computer and you are good to go.

    image

    There are a zillion other ways to rename on the PowerShell command-line, shelling netdom.exe, wmic.exe, using various WMI syntax, new functions, etc.

    Question

    Does disabling a DFS Namespace link target still give the referral back to clients, maybe in with an “off” flag or something? We’re concerned that you might still accidentally access a disabled link target somehow.

    Answer

    [Oddly, this was asked by multiple people this week.]

    Disable actually removes the target from referral responses and nothing but an administrator’s decision can enable it. To confirm this, connect through that DFS namespace and then run this DFSUTIL command-line (you may have to install the Win2003 Support Tools or RSAT or whatever, depending on where you run this):

    DFSUTIL /PKTINFO

    It will not list out your disabled link targets at all. For example, here I have two link targets – one enabled, one disabled. As far as DFS responds to referral requests, the other link target does not exist at all when disabled.

    clip_image002

    When I enable that link and flush the PKT cache, now I get both targets:

    clip_image002[4]

    Question

    When DFSR staging fills to the high watermark, what happens to inbound and outbound replication threads? Do we stop replicating until staging is cleared?

    Answer

    Excellent question, Oz dweller.

    • When you hit the staging quota 90% high watermark, further staging will stop.
    • DFSR will try to delete the oldest files to get down to 60% under the quota.
    • Any files that are on the wire right now being transferred will continue to replicate. Could be one file, could be more.
    • If those files on the wire are ones that the staging cleanup is trying to delete, staging cleanup will not complete (and you get warning 4206).
    • No other files will replicate (even if they were not going to be cleaned out due to “newness”).
    • Once those outstanding active file transfers on the fire complete, staging will be cleaned out successfully.
    • Files will begin staging and replicating again (at least until the next time this happens).

    So the importance of staging space for very large files remains to ensure that quota is at least as large as the N largest files that could be simultaneously replicated inbound/outbound, or you will choke yourself out. From the DFSR performance tuning post:

    • Windows Server 2003 R2: 9 largest files
    • Windows Server 2008: 32 largest files (default registry)
    • Windows Server 2008 R2: 32 largest files (default registry)
    • Windows Server 2008 R2 Read-Only: 16 largest files

    If you want to find the 32 largest files in a replicated folder, here’s a sample PowerShell command:

    Get-ChildItem <replicatedfolderpath> -recurse | Sort-Object length -descending | select-object -first 32 | ft name,length -wrap –auto

    Question

    If I create a domain-based namespace (\\contoso.com\root) and only have member servers for namespace servers, the share can’t browsed to in Windows Explorer. It is there, I just can’t browse it.

    But if I add a DC as a namespace server it immediately appears. If I remove the DC from namespace it disappears from view again, but it is still there. Would this be expected behavior? Is this a “supported” way create a hidden namespace?

    Answer

    You are seeing some coincidental behavior based on the dual meaning of contoso.com in this scenario:

    • Contoso.com will resolve to a domain controller when using DNS
    • When a DC hosts a namespace share and you are browsing that DC, you are simply seeing all of its shares. One of those shares happens to be a DFS root namespace.
    • When you are browsing a domain-based namespace not hosted on a DC, you are not going to see that share as it doesn’t exist on the DCs.
    • You can see what’s happening here under the covers with a network capture.
    • Users can still access the root and link shares if they type them in, had them set via logon script, mapped drive, GP Preference Item, etc. This is only a browsing issues.

    It’s not an “unsupported” way to hide shares, but it’s not necessarily effective in the long-term. The way to hide and prevent access to the links and files/folders is through permissions and ABE. This solution is like a share with $ being considered hidden: only as long as people don’t talk about it. :) Not to mention this method is easy for other admins to accidentally “break” it through ignorance or reading blog posts that tell them all the advantages of DFS running on a DC.

    PS: Using a $ does work – at least on a Win2008 R2 DFS root server in a 2008 domain namespace:

    clip_image002[7]

    clip_image002[9]

    clip_image002[11]

    But only until your users talk about it in the break room…

    Other Random Goo

    • The Cubs 2011 schedule is up and you can download the calendar file here. You know you wanna.
    • And in a related story, Kerry Wood has come back with a one year deal! Did you watch him strike out 20 as a rookie in 1998? It was insane. The greatest 1-hitter of all time.
    • IO9.com posted their spring sci-fi book wish list. Which means that I now have eight new books in my Amazon wish list. >_<
    • As a side note, does anyone like the new format of the Gawker Media blogs? I cannot get used to them and had to switch back to the classic view. The intarwebs seem to be on my side in this. I find myself visiting less often too, which is a real shame – hopefully for them this isn’t another scenario like Digg.com, redesigning itself into oblivion.
    • Netflix finally gets some serious competition – Amazon Prime now includes free TV and Movie streaming. Free as in $79 a year. Still, very competitive pricing and you know they will rock the selection.
    • I get really mad watching the news as it seems to be staffed primarily by plastic heads reading copy written by people that should be arrested for inciting to riot. So this Cracked article on 5 BS modern myths is helpful to reduce your blood pressure. As always, it is not safe for work and very sweary.

    • But while you’re there anyway (come on, I know you), check out the kick buttitude of Abraham Lincoln.
    • Finally: why are the Finnish so awesomely insane at everything?
    And by everything, I mean only this and rally sport.

     

    Have a nice weekend folks.

    - Ned “simple and readable” Pyle

  • Monitoring and Maintaining DFS Namespaces

    Hello all, David here again. If you are reading this post, you likely have Distributed File System Namespaces (DFSN) deployed or are at least considering it. In large environments, DFS Namespaces may stretch across many sites and target tens or hundreds of file servers. Depending on the size and quantity of namespaces, you may be wondering about the methods available to monitor the health of namespaces and ensure their proper function. I have written the information below to provide such methods.

    Utilize the DFSDiag.exe utility

    First, the administrator of any environment with namespaces should routinely run the DFS Diagnostics (DFSDiag.exe) tool. DFSDiag is available in Windows Vista, Server 2008, 7, and 2008 R2. For any 2008 or 2008 R2 systems not hosting the DFS Namespaces service, you will need to install the Distributed File System Tools found in the Remote Server Administration Tools (RSAT) by using Server Manager. RSAT is a separate download for Vista and Windows 7. In addition, you may leverage the tool in an environment consisting of Windows Server 2003 domain controllers and namespace servers, but you will need at least one of the later OS's to run DFSDiag. If possible, use the Windows 7 or 2008 R2 version of DFSDiag--it contains additional help text describing each option. Lastly, it supports both domain-based and standalone namespaces.

    While there have been a few other blog posts about DFSdiag (look here and here), I will mention the key issues it detects within an environment:

    • Offline file servers, domain controllers, and DFSN servers (Helpful in detecting retired servers that are still referenced within the namespace!)
    • Inaccessible file servers because they have inconsistent NTFS and share permissions when compared to other targets of the DFSN folder
    • Invalid site associations of the system running DFSDiag locally or of any targets defined in the namespace
    • Inconsistent registry settings for the DFSN service compared between namespace servers
    • Inconsistent Active Directory metadata between the domain's domain controllers (may indicate replication latencies or failures)
    • Overlapping folders, folder targets, and duplicated folders
    • Inconsistencies with Access Based Enumeration (ABE) of the namespace and of the namespace share

    Here is a screenshot of DFSDiag output while file server "2008fs1" is offline and the NTFS permissions of the two targets of \\CONTOSO\Namespace1\folder1 are not consistent:

    image

    As you can see, checking all these dependencies manually would take an enormous amount of time. So let DFSDiag do all the work and allow you to fix problems before your users have an opportunity to call the helpdesk!

    Be mindful when configuring subnets and their site associations within the Active Directory. Clients and servers which cannot be mapped to a site will prevent DFS from referring clients to their local targets. DFSDiag will report a failure to map a server's IP address to a site, but it will not alert you if there are random clients in your environment not belonging to an Active Directory site. For this reason, periodically check if any Netlogon ‘5807’ events have been reported on any domain controllers. If any are found, follow the instructions within the event to review the Netlogon.log debug log file located within "%systemroot%\debug" and search for all occurrences of 'NO_CLIENT_SITE'. These indicate the name and IP address of clients on your network which cannot be mapped to an Active Directory site. Then, create the appropriate subnets.

    Event ID: 5807
    Source: NETLOGON
    Description:

    During the past number hours there have been number connections to this Domain Controller from client machines whose IP addresses don't map to any of the existing sites in the enterprise. Those clients, therefore, have undefined sites and may connect to any Domain Controller including those that are in far distant locations from the clients. A client's site is determined by the mapping of its subnet to one of the existing sites. To move the above clients to one of the sites, please consider creating subnet object(s) covering the above IP addresses with mapping to one of the existing sites. The names and IP addresses of the clients in question have been logged on this computer in the following log file 'SystemRoot\debug\netlogon.log' and, potentially, in the log file 'SystemRoot\debug\netlogon.bak' created if the former log becomes full. The log(s) may contain additional unrelated debugging information. To filter out the needed information, please search for lines which contain text 'NO_CLIENT_SITE:'. The first word after this string is the client name and the second word is the client IP address. The maximum size of the log(s) is controlled by the following registry DWORD value'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\LogFileMaxSize'; the default is 20000000 bytes. The current maximum size is 20000000 bytes. To set a different maximum size, create the above registry value and set the desired maximum size in bytes.

    Leverage the File Services Management Pack found in SCOM

    If you have System Center Operations Manager (SCOM) deployed, you may utilize the File Services Management Pack to retrieve health information from all File Services roles, including DFS Namespaces. Download is available here.

    Ready a “toolkit” of common tools and utilities

    Troubleshooting DFS Namespace issues can be difficult. It usually makes sense to begin investigations on a client experiencing a specific failure to access the namespace. Consider building a “toolkit” to make it easier to run DFSDiag, DFSUtil, and Network Monitor 3.4 on the problematic client. Otherwise, you will be forced to download and install the Remote Server Administration Tools (RSAT) to the system you wish to diagnose, and then configure the DFSN RSAT component. A much easier method is to copy DFSUtil.exe and DFSDiag.exe (both found in %systemroot%\system32) to a share, to a thumbdrive, or directly to the client. Ensure that you also copy the dfsdiag.exe.mui language file from the '%systemroot%\system32\en-us' folder and place it into an 'en-us' subfolder where you intend to run dfsdiag. Note, you will also need to maintain separate versions of dfsdiag if you maintain both Vista/2008 and Window 7/2008 R2 systems. If you were to rename dfsdiag.exe to 'dfsdiagwin7.exe', ensure you similarly rename the MUI file to 'dfsdiagwin7.exe.mui'.

    Create a disaster recovery plan

    Do you have a disaster recovery plan in the event all or portions of your namespace are lost due to hardware failures or accidental deletions? If not, strongly consider exporting your namespace using DFSUtil.exe periodically. The XML-based output file may be imported back into the namespace (or a completely new namespace) in the event of a problem, or it may be utilized simply as a historical record of the namespace's design. The export should be considered in addition to regular Active Directory (system state) and namespace server backups (you are backing up Active Directory regularly, right???). Trust me... you won't realize the value of this exported namespace data until you experience a situation where it takes you hours to recover the namespace rather than minutes. A sample command to export a namespace 'sales' in domain 'contoso.com' is:

    dfsutil /root:\\contoso.com\sales /export:c:\NameSpaceBackups\sales_namespace_1-10-2011.txt

    Install Service Updates

    Ensure that you are running the latest version of DFS Namespace-related components. The articles listed in the link below are routinely updated to reflect the latest updates available for both DFSN and DFSR: http://www.microsoft.com/windowsserversystem/dfs/hotfixes.mspx

    Increase the scalability of DFSN

    If your namespaces host thousands of links and operates in 'Windows 2000 Server mode', strongly consider converting the namespace to 'Windows Server 2008 mode'. You will gain increased scalability and the option to use Access Based Enumeration (ABE). For more information, please see http://technet.microsoft.com/en-us/library/cc753875.aspx.

    Utilize the Best Practices Analyzer

    Lastly, on Windows Server 2008 R2 DFSN servers run the Best Practices Analyzer for File Services. While it is focused on DFSN settings of the local server, it covers a few scenarios not covered by DFSDiag. More information about the BPA, please visit http://technet.microsoft.com/en-us/library/ff633466(WS.10).aspx and also download an updated version of the BPA here.

    Any distributed service can be very difficult to monitor and maintain. My hope is the strategies and methods above keep your DFS Namespaces in tiptop shape. Happy DFSN'ing!

    - Dave “Fire Marshall Bill” Fisher

  • Friday Mail Sack: The Year 3000 Edition

    Hello all, Ned here again. Today we talk DCDIAG, DFSN, DFSR, group policy, user profiles, migrations, USMT, and the fuuuuuuturrrrrrrrre.

    Question

    I have a mixed environment of Win2003 and Win2008 DCs. When I run DCDIAG.EXE it tells me the Windows Server 2003 DCs are failing a service test around RPCSS:

    Starting test: Services

          Invalid service type: RpcSs on DC01, current value

          WIN32_OWN_PROCESS, expected value WIN32_SHARE_PROCESS

       ......................... DC01 failed test Services

    I see some Internet posts that say I should change the value using the SC.EXE command. Do you know why this is different and what’s going on? It looks like the difference is being a service in a shared versus isolated process.

    Answer

    It’s expected and normal for this service’s behavior type to be 0x10 on Win2003 and 0x20 on Win2008 and later. Do not change it based on what DCDIAG says unless you are running the version of DCDIAG that goes with that OS (this is where much of the Internet got confused on causality versus correlation). Win2008 DCDIAG doesn’t know that Win2003 was designed this way so he can’t give you a reasonable answer – he just wants it to be default in 2008 terms.

    Your assumption around shared versus isolated is totally correct:

    Win2008 R2
    clip_image002

    Win2003
    clip_image002[4]

    Between Win2003 and Win2008, the behavior changed for the RPC service, but there was nothing yet to “share” in that svchost.exe process. In Win2008 R2, the new RPCEptMapper service was added to that shared svchost. You can see who would launch in that same process by looking for this value in the service registry keys:

    %systemroot%\system32\svchost.exe –k RPCSS

    Later versions of Task Manager make this easier too, if you’re allergic to command-line:

    image

    Svchost.exe exists mainly to lower computer resource usage: the more DLLs that can run in fewer shared processes, the less memory/CPU the OS has to allocate for services. You might think it was OK to change this on Win2003 to stop the error and maybe even get back some resources. The problem with that theory is that on Win2003, you get no resources back (as no one else is going to share that process) and you open yourself up to weird issues – when I tell Windows developers about issues caused services being modified by customers, their first response is “Why on earth would anyone change the service? We don’t test for that at all!”

    Playing around with service configurations is not something you do without valid reason and some tool complaining doesn’t meet that bar.

    Best long term solution: get rid of those remaining Win2003 servers. Then you get all sorts of advantages, like features unlocked by higher functional levels or magically load-balancing bridgeheads

    Plus I get paid.

    Question

    Is there a way to disable and enable DFS namespace targets from the command-line? We’re building some automation.

    Answer

    You can use the Win2008/Vista RSAT (or later) versions of dfsutil.exe with this syntax:

    dfsutil property state offline <DfsPath> [<\\server\share>]
    dfsutil property state online <DfsPath> [\\server\share]

    Nicely buried…

    Question

    When I use RSOP.MSC on a Windows 7 computer, I see a lot of missing entries and errors and whatnot.

    Answer

    Blink and you may miss the reason why:

    image

    Since Vista, the OS has been trying to tell you not to use this tool (which is no longer updated and has no idea about a great number of policies). To get a nice, readable resultant set of policy you need to use GPRESULT.EXE /H foo.htm. Mike has yammered about this before.

    Question

    I was curious - has the team heard what the future is for Active Directory, beyond Win2008 R2?

    Answer

    Lots (that’s my full time job now) but we cannot discuss anything. Don’t worry, the marketing people won’t keep it a secret one moment longer than necessary. And our fearless leader lets things out every so often.

    Question

    Can the new MIGAPP.XML included in KB2023591 be used with USMT 3.01?

    Answer

    [A reprint of a comment reply made to the Deployment Guy site]

    The 4.0 migapp.xml does "work" when used with USMT 3.01 - and by that I mean it is schema compatible, will not cause a fatal error during 3.0 scanstate/loadstate, and will not corrupt the store in any way that I have identified. However, under the covers it may be causing issues within the migration. That XML and Office 2010 have not been tested in any fashion with USMT 3 (and never will be), so while it might appear to work fine on the surface, we have zero idea of any more insidious problems.

    Now, if you are using USMT 3.01 because you have to - such as migrating from Win2000 or to Win XP - I can offer you a supported workaround: migrate to a computer that has Office 2007 installed, then upgrade the Office install to 2010 after the migration is done but before the users log on. Office 2010 will upgrade the Office 2007 settings (mostly – see that KB for details on the limits). 

    Naturally, if you don’t have to use 3… use 4.

    Question

    We have Windows Server 2003 DFSR and have started to explore adding Win2008 R2 servers. Is mixing supported and are there any known issues?

    Answer

    Supported all day. You will need to install this hotfix on all Win2003 R2 DFSR servers:

    KB2462352 DFSR fails from a computer that is running Windows Server 2008 R2 to a computer that is running Windows Server 2003 R2
    http://support.microsoft.com/default.aspx?scid=kb;en-US;2462352

    You will also need the Win2008 (version 44) or later AD schema added if you want to use DFSR for RODCs and if you wanted to customize staging compression behavior:

    What are the Schema Extension Requirements for running Windows Server 2008 DFSR?
    http://blogs.technet.com/b/askds/archive/2008/07/02/what-are-the-schema-extension-requirements-for-running-windows-server-2008-dfsr.aspx

    If you want to use Win2008/R2 DFSR throughout and start replacing old servers (and you really should – we’re working pretty hard on the 3rd OS since 2003 came out):

    Series Wrap-up and Downloads - Replacing DFSR Member Hardware or OS
    http://blogs.technet.com/b/askds/archive/2010/09/10/series-wrap-up-and-downloads-replacing-dfsr-member-hardware-or-os.aspx

    Question

    I have a large number of users with computers that were in a workgroup. They are now moving to a domain, and we need their user profiles converted. USMT seems to be overly complex for me – is there another way?

    [Asked by multiple customers this week, oddly enough. The last gasps of Netware?]

    Answer

    Yes, we have two ways to do this:

    MOVEUSER.EXE - XP and older, comes from the resource kit

    Win32_UserProfile WMI - Vista and newer:

    These tools correctly change permissions and ProfileList registry settings in order to “move” (i.e. convert) a user profile between local and domain accounts.

    Other Dorky Goo

    • This year is gonna be a sci-fi movie bonanza:
    I didn’t want to like it… but I did.
    The name is Bond. Wyatt Bond.
    No shots of Bucky yet.
    Close encounters of the eleventyth kind
    • Speaking of which, I was able to fight my way through the e-crowds and get tickets to Comic-Con 2011 for self and the wife. She is not exactly geeky but is an epic people watcher – she especially wants to see the day care center. Her theory being that kids will be wearing little gray suits and power ties to rebel against their parent’s uber-nerdiness. Anyone else going?
    • The latest Cracked photo contest was a zingfest - If Everything Was Made By Apple. My favorite was this subtle dig (pretty timely, having read about their latest iPhone security woes yesterday):

    Have a nice weekend folks.

    Ned “the future, Conan?” Pyle

  • New KB Articles 2/6–2/19

    I was out of the office last week and did not have a chance to post the new KB articles, so this week you’re in for double the fun.  There are quite a few updates between the last two weeks.  Here they are, separated by week:

    2/13/2011 – 2/19/2011

    Article #

    Title

    977611

    After you apply a GPO to redirect a folder to a new network share, the redirected folder is empty on client computers that are running Windows Vista or Windows Server 2008

    2498185

    How To Diagnose Active Directory Replication Failures

    976033

    "Terminal Session" targeting item does not work for a Group Policy preferences setting on a client computer that is running Windows Server 2008 or Windows Vista

    981704

    The file name of an ADM file is displayed incorrectly in the GPMC report in Windows Vista, in Windows Server 2008, in Windows 7 or in Windows Server 2008 R2

    322244

    How to Turn On Debug Logging of the Active Directory Users and Computers Snap-In

    977511

    About the DFS Namespaces service and its configuration data on a computer that is running Windows Server 2003 or Windows Server 2008

    2/6/2011 – 2/12/2011

    Article #

    Title

    2493933

    FIX: The "Validate server certificate" option is enabled on a computer that is running Windows Vista or Windows Server 2008 when you disable this option by using a Group Policy object

    2464365

    The ACL permission of some DFS folders is  incorrectly reset after you restart the DFS Namespace service in Windows Server 2008

    2505568

    Cannot use Romanian "Ș" character in computer name on Windows 7

    2492505

    Computer does not crash when the disk is full after CrashOnAuditFail is set in Windows 7 or in Windows Server 2008 R2

    2460922

    Group Policy preference item-level targeting does not work for 64-bit versions of Windows 7

    2462585

    You cannot save changes to the Dial-in property settings of a user account on a Windows Server 2008-based domain controller or on a Windows 7-based computer that has RSAT installed

    2468353

    The MPR still calls the NPPasswordChangeNotify function to notify a password change event in Windows 7 or in Windows Server 2008 R2 even though the password change is unsuccessful

    2506030

    Text on logon screen or lock screen may be truncated with some third-party credential providers on Windows 7

    2466048

    Previous versions of a file or of a folder in a DFS share are not listed if you access the share through a nested DFS link from a computer that is running Windows 7 or Windows Server 2008 R2

    2424375

    A remote desktop session may be incorrectly disconnected when a smart card is removed in another remote desktop session in Windows Server 2008 or in Windows Server 2008 R2

    2471430

    You cannot restore large files in the NTFS file system when all the data streams that have sparse attributes are deleted in Windows Server 2008 or in Windows Vista

    2413670

    Events 1659, 1481, and 1173 are recorded in the Directory Service event log on Windows Server 2008 R2-based domain controllers after you remove Active Directory Domain Services from the last domain controller in a tree root domain

    2445324

    An AD FS-enabled web application that is published for AD FS authentication on a Windows Server 2008 R2-based computer cannot decode session cookies that are received out of order

    2468316

    A paged LDAP query fails on the second page and the pages that follow in Windows Server 2008 R2

    2483564

    Renewal request for an SCEP certificate fails in Windows Server 2008 R2 if the certificate is managed by using NDES

    2462137

    The Active Directory Users and Computers MMC snap-in and Active Directory Administrative Center display Serbia and Montenegro as one country instead of as two countries in Windows Server 2008 R2 and in Windows 7

    2411938

    "An external error has occurred" error when you change the user rights of an account in Local Security Policy in Windows Server 2008 R2

    2461385

    The DFS Replication service leaks download tasks, and an outgoing replication backlog occurs in Windows Server 2008 R2

    2301288

    A Remote Desktop Services session is disconnected automatically if you apply the "Interactive logon: smart card removal behavior" Group Policy setting in Windows Server 2008 R2 or in Windows 7

    2466181

    A crash in the Ntfs.sys component on a computer that is running an IA-64-based version of Windows Server 2008 R2

    2466373

    BACKSPACE or arrow keys do not work in MMC on a computer that is running Windows 7 or Windows Server 2008 R2