Blog - Title

June, 2010

  • Friday Mail Sack: 1970’s Conversion Van Edition

    Hello folks, Ned here again with another ridiculously overdue Friday Mail Sack. This week we talk about patching, admin rights, Kerberos, hiring, ADMT, and PKI. Next week we talk about… nothing. I will be out celebrating an Important Wife Birthday™ and unless Jonathan takes pity on you, there will be crickets. So bother him A LOT for me, would you?

    Now…let’s get groovy.


    What are the best practices for installing security updates on Domain Controllers? I always transfer the FSMO roles before rebooting any DC, is it correct, wrong? Is there anything else I should monitor or do before or after the restarts?


    There’s no requirement that you move the FSMO roles as none of them need to be online for general domain functionality in the short term; heck, I had one customer with a PDCE offline for more than a year before they noticed – nice! Even if something awful happens and the DC doesn’t immediately come online, most of the FSMO roles serve no immediate purpose (like Schema Master and Domain Naming Master) or are used in only a periodic/failsafe role (RID, PDCE, Infrastructure) where a few minutes won't matter.

    The important things are pretty common sense, but I’ll repeat them:

    1. Make sure not all DC’s are rebooted at once; stagger them out a bit.
    2. Make sure clients are not pointing to DC’s acting as DNS servers that are all being rebooted at once.
    3. Make sure you are using a patching system so you don’t miss DC’s; these include WSUS, SCCM, or a third party.
    4. Do it all off hours to minimize service interruption and maximize recover time if a DC doesn’t want to come back up!


    What group do I use to install security updates on DC’s and member servers if I do not want those users being Administrators.


    It’s called “Power Unicorn Operators”.


    No such group. Non-admins cannot install patches and security updates, and this is very much by design. If they could, they could also uninstall them – making a system unsecure. If they could, they could also install malware masquerading as patches and security updates, compromising a system. Use WSUS (free), SCCM ($), Automatic Updates with “download and install” (free), or a third party ($). Installing updates by hand is only going to work for admins, but even then it’s a poor management solution. Just ask all my Conficker-infected customers that were using that methodology…

    Man, what a kick-$%# group that would be!!!



    I want to create a startup script via GPO. When I use the DC's FQDN in the path the script runs just fine - i.e. \\\sysvol\netlogon\script1.cmd But when I specify DC1's IP address, the script silently fails: \\\sysvol\netlogon\script1.cmd

    I suppose it is an authentication issue (Kerberos?), but I cannot prove it – am I right?


    You are correct, it is Kerberos. :-)

    When a domain-joined client starts up and talks to an AD DC, it must use Kerberos as NTLM is not allowed for computer-to-computer communication. When given a network resource, it needs to be able to pass that host and service info off to the KDC to request a TGS ticket. For that to work, you have to be able to take that computer/service info and use it to find a Service Principal Name, and that starts with a computer or user principal.

    So when you give it an IP address, there is no way to get a SPN, and therefore no way to get Kerberos. So it fails, expectedly and by design. You need to use FQDN (or if you must, NetBIOS name). You will see all this in a network capture as well.

    Question (Continued)

    The key was “…NTLM is not allowed for computer-to-computer communication...”

    That really makes sense now :-). But staring at a network trace, captured during XP startup, I noticed the PC is looking for SPN CIFS/ (when I used DC’s IP address in the startup script path). I was tempted and I added this SPN in the ServicePrincipalName attribute of DC’s object in the lab. After restarting both machines – the startup script run even with DCs IP in the file path (i.e. \\\sysvol\netlogon\script1.cmd ).

    Sounds logical, but is it practical? I suppose this is one of the “do not ever do this!” things? What would be the impact (security/design) if I add SPN like this?


    Oh you sneaky engineers in the field, always clever and always hacking. :-)

    Possible, yes. Practical, no. For a few reasons:

    1. The computer will not self-maintain that SPN, unlike the other SPN’s.
    2. This means you will need to maintain this on all SPN’s for all file servers.
    3. It also means you need to remember to change this when IP addresses change, or serious confusion will ensue.
    4. It also means all IT staff will need to know this, since you will not be there forever and you may like taking vacation from time to time.
    5. It also means that if anyone forgets any of this, huge numbers of computers will not be getting policy/scripts and unless you are monitoring all client event logs, you won’t know it.
    6. Update Jan 21 2011: and starting in Vista, it won't work at all!

    So all that adds up to not recommended, leaning towards highly discouraged. Not to mention that pointing to a specific server isn’t needed when using DFSN (such as with SYSVOL). This will work perfectly well and guarantee the computer talks to the nearest DC first, then continue to work if that DC is down:




    I am trying to install ADMT 3.2 and there’s an error that it is not a valid Win32 application.



    Naughty naughty, you did not read the requirements. You are trying to install this on a Windows Server 2008 or Windows Server 2003 computer running x86 (32-bit). ADMT 3.2 only installs on Windows Server 2008 R2. Since that can only be x64, the installer was only compiled in 64-bit. When you run x64 on x86, you naturally get that error.

    If you tried to install this on Win2003/2008 X64, it would instead say that it requires Win2008 R2.


    I’ve not seen the Weekly DS KB articles from the AD Team blog for a while…. Is it because there aren't any? Or are you just no longer providing those?


    No, Craig just got a bit behind. He plans to resume that soon. Soon being sometime between now and the zombie apocalypse.


    Holy crap, do you believe we put pictures like that in Office Clipart?! We must give kids nightmares.


    Is constrained delegation between different domains (with trusted relationship) ever going to be supported? Maybe Windows Server 2014, Windows Server 2020, Windows server 2096, etc.  ;-)


    While I cannot speak about future releases, this is definitely something we get asked about all the time. When you ask us over and over for something, that helps make it more likely - not guaranteed, mind you - to happen. So if you have a Premier contract, whale on your TAM and let them know you need this functionality and why. The more compelling the argument and the more often it is made, the more likely to get examined for a future release. This goes for pretty much everything in Windows.


    We just created a new Win2008 R2 PKI (one Root CA and one Issuing Sub CA). We have two domains, so we placed the CA’s in our child domain, as we have an empty forest root domain. Should we have placed those CA’s in the empty root?


    [This answer courtesy of Rob Greene – Ned]

    I would recommend that you put the CA’s in the domain where the largest amount of certificate requests are going to be generated.  I say this this because if you configure your certificate templates to publish the certificate in AD, then the CA computer will contact a local domain controller to get it added to the domain. Less traffic, less hopping, generally more efficient.

    The other thing I would recommend is to add the CA’s computer account to the Cert Publishers group in both the child and root domains.  This allows the CA to publish certificates for users / computers in both domains.  


    I heard you are hiring, what are some good things to study up on if I want to interview and really rock your face off?


    Start below. These are the core technologies - mainly as represented in XP and 2003 – that every DS Support Engineer has to know inside and out to be worth a darn in MS Support. Once you have those down you can find the Vista/08/7/R2 differences on your own.

    Active Directory
    Active Directory Replication Model
    Active Directory Replication Topology
    Group Policy
    Interactive Logon
    Public Key Infrastructure (PKI)
    User Profiles

    Note: you can use these free trial editions below in order to do live repros of all this, and repros are highly suggested. Especially with the use of Netmon 3.4 to see how things look on the wire. Running these in Hyper-V, in Virtualbox, etc. will make the materials more understandable.

    Next time I’ll give some links to the post-graduate level studying. Most people think they know these above really well… then the hyperventilating starts in the interview.

    Until next time,

    - Ned “shag carpet” Pyle

  • Announcing the Group Policy Search service

    Hello, Kapil here. I am a Product Quality PM for Windows here in Texas [i.e. someone who falls asleep cuddling his copy of Excel - Ned]. Finding a group policy when starting at the "is there even a setting?" ground zero can be tricky, especially in operating systems older than Vista that do not include filtering. A solution that we’ve recently made available is a new service in the cloud:

    Group Policy Search

    With the help of Group Policy Search you can easily find existing Group Policies and share them with your colleagues. The site also contains a Search Provider for Internet Explorer 7 and Internet Explorer 8 as well as a Search Connector for Windows 7. We are very interested in hearing your feedback (as responses to this blog post) about whether this solution is useful to you or if there are changes we could make to deliver more value.

    Note - the Group Policy search service is currently an unsupported solution. At this time the maintenance and servicing of the site (to update the site with the latest ADMX files, for example) will be best-effort only.

    Using GPS


    In the search box you can see my search string “wallpaper” and below that are the search suggestions coming from the database.

    On the lower left corner you see the search results and the first result is automatically displayed on the right hand side. The search phrase has been highlighted and in the GP tree the displayed policy is marked bold.

    Note: Users often overlook the language selector in the upper right corner, where one can switch the policy results (not the whole GUI itself) to “your” language (sorry for having only UK English and not US English ;-) [Whut the heck ur yoo tawkin' 'bout - Ned])


    Using the “Tree” menu item you can switch to the “registry view”, where you can see the corresponding registry key/value, or you can reset the whole tree to the beginning view:


    In the “Filter” menu, you can specify for which products you want to search (this means, if you select IE7, it will find all policies which are usable with IE7, not necessarily only these only available for IE7 and not for IE6 or IE8; this is done using the “supported on” values from the policies):


    In the “copy” menu you can select the value from the results that you want to copy. Usually “URL” or “Summary” is used (of course you can easily select and CTRL+C the text from the GUI as well):


    In the “settings” menu you can add the search provider and/or Connector.


    Upcoming features (planned for the next release)

    • “Favorites” menu, where you can get a list of some “interesting” things like “show me all new policies IE8 introduced”:


    • “Extensions” menu:


    • We will introduce a help page with a description for the usage of the GPS.

    GPS was written by Stephanus Schulte and Jean-Pierre Regente, both MS Internet Explorer Support Engineers in Germany. Yep, this tool was written by Support for you. :-) 

    The cool part – it’s all running in:


    Kapil “pea queue” Mehra

  • We are hiring

    We are hiring again (in your face, 2008 economy!) and if you think you have what it takes to do my job, come show me:

    Support Escalation Engineer, Directory Services

    We are looking for talented, experienced, motivated engineers who want to take their career and IQ to the zenith. This is the most challenging technical job you will ever have, but it is also the most rewarding: it is never boring, you work on the most complex environments, and you will learn more in one month about Directory Services than you will in five years out in the field - I personally guarantee that. All this and more can be yours.

    It also doesn't hurt that Microsoft benefits are unparalleled and Microsoft is on Fortune's 100 best companies to work for list year after year. :-)

    These positions are in Charlotte, NC and Las Colinas, TX. If you are anywhere in the United States, you're just a few hours away from changing your life forever. Come and get it.

    - Ned "I charge for autographs" Pyle

  • ADMT 3.2 released

    Come and get it. Especially you commenters that liked to swear at me anonymously :-). Besides now supporting Windows Server 2008 R2 and fixing some bugs, this version now supports Managed Service Accounts.

    Active Directory Migration Tool version 3.2 - download here.
    ADMT 3.2 Migration Guide (DOC) - download here

    System Requirements

    • Supported Operating Systems: Windows Server 2008 R2
    • ADMT can be installed on any computer capable of running the Windows Server 2008 R2 operating system, unless they are Read-Only domain controllers or in a Server Core configuration.
    • Target domain: The target domain must be running Windows Server 2003, Windows Server 2008, or Windows Server 2008 R2
    • Source domain: The source domain must be running Windows Server 2003, Windows Server 2008, or Windows Server 2008 R2
    • The ADMT agent, installed by ADMT on computers in the source domains, can operate on computers running Windows XP, Windows Server 2003, Windows Vista, Windows Server 2008, Windows 7, and Windows Server 2008 R2.

    The updated 3.2 Migration Guide is coming shortly, I will update this post with that link when I have it. It's out.

    - Ned "you wouldn't say that to my face!" Pyle

  • Friday Mail Sack: Ride ‘Em Cowboy Edition

    Howdy partners, Ned here. This week we talk event logs, auditing, NTLM “fallback”, file server monitoring, and SCOM 2007 management pack dissection. It was a fairly quiet week for questions since everyone is off for vacation at this point, I reckon. That didn't mean it wasn't crazy at work - our folks take vacation too, and that leaves fewer of us to handle the cases. Hopefully you weren't on hold too long...

    Oh, and it’s my fifth anniversary as an employee at Microsoft today. So being from the Midwest and not wanting to do the usual Microsoft M&M cliché, I brought 5 pounds of delicious Hickory Farms meat. It disappeared fast, people here are animals. Sausage-loving animals.

    Anyhow, on to the goods.


    Is there a way to set security logs to be retained for X days automatically? What about having them automatically archive?


    Starting in Windows Vista we added Group Policy to handle the archiving piece. See:

    Computer configuration \ <policies> \ Administrative templates \ Windows components \ Event Log Service \ Security \

                    Backup log automatically when full
                    Retain old events

    This also works for Application, Setup, and System logs. The big old chatty ones.


    This does not help you on age, but if you are archiving the log every time it fills you get the same effect. Obviously you would need to start backing up all these event logs and deleting them or you would risk running out of disk space. And what about Windows Server 2003, you ask? We have a registry key there that will do the same thing – see the AutoBackupLogFiles value buried in KB312571.

    Rather than going this route though, I instead suggest deploying some kind of security event collection tool, like System Center 2007’s free ACS component or a third party. It will scale much better and be less of a hassle to maintain. Then you are always intercepting and collecting your security events. Hopefully you have a plan to do something with them!


    <A conversation about why you should not skew clocks as that makes Kerberos break, as everyone knows. But then:>

    However the vast majority of app servers should “just work” with NTLM fallback when Kerberos doesn’t work, correct?


    Not necessarily! When MS started implementing Kerberos eleven years ago, NTLM was being replaced as the preferred security protocol. However, we knew that a million apps and down-level or 3rd party clients would not be able to handle Kerberos through negotiation. In order to make the experience less painful, we decided that when using the Windows Negotiate security package, we’d allow applications to first try Kerberos and if that failed, then try NTLM. Pretty much any failure was ok, such as the target server not supporting Kerberos or Kerberos being possible but malfunctioning due to environmental problems. If you simply asked for Kerberos only or NTLM only, there was no fallback because you were being specific. Some languages also provide for blocking fallback post negotiation, such as WCF’s ALLOWNTLM=FALSE flag. So NTLM fallback was never guaranteed or even tried in many scenarios. There are a lot of misunderstandings and mythology about this out there, but this is how it works - when it comes to your specific app, just test it under a network capture to see how it behaves.

    Then starting Windows Vista SP1 and Windows Server 2008 we made a significant change – from then on, interactive logon stopped allowing NTLM fallback if Kerberos had errors. So for example, if someone duplicated a DC’s SPN, the user cannot logon (with error “The security database on the server does not have a computer account for this workstation trust relationship”) and examining their event log would show KDC 11 error and you'd see 4625 events on the DC security log. So if Kerberos was supposed to work and didn’t, too bad - no more fallback. Obviously that is also in place in Windows 7 and Win2008 R2, and for the foreseeable future.

    Furthermore, in Windows 7 and Windows Server 2008 R2 we added a new extension to the negotiate security package to start making fallback less likely everywhere, not just in interactive logon. That is called negoexts and does stuff like federation support –  from the beginning it has no concept of fallback at all.

    So why change all this? Because it’s more secure. Better to prevent auth rather than allow someone to somehow damage Kerberos then use that opportunity to go through a weaker protocol.


    I would like to start examining the File Services management pack and other MP for System Center Operations Manager 2007. I don’t always find complete documentation on what these packs do (or I find this). I’d also rather not download and configure 1.25GB of SCOM trial edition just yet either.

    What to do?


    Here’s the trick:

    1. Install the System Center Operations Manager 2007 R2 Authoring Resource Kit (free download, very small)

    2. Install the management packs you are interested in (such as File Services MP).

    3. Start the Authoring Console and load your management pack. Generally, the “Library” MP will contain the majority of info – that’s why it’s bigger than the other files. For our File Services example:



    4. Now in the Health Model area you will see all the monitoring… goo. In this case, DFSR monitoring stuff:


    5. Now start drilling down below the Aggregate Monitors. There’s a lot to see here. 


    6.  At a glance, you can see some interesting info about each monitor:


    7. If you click Edit Monitor, then the Product Knowledge tab, you can see how the monitor works, what the known causes are of the issue, what the resolutions are, and more info to take in. This is the part that makes you smart.




    This works anywhere; you don’t even need to install a server – I am doing this all on my Win7 client.

    And what this really highlights is just how important using these monitors are. The resolution sections are written by the Product Group to tell you the appropriate way to fix things and in many cases are also vetted and expanded on by MS Support. I spent a hellish few weeks going through the File Services one for example: 200 pages of spec, arrrrghhhh! Rather than relying on some uninformed stranger on the Internet you can instead get the official answer to each problem that SCOM finds, and it can even react on your behalf. It’s slick stuff.


    I learned an important lesson today. When you are in a team meeting and you describe some broken process as “a real goat rodeo”, your colleagues will use the opportunity to remind you how short you are using terrible artwork:


    And if you complain that the picture isn’t “bling” enough, they will improve it Kanye West style:



    Have a great weekend folks.

    - Ned “Yeeeeeee-hhhhhhaaaaaahhhh!”Pyle

  • How to Virtualize Active Directory Domain Controllers (Part 2)

    Hello everyone, this is Shravan from the Active Directory team and Jason from the System Center VMM team here at Microsoft. This is part 2 of the blog series where we discuss how to migrate Active Directory domain controllers to a virtualized system. Last time we discussed how to plan for moving the physical domain controllers to virtual servers, identified the concerns with USN rollback and the methods of performing the P2V migrations. Here we will identify the competitive features of SC VMM, system requirements to identify source machines and target servers and some tricks with the tool we have learnt working with other customers..

    Why is SC VMM better?

    Basically, SCVMM has two modes of P2V operation - ONLINE and OFFLINE. In the ONLINE mode, the source and destination are kept turned ON during the migration process whereas in OFFLINE mode, the source machine is turned OFF before the restore process is completed on the Destination or Virtual DC. OFFLINE Mode is the recommended P2V method for DCs.

    First and foremost, here’s the gotcha – the default selection for SCVMM is ONLINE mode. The option to change it is hidden under the “Conversion Options” expanded menu as shown below.


    This has been easily overlooked by me and some of my customers but it results in a warning message below stating “Online physical to virtual conversion of domain controller is not recommended” but the wizard lets you proceed with it. Unless you read the warning message and stop to find out where the switch to OFFLINE P2V conversion is, you may run into the problem with USN Rollback we discussed earlier.


    Going back to the previous screen, we expand the Conversion options and choose OFFLINE conversion. Also it’s recommended to select the checkbox for “Turn off source computer after conversion” to avoid the potential for a USN rollback.


    Additionally when you choose OFFLINE Conversion mode, you get presented with the following UI which lets you select how you want to handle the IP assignment on the virtual DC.


    Below I have pasted in some important verbatim from the following article:

    P2V: Requirements for Physical Source Computers

    Requirements on the Source Machine

    To perform a P2V conversion, your source computer:

    • Must have at least 512 MB of RAM.
    • Cannot have any volumes larger than 2040 GB.
    • Must have an Advanced Configuration and Power Interface (ACPI) BIOS Vista WinPE will not install on a non-ACPI BIOS.
    • Must be accessible by VMM and by the host computer.
    • Cannot be in a perimeter network. A perimeter network, which is also known as a screened subnet, is a collection of devices and subnets placed between an intranet and the Internet to help protect the intranet from unauthorized Internet users. The source computer for a P2V conversion can be in any other network topology in which the VMM server can connect to the source machine to temporarily install an agent and can make Windows Management Instrumentation (WMI) calls to the source computer.

    The following restrictions apply to P2V operation system support:

    • VMM does not support P2V conversion for computers with Itanium architecture based operating systems.
    • VMM does not support P2V on source computers running Windows NT Server 4.0. However, you can use the Microsoft Virtual Server 2005 Migration Toolkit (VSMT) or third-party solutions for converting computers running Windows NT Server 4.0.
    • VMM 2008 R2 does not support converting a physical computer running Windows Server 2003 SP1 to a virtual machine that is managed by Hyper-V. Hyper-V does not support Integration Components on computers running Windows Server 2003 SP1. As a result, there is no mouse control when you use Remote Desktop Protocol (RDP) to connect to the virtual machine. To avoid this issue, update the operating system to Windows Server 2003 SP2 before you convert the physical computer. As an alternative, you can convert the computer by using VMM 2008 and then deploy the virtual machine in VMM 2008 R2.

    Requirements for the Destination Host Server

    In VMM, a host is a physical computer on which you can deploy one or more virtual machines. To run P2V, you need a host on which to place the image of the source computer.

    Requirements for the host server include:

    • The destination host during a P2V conversion can be running Windows Server 2008 with Hyper-V, Windows Server 2008 R2 with Hyper-V, or Virtual Server  R2  SP1 (or later).
    • The destination host cannot be in a perimeter network.
    • As in any virtual machine creation or migration, the destination host for a P2V conversion must have sufficient memory for the virtual machine in addition to memory reserved for the host operating system. By default, the amount of memory reserved for the host operating system is 256 MB in VMM 2008 or 512 MB in VMM 2008 R2. If the host does not have enough memory for the virtual machine in addition to the memory reserved for the host, you will get a placement error in the Convert Physical Server Wizard

    Deciding Which Computers to Convert

    To successfully perform P2V, you must be able to identify appropriate physical workloads for consolidation into the virtualized environment. This section will help you identify which computers are good candidates for conversion.

    Identifying Virtualization Candidates

    If you have deployed Microsoft System Center Operations Manager 2007, VMM can help you identify the right physical servers for consolidation based on direct analysis of the performance counters of the target machine or historical performance data stored in the Operations Manager database.

    The Virtualization Candidates report helps you identify underutilized computers by displaying average values for a set of commonly requested performance counters for CPU, memory, disk usage, hardware configurations, including processor speed, number of processors, and total RAM. To use the Virtualization Candidates report, you must deploy the System Center VMM 2008 Management Pack. For more information about reporting, see Configuring Reporting for VMM.

    Prioritizing Virtualization Candidates

    When identifying the best candidates for P2V conversion, consider converting these types of computers, in order of preference:

    1. Non business-critical underutilized computers. By starting with the least utilized computers that are not business critical, you can learn the P2V process with relatively low risk. Web servers may make good candidates.
    2. Computers with outdated or unsupported hardware that needs to be replaced.
    3. Computers with low utilization that are hosting less critical in-house applications.
    4. Computers with higher utilization that are hosting less critical applications.
    5. The remaining underutilized computers.
    6. In general, business-critical applications, such as e-mail servers and databases that are highly utilized, should only be virtualized to the Hyper-V platform in the Windows Server 2008 (64-bit) operating system.

    Some Problem Cases:

    • Missing Driver Issue:

    Since VMM uses WinPE to boot into the source when performing the OFFLINE migration, if the drivers for any device on the source machine doesn’t exist in WinPE, you may get an error similar to the one below:

    No compatible drivers were identified for the device: <DEVICE_NAME>

    For instance, if the VMM server does not have the drivers for the NIC “3COM 3C920 Integrated Fast Ethernet Controller” that is present on the physical source DC, then you will see an error similar to the one below as we require the driver in order to boot using WinPe on the physical DC.


    • If you receive the error above, you need to copy the drivers for the NIC “3COM 3C920 Integrated Fast Ethernet Controller” on the VMM server in the following folder location <%ProgramFiles%\Microsoft System Center Virtual Machine Manager 2008 R2\Driver Import> on the VMM server and click the “Check Again” button which should retry the process.
    • Disk Issue:

    If the primary boot and active partition on a server is FAT32, then SCVMM will be unable to perform the migration.



    While VMM does support migration of the FAT32 partitions to the target virtual guest, it does NOT support migration of servers where FAT32 partition is the boot and active one.


    That’s it for now. We are certain that there are a million other hardware combinations that we may not have tested in our above experiments but we hope to hear back from you with any specific situations you may have run into during your journey to a virtualized world.

    Happy Virtualization!

    -Shravan Kumar and Jason Alanis.

  • Friday Mail Sack: Shut Up Laura Edition

    Hello again folks, Ned here for another grab bag of questions we’ve gotten this week. This late posting thing is turning into a bad habit, but I’ve been an epileptic octopus here this week with all the stuff going on. Too many DFSR questions though, you guys need to ask other stuff!

    Let’s crank.


    Is it possible to setup a DFSR topology between branch servers and hub servers, where the branches are an affiliate company that are not a member of our AD forest?


    Nope, the boundary of DFSR replication is the AD forest. Computers in another forest or in a workgroup cannot participate. They can be members of different domains in the same forest. In that scenario, you might explore scripting something like:

    robocopy.exe /mot /mir <etc>


    I was examining KB 822158 – with the elegant title of “Virus scanning recommendations for Enterprise computers that are running currently supported versions of Windows” - and wanted to make sure these recommendations are correct for potential anti-virus exclusions in DFSR.


    They better be, I wrote the DFSR section! :-)


    Is there any way to tell that a user’s password was reset, either by the user or by an admin, when running Win2008 domains?


    Yes – once you have rolled out Win2008 or R2 AD and have access to granular auditing, this becomes two easy events to track once you enable the subcategory User Account Management:




    An attempt was made to change an account's password.  


    An attempt was made to reset an account's password.


    Once that is turned on, the 4724 event tells you who changed who’s password:


    And if you care, the 4738 confirms that it did change:


    If a user changes their own password, you get the same events but the Subject Security ID and Account Name change to that user.


    Any recommendations (especially books) around how to program for the AD Web Service/AD Management Gateway service?


    Things are a little thin here so far for specifics, but if you examine the ADWS Protocol specification and start boning up on the Windows Communication Foundation you will get rolling.

    Windows Communication Foundation

    WCF Books -

    [MS-ADCAP]: Active Directory Web Services: Custom Action Protocol Specification

    Remember that we don’t do developer support here on AskDS so you should direct your questions over to the AD PowerShell devs if you get stuck in code specifics.


    Is their any guidance around using DFSR with satellite link connections?


    Satellite connections create a unique twist to network connectivity – they often have relatively wide bandwidth compared to low-end WAN circuits, but also have comparitively high latency and error levels. When transmitting a packet through a geosynchronous orbit hop, it hits the limitation of the speed of light – how fast you can send a packet 22,000 miles up, down, then reply with a packet up and down again. And when talking about a TCP conversation using RPC, one always uses round trip times as part of the equation. You will be lucky to average 1400 millisecond response times with satellite, compared to a frame relay circuit that might be under 50ms. This also does not account for the higher packet loss and error rates typically seen with satellite ISP’s. Not to mention what happens when it, you know, rains :-).  In a few years you can think about using medium and low earth orbit satellites to cut down latency, but those are not commercially viable yet. The ones in place have very little bandwidth.

    When it comes to DFSR, we have no specific guidance except to use Win2008 R2 (or if you must, Win2008) and not Win2003 R2. That first version of DFSR uses synchronous RPC for most communications and will not reliably work over satellite’s high latency and higher error rates – Win2008 R2 uses asynchronous RPC. Even Win2008 R2 may perform poorly on the lower bandwidth ranges. Make sure you pre-seed data and do not turn off RDC on those connections.


    Totally unrelated, I found this slick MCP business card thing we’re doing now since we stopped handing out the laminates. It’s probably been around for a while now, but hey, new to me. :) If you go to and provide your MCP ID # and Live ID you can get virtual business cards that link to your transcript.

    Then you can have static cards: 

    Or get fancy stuff like this javascript version. Mouse over the the right side to see what I mean:

    Oh yeah, did you know my name is really Edward? They have a bunch of patterns and other linking options if you don't want graphics; give it a look. 


    Finally, I want to welcome the infamous Laura E. Hunter to the MSFT borg collective. Author and contributor to TechNet Magazine, the AD Cookbook, AD Field Guide, Microsoft Certified Masters, and endless boring a considerable body of ADFS documents, Laura is most famously known for her blog. And now she’s gone blue – welcome to Microsoft, Laura! Now get to work.

    Have a nice weekend folks,

    - Ned “what does the S stand for Bobby?” Pyle

  • How to Virtualize Active Directory Domain Controllers (Part 1)

    Hello Everyone, this is Shravan from the Active Directory team and Jason from the System Center VMM team here at Microsoft. We will be discussing a scenario that comes up often: how to migrate active directory domain controllers to a virtualized system.

    Why Now?

    Reduce Cost! Reduce Cost! Reduce Cost! It’s an old adage. When this conclusion reaches the folks who work within large data centers, this means a big push to consolidate how much space, cost and energy we consume on the big beefy servers. Virtualization serves as a good method to optimize the use of the server resources but data center administrators need be cautious as they proceed. Therefore let’s discuss some of the common concerns regarding virtualized domain controllers as to when/where/how to move the resources to virtual hardware.

    How to plan?

    When introducing virtualized DC’s, one needs to think of virtual DC's the same way they think about scalability planning with physical DC's with the extra dimension of virtualization platform. Conventional wisdom says not to put all eggs in one basket and avoid single point of failures as much as possible. Some of the logical examples of these single points of failure for physical DC’s are as follows:

    • All DC's in the same data center
    • All DC’s on the same network switch
    • All DC’s on the same power grid
    • All DC’s same make/model of hardware etc.

    Administrators have learned to avoid these pitfalls by adequately planning the resources. Taking this to the next level, the same applies to the virtualized DC’s as well. Here are some examples of single points of failure specifically to the virtualized DC’s:

    • Multiple DC's on a common host virtual server
    • Multiple DC’s using the same hard disk spindle
    • Multiple DC’s using the same network adaptor on a virtualized host
    • Multiple DC’s hosted on different hosts but using single UPS for power failures

    One of the most obvious single points of failure is that when the machine- on which all the virtualized solutions run - fails or when the virtualization solution itself fails. This event causes all Virtual Machines hosted by that machine to go offline. This might sound scary but actually, this risk is relatively easy to handle. Redundant capacity and regular backups of the virtualized operating systems (together with the virtualized applications) are a warranty against data loss and downtime due to the single point of failure.

    Another question is in what order to virtualize the DC's in the Hub and Branch sites. The same considerations that went into place when placing the number of physical DC’s in each site needs to be revisited. There may be specific cases which call for specific plan. Our general recommendation would be to start with optimizing the number of the DC's needed in the branch office sites first while constantly testing the load bearing capacity in each step. Then virtualize the DCs in the Hub site. Performing the steps in this bottom-up fashion ensures you don’t starve the branches sites while virtualizing your hub DCs. As always, nothing beats comprehensive testing in your own environment as one size may not fit all.

    Pardon the geek-speak while we review some performance considerations: The peak and steady state load generated by a collection of VM guests should not exceed the capabilities of the virtual host computer and network infrastructure. Specifically, collection of VM guests should not exceed the capabilities of the CPU, disk subsystem, memory, and network bandwidth on a common host computer. Some load scenarios can exceed capabilities that a DC on a single physical computer can service so multiple physical or virtual computers may be required. So for instance, we have one virtual server hosting individual virtual machines in the following roles:

    • Domain Controller (DC)
    • Exchange server front-end server
    • Exchange back-end server
    • SQL server

    The peak load on the DC as a guest is not merely dependent on the authentication traffic coming to the DC but a cumulative load on the Virtual server can also affect the capacity on the DC. Therefore, please take into account the factor the total load on the virtual server.

    While we have not seen any specific issues with any roles (FSMO, GC, DNS, RODC etc) running on virtual servers. Please take load and criticality into consideration before you make the switch to virtual or deciding to keep them as physical servers.

    Regardless of the virtual host software product that you are using, here are some rules on the “don’t do this when hosting virtualized DC guests on VM hosts.” These rules include but are not limited to the following:

    • Do not stop or pause domain controllers.
    • Do not restore snapshots of domain controller role computers. This action causes an update sequence number (USN) rollback that can result in permanent inconsistencies between domain controller databases. USN rollback is discussed further in this blog.
    • Do not perform ONLINE physical-to-virtual (P2V) conversions. All P2V conversions for domain controller role computers should be done in OFFLINE mode. System Center Virtual Machine Manager enforces this for Hyper-V. Please read further to understand the difference between ONLINE and OFFLINE modes for P2V. For information about other virtualization software, see the vendor documentation. The exception to this is tools such as disk2vhd which convert the DC while the source stays online because the virtual DC is not turned on the production network.
    • Configure virtualized domain controllers to synchronize with a time source in accordance with the recommendations for your hosting software. For Microsoft Virtual Server or Hyper-V server, turn off host time synchronization from the properties of the VM.
    • If you do not have uninterruptable power supplies (UPS) for your VM hosts or the storage disk where the active directory database resides, then ensure write-caching is disabled on the virtual machine’s host computer. Please refer this link for additional guidance. Conversely, if the write caching needs to stay enabled for the VM host which hosts the DC, then install a UPS to avoid damage to the DC(s).
    • Virtual DC’s are subject to the same backup requirements as physical DCs. Please refer this TechNet article for details.
    • Be careful when you are adding the Virtual Server host as a member of the same domain as the guest DCs it’s hosting as you may run into a Chicken & Egg problem if a DC is not available during boot time for the host.

    For more considerations about running domain controllers in virtual machines, see Microsoft Knowledge Base article 888794. Also, see the following TechNet article for additional information:

    Deployment Considerations for Virtualized Domain Controllers

    Two methods to DC virtualization

    With all that behind us let’s dig deeper into the two methods on how to introduce virtualized domain controllers into an environment.

    1. DCPromo

    Stand up a member server in the virtual environment and run dcpromo. Configure it as an additional domain controller to replicate the data from another DC in the same domain. If you want to reuse the same name as one of the physical DC’s, you must first demote the physical DC. Then rename the virtual server while still as a member server and then promote it as a physical server. If you choose to use the same name as an existing DC, ensure that you allow end-to-end AD replication of the demotion to complete prior to running dcpromo on the virtualized guest.

    2. Physical-to-Virtual (P2V)

    As per the VMM 2008 glossary, physical-to-virtual machine (P2V) conversion [describes] the process of creating a virtual machine by copying the configuration of a functioning physical computer.”. In simple terms, here we convert a physical domain controller server to a virtual domain controller guest using a P2V tool.

    Today SCVMM (System Center Virtual Machine Manager) is available from Microsoft, as are similar 3rd party P2V tools where you run the tool against a physical server to convert to a virtual server. In concept it performs a backup on the physical server and restores the machine to virtual hardware. The end result is you have converted the physical server to a virtual domain controller which looks and act as the original. You then turn off the converted physical DC and then connect the virtual DC to the network and clients don't see any difference in the functionality with authentication.

    Since most of us are familiar with dcpromo promote/demote process, we will focus on the second method of the P2V tool. If the P2V conversion goes as expected and there are no problems after the conversion, there is no service outage other than the duration where the P2V tool is performing the backup/restore. A USN rollback will occur if for some reason you decide to move back to the physical DC after you have already performed the P2V process, and the new virtualized DC has replicated with other DCs. So don’t ever do it.

    What’s USN ROLLBACK?

    Back to the geek-speak: Active Directory Domain Services (AD DS) uses update sequence numbers (USNs) to keep track of replication of data between domain controllers. Each time that a change is made to data in the directory, the USN is incremented to indicate that a change has been made. For each directory partition that a destination domain controller stores, USNs are used to track the latest originating update that a domain controller has received from each source replication partner. Also, it helps with the status of every other domain controller that stores a replica of the directory partition. When a domain controller is restored after a failure, it queries its replication partners for changes with USNs that are greater than the USN of the last change it has recorded. USN rollback occurs when the normal updates of the USNs are circumvented and a domain controller tries to use a USN that is lower than its latest update.

    If you are still wondering why are we talking about USN Rollback with our P2V tool, remember how we discussed that it’s performing a backup of the physical DC and restoring it to the virtual DC. If the virtual DC replicated with the rest of the DC’s and we try to reinstate the physical DC and bring it online, it will detect that the highest USN it has for itself is lower than what others have for it. When this happens, the physical DC detects that it’s in a USN ROLLBACK state, stops replication, and pauses the Netlogon service on machine startup. A USN rollback can also occur on the virtual DC if the physical DC isn't turned off immediately after the P2V finishes taking its backup of the original.

    Please refer the following TechNet link for a detailed understanding of USN Rollback -

    NOTE: In Windows Server 2003 (SP1) and later, USN rollback will be detected and replication will be stopped before divergence in the forest is created, in most cases. For Windows 2000 Server, the updates in Microsoft Knowledge Base article 885875 must be installed to enable this detection. Remember that Win2000 support ends on July 13, 2010 though, so your real answer here is to not be running it at all!

    The supported recovery options when in USN Rollback state are pretty limited - you have to forcibly demote the DC, perform a metadata cleanup and re-promote the domain controller.

    How to P2V Domain Controllers

    During the course of writing this blog, we did a bunch of different tests and tried out different combinations of hardware, FSMO roles, GC, domains etc. We will be sharing our takeaways during this experiment. For those who are unfamiliar with SCVMM as a product and how P2V works, the detailed steps regarding the SCVMM P2V process are thoroughly documented in the following links:

    P2V: How to Perform a Conversion

    P2V: Converting Physical Computers to Virtual Machines in VMM

    One of our customers shared the following link with us which outlines VMWARE’s P2V method which uses online migration.

    Please note ONLINE mode keeps the source and target running at the time and is not recommended. When using this un-recommended method, it’s up to the administrator to keep the network cable disconnected on the respective machines to keep them isolated. A lot of our customers experience that keeping the new target virtual DC completely isolated from the source physical DC is easier said than done. There is a big risk of USN rollback if the machines are not isolated as identified by VMWARE. We have seen a number of customers who try to perform an ONLINE P2V and end up in a USN Rollback state, leading to the forced demotion of the problem DCs.

    Good place to mention our disclaimer for any 3rd party software for virtualization.

    897615 Support policy for Microsoft software running in non-Microsoft hardware virtualization software;EN-US;897615

    By now, you should be able to able to identify some of the benefits and pitfalls of going virtual on your domain controllers. Next time we will go into the details on how to perform the Offline P2V migration of domain controllers using SC VMM, requirements on the source machines, destination servers, identifying the suitable candidates that can be moved over to the virtual world.

    More on this topic in Part 2.

    - Shravan Kumar and Jason Alanis.