Blog - Title

February, 2010

  • Inventorying Computers with AD PowerShell

    Hi, Ned here again. Have you ever had to figure out what operating systems are running in your domain environment so that you can plan for upgrades, service pack updates, or support lifecycle transitions? Did you know that you don’t have to connect to any of the computers to find out? It’s easier than you might think, and all possible once you start using AD PowerShell in Windows Server 2008 R2 or Windows 7 with RSAT.

    Get-ADComputer

    The cmdlet of choice for inventorying computers through AD is Get-ADComputer. This command automatically searches for computer objects throughout a domain, returning all sorts of info.

    As I have written about previously my first step is to fire up PowerShell and import the ActiveDirectory module:

    image

    Then if I want to see all the details about using this cmdlet, I run:

    Get-Help Get-ADComputer -Full

    Getting OS information

    Basics

    Now I want to pull some data from my domain. I start by running the following:

    Important note: in all my samples below, the lines are wrapped for readability.

    Another important note (thanks dloder): I am going for simplicity and introduction here, so the -Filter and -Property switches are not designed for perfect efficiency. As you get comfortable with AD PowerShell, I highly recommend that you start tuning for less data to be returned - the "filter left, format right" model described here by Don Jones.

    Get-ADComputer -Filter * -Property * | Format-Table Name,OperatingSystem,OperatingSystemServicePack,OperatingSystemVersion -Wrap –Auto

    image

    This command is filtering all computers for all their properties. It then feeds the data (using that pipe symbol) into a formatted table. The only attributes that the table contains are the computer name, operating system description, service pack, and OS version. It also automatically sizes and wraps the data. When run, I see:

    image

    It looks like I have some work to do here – one Windows Server 2003 computer needs Service Pack 2 installed ASAP. And I still have a Windows 2000 server that is going to move quickly and replace that server.

    Server Filtering

    Now I start breaking down the results with filters. I run:

    Get-ADComputer -Filter {OperatingSystem -Like "Windows Server*"} -Property * | Format-Table Name,OperatingSystem,OperatingSystemServicePack -Wrap -Auto

    I have changed my filter to find all the computers that are running “Windows Server something”, using the –like filter. And I stopped displaying the OS version data because it was not providing me anything unique (yet!).

    image

    Cool, now only servers are listed! But wait… where’d my Windows 2000 server go? Ahhhh… sneaky. We didn’t start calling OS’s “Windows Server” until 2003. Before that it was “Windows 2000 Server”. I need to massage my filter a bit:

    Get-ADComputer -Filter {OperatingSystem -Like "Windows *Server*"} -Property * | Format-Table Name,OperatingSystem,OperatingSystemServicePack -Wrap -Auto

    See the difference? I just added an extra asterisk to surround “Server”.

    image

    As you can see, my environment has a variety of Windows server versions running. I’m interested in the ones that are running Windows Server 2008 or Windows Server 2008 R2. And once I have that, I might just want to see the R2 servers – I have an upcoming DFSR clustering project that requires some R2 computers. I run these two sets of commands:

    Get-ADComputer -Filter {OperatingSystem -Like "Windows Server*2008*"} -Property * | Format-Table Name,OperatingSystem,OperatingSystemServicePack -Wrap -Auto

    Get-ADComputer -Filter {OperatingSystem -Like "Windows Server*r2*"} -Property * | Format-Table Name,OperatingSystem,OperatingSystemServicePack -Wrap -Auto

    image

    image

    Starting to make sense? Repetition is key; hopefully you are following along with your own servers.

    Workstation Filtering

    Okeydokey, I think I’ve got all I need to know about servers – now what about all those workstations? I will simply switch from -Like to -Notlike with my previous server query:

    Get-ADComputer -Filter {OperatingSystem -NotLike "*server*"} -Property * | Format-Table Name,OperatingSystem,OperatingSystemServicePack -Wrap -Auto

    And blammo:

    image

    Family filtering

    By now these filters should be making more sense and PowerShell is looking less scary. Let’s say I want to filter by the “family” of operating system. This can be useful when trying to identify computers that started having a special capability in one OS release and all subsequent releases, and where I don’t care about it being server or workstation. An example of that would be BitLocker – it only works on Windows Vista, Windows Server 2008, and later. I run:

    Get-ADComputer -Filter {OperatingSystemVersion -ge "6"} -Property * | Format-Table Name,OperatingSystem,OperatingSystemVersion -Wrap -Auto

    See the change? I am now filtering on operating system version, to be equal to or greater than 6. This means that any computers that have a kernel version of 6 (Vista and 2008) or higher will be returned:

    image

    If I just wanted my Windows Server 2008 R2 and Windows 7 family of computers, I can change my filter slightly:

    Get-ADComputer -Filter {OperatingSystemVersion -ge "6.1"} -Property * | Format-Table Name,OperatingSystem,OperatingSystemVersion -Wrap -Auto

    image

    Getting it all into a file

    So what we’ve done ‘til now was just use PowerShell to send goo out to the screen and stare. In all but the smallest domains, though, this will soon get unreadable. I need a way to send all this out to a text file for easier sorting, filtering, and analysis.

    This is where Export-CSV comes in. With the chaining of an additional pipeline I can find all the computers, select the attributes I find valuable for them, then send them into a comma-separated text file that is even able to read the weirdo UTF-8 trademark characters that lawyers sometimes make us put in AD.

    Hey, what do you call a million lawyers at the bottom of the ocean? A good start! Why don’t sharks eat lawyers? Professional courtesy! What do have when a lawyer is buried up to his neck in sand? Not enough sand! Haw haw… anyway:

    Get-ADComputer -Filter * -Property * | Select-Object Name,OperatingSystem,OperatingSystemServicePack,OperatingSystemVersion | Export-CSV AllWindows.csv -NoTypeInformation -Encoding UTF8

    image

    Then I just crack open the AllWindows.CSV file in Excel and:

    image

    What about the whole forest?

    You may be tempted to take some of the commands above and tack on the necessary arguments to search the entire forest. This means adding:

    -searchbase “” –server <domain FQDN>:3268

    That way you wouldn’t have to connect to a DC in every domain for the info – instead you’d just ask a single GC. Unfortunately, this won’t work; none of the operating system attributes are replicated by global catalog servers. Oh well, that’s not PowerShell’s fault. All the data must be pulled from domains individually, but that can be automated – I leave that to you as a learning exercise.

    Conclusion

    The point I made above about support lifecycle is no joke: 2010 is a very important year for a lot of Windows products’ support:

    Hopefully these simple PowerShell commands make hunting down computers a bit easier for you.

    Until next time.

    - Ned “bird dog” Pyle

  • Certificate Enrollment Web Services

    Hey everyone, Rob here again. With the release of Windows Server 2008 R2 and Windows 7 we have added new methods of enrolling for certificates: Certificate Enrollment Policy (CEP) and Certificate Enrollment Service (CES). CEP is a web service that enables users and computers to obtain certificate enrollment policy information. This information includes what types of certificates can be requested and which CAs can issue them. CES is another web service that allows users and computers to perform certificate enrollment by using the HTTPS protocol. Together with the CEP web service, CES enables policy-based certificate enrollment when the client computer is not a member of a domain or when a domain member is not connected to the domain. CEP/CES also enables cross-forest policy-based certificate enrollment for Windows 7 or Windows Server 2008 R2 clients.

    Certificate enrollment without CEP / CES

    Prior to Windows 7 and Windows Server 2008 R2 the client requesting a certificate requires network access to a domain controller and the Certification Authority (CA).

    Here is a high level description of the process that is used:

    image

    Figure 1 - legacy certificate enrollment

    Step 1. LDAP queries to a domain controller for a list of templates and enterprise CA’s.

    The client computer does several LDAP queries to a local domain controller to get the following:

    • Queries for a list of pKICertificateTemplate objects (Certificate Templates) within the forest.
    • Queries for a list of pKIEnrollmentService objects (Enterprise CA’s) within the forest.
    • Queries for a list of msPKI-Enterprise-Oid objects within the forest.

    Once all of objects are returned to the client, it determines what Enterprise CA’s are available, and what certificate templates can be issued by each one of them. The client then determines the certificate templates for which it has permissions to enroll or autoenroll. If you are enrolling for certificates via the certificates snap-in it will display this list of available templates to the user.

    Step 2. DCOM connection an Enterprise Certification Authority.

    Once the client selects the certificate template for which to enroll, a DCOM connection is made to the CA. DCOM connects to the CertSrv Request DCOM interface to enroll for the certificate. The certificate is then handed back to the client.

    You may be thinking at this point “how does it work with the Web Enrollment pages?”

    Certificate web enrollment behaves in nearly the same way. The main difference is that in Figure 1 the web server running the CertSrv web pages would replace the Client. The actual Client communicates with the web enrollment pages over HTTP, so the web enrollment pages are acting as a proxy, querying Active Directory for a list of templates and converting the client’s HTTP based certificate request into a DCOM-based request that can be sent along to the CA.

    As you can see the client has to have direct network connectivity to a domain controller and the Certification Authority to be able to enroll for certificates. With this as a requirement here are a few examples of where enrollment would fail:

    • Internet based clients that need to enroll for a certificate or renew a certificate.
    • Computers in a DMZ network. Typically computers in a DMZ do not have access to internal corporate resources like domain controllers and CA’s because either they are in a workgroup or they belong to a DMZ forest with a one way trust in place.
    • Non-domain joined workstations. They are unable to authenticate to a DC and perform the initial LDAP queries, and thus will never make it to step 2 - the RPC / DCOM call.

    Certificate enrollment with CEP / CES

    We listened to feedback from customers about the above limitations of enrolling for certificates. Our answer was to create two new web services to proxy the enrollment requests. This allows for CA isolation and removes the requirement that the client be able to contact a domain controller or CA directly.

    These new roles are only available on Windows Server 2008 R2 and the only clients that are capable of requesting certificates via CEP and CES is Windows 7 and Windows Server 2008 R2. However the roles can be used with Windows Server 2003, 2008, and 2008R2 Certification Authorities (CA).

    image

    Figure 2 - CEP / CES certificate enrollment

    NOTE: The CEP and CES web services can be installed on the same server or, as Figure 2 shows, installed on two separate servers.

    Step 1. Client connects to the CEP web service over HTTPS.

    The Windows 7 / Windows Server 2008R2 computer is configured to enroll for certificates against a CEP server. When a CEP server is configured in the environment the client will connect to the CEP server via port 443 (HTTPS), and connect to the CEP web service.

    Administrators can configure via local / group policy what CEP server to use at the following locations:

    Computer Configuration\Policies\Windows Settings\Security Policy\Public Key Policies\Certificate Services Client – Certificate Enrollment Policy

    User Configuration\Policies\Windows Settings\ Security Policy\Public Key Policies\Certificate Services Client – Certificate Enrollment Policy

    Step 2. – CEP web service queries LDAP.

    The CEP service will send an LDAP query to a domain controller to get the following:

    • Queries for a list of pKICertificateTemplate objects (Certificate Templates) within the forest.
    • Queries for a list of pKIEnrollmentService objects (Enterprise CA’s) within the forest.
    • Queries for a list of msPKI-Enterprise-Oid objects within the forest.

    Once all the objects are collected and sent back to the client computer it determines the types of certificate for which it can enroll and which enterprise CAs can issue those certificates. There is a new attribute on the CA’s pKIEnrollmentService object that tells the client computer what the URI’s are for the CES servers in the environment. The attribute name is msPKI-Enrollment-Servers. The attribute is a multi-valued string so there can be multiple URI’s defined if you need to support different authentication methods. More on that later.

    Step 3. Client connects to the CES web service over HTTPS.

    The client then connects to the CES web service that answers for the Certification Authority that is configured to issue the certificate. The actual CES URI is defined in the msPKI-Enrollment-Servers attribute on the pKIEnrollmentService object for that CA.

    Step 4. CES web service impersonates the client security context to request a certificate via DCOM, and then hands the certificate back to the client.

    Here are some common questions and answers around CEP / CES:

    1. If I have Windows 7 or Windows Server 2008R2 are either CEP and CES required for certificate requests?

    If the Windows 7 / 2008R2 computer exists in the same Active Directory forest as the CA, then no. If you do not install the new roles Windows 7/2008R2 can still request certificates in the way that legacy clients do (Figure 1). Just like those legacy clients, however, Windows 7/ 2008R2 clients will need network connectivity to a domain controller and the CA.

    2. When would CEP / CES be a good solution for my environment?

    CEP / CES should be used in the environment when you require any of the following:

    • Windows 7/2008R2 internet based clients need to be able to enroll for certificates.
    • Windows 7 / 2008R2 based clients in another forest need to enroll for certificates against a 2008R2 CA in a separate forest.
    • There is a requirement that client computers should not be able to access the CA directly over the network, or there is a Firewall between the CA and client computer and your clients are Windows7 /2008R2.

    3. Where can the CEP / CES roles be installed?

    • The roles can be installed on the CA, but that would defeats the purpose since the client will still need network connectivity to the CA.
    • The roles can be installed on a domain member. The domain member could be on the internal network, or possibly in a DMZ. Please keep in mind that the CES role will require Kerberos delegation to be configured because it impersonates the user to the CA DCOM interface.
    • The roles can be installed on the same computer or on separate computers. Please keep in mind that the CES role will require Kerberos delegation to be configured because it impersonates the user to the CA DCOM interface.
    • Multiple instances of the CES web service can be installed on the same server. This allows you to increase the availability of the web service in environments with a large number of clients.

    I hope that you have been able to learn a little more about these two new roles available on Windows Server 2008R2, and how to determine if you need to install and configure them. If you want more detailed information on CEP and CES you can review the Certificate Enrollment Web Services whitepaper.

    Rob “minty” Greene

  • USMT, OST, and PST

    Ned here again. Do you remember time before email in the office? It’s been a while now. Email isn’t an option these days any more than a phone. And since Exchange has reached around 65% market share, odds are decent that you’re using some version of Outlook. If you’re planning on migrating to Windows 7 with USMT 4.0, you likely have two questions:

    1. How can I get Outlook OST’s to migrate?
    2. How can I get Outlook PST’s to migrate?

    Updated April 16, 2012 to be more complete 

    OST migration

    I am just going to rip the bandage off here, so grit your teeth: you cannot migrate Outlook OST files with USMT. USMT makes no special allowances for OST files because the Outlook team does not support those files moving between computers; they were never designed for portability and Outlook’s developers don’t want anyone copying them around. The only supported way to get an OST file is for Outlook to create it.

    If you create a custom Include XML file that copies over OST files then Outlook may state the OST is invalid or that it cannot be opened. Please don’t call us asking for workarounds: this is expected behavior and there is absolutely nothing that the USMT or Outlook support teams can do. The OST may appear to work also; you got lucky for now, but there may be buried badness in that file that rises up someday like a mail-enabled Dracula – no one knows.

    And it's not just me saying this: http://technet.microsoft.com/en-us/library/cc179110.aspx#BKMK_MigrationConsiderations

    PST migration

    By default, USMT 4.0 migrates PST files that are linked to a user’s Outlook profile. This is done through internal USMT functions named SetPstPathInMapiStruct and UpdateMvBinaryMapiStruct which is called from within migapp.xml.

    What this means is that PST files which are simply stored on the drive but not actually connected to Outlook might not migrate when using /i:migapp.xml and /i:migdocs.xml.  For example, if a user has a PST file store in their c:\users\someuser\appdata\local\microsoft\outlook folder: USMT doesn't just copy that folder's contents willy-nilly; after all, those are mostly local per-computer settings and data, and just blasting them onto another computer could cause problems.

    It’s very possible a user is keeping those around for archive purposes, though. Migdocs.xml will gather customer folder contents from the roots of drives with little though to their contents, but you may need some XML to target PSTs appropriately and surgically. Consider this scenario, where the archive.pst and nov2009on.pst are not attached to Outlook, and are just loose files on the computer:

    image

    If you just wanted the c:\pst copy, you get it for free as long as you use migdocs.xml. In order to migrate  all of those PST files though, you’ll need to use a custom XML override. Here is a sample that will gather all PST files from all fixed drives, regardless of their location:

    <?xml version="1.0" encoding="utf-8" ?>
    <migration urlid="http://www.microsoft.com/migration/1.0/migxmlext/pst">
      <component type="Documents" context="UserAndSystem">
        <displayName>All PST migrated from all fixed drives, regardless of location</displayName>
        <role role="Data">
          <rules>
            <include>
              <objectSet>
                <script>MigXmlHelper.GenerateDrivePatterns ("* [*.pst]", "Fixed")</script>
              </objectSet>
            </include>
          </rules>
        </role>
      </component>
    </migration>

    Paste that into Notepad and save as PST.XML into your USMT folder. When you add that to your scanstate and loadstate command-lines with /i:pst.xml then all fixed drive PST files will be migrated. You could also paste it into Visual Studio 2008 Express, nudge nudge.

    Here's another sample XML, which has the added bonus of not accidentally copying PST files that a user connected to Outlook through a UNC path (which is unsupported except for one small scenario, by the way - http://support.microsoft.com/kb/297019):

    <?xml version="1.0" encoding="utf-8" ?>
    <migration urlid="http://www.microsoft.com/migration/1.0/migxmlext/pst">
      <!-- This component migrates .PSTs except for ones on the network-->
      <component type="Documents" context="UserAndSystem">
        <displayName>All PST files migrated from all fixed, no PST’s migrated from network, regardless of context or outlook registry catalog settings</displayName>
        <role role="Data">
          <rules>
            <unconditionalExclude>
              <objectSet>
                <script>MigXmlHelper.GenerateDrivePatterns ("* [*.pst]", "Remote")</script>
              </objectSet>
            </unconditionalExclude>
            <unconditionalExclude>
              <objectSet>
                <pattern type="File">\\* [*.pst]</pattern>
              </objectSet>
            </unconditionalExclude>
            <include>
              <objectSet>
                <script>MigXmlHelper.GenerateDrivePatterns ("* [*.pst]", "Fixed")</script>
              </objectSet>
            </include>
          </rules>
        </role>
      </component>
    </migration>

    Side note: the migXmlHelper.GenerateDrivePatterns option allows me to avoid hard coding paths, drive letters, or non-fixed storage. It’s something you should get comfortable with as it’s extremely useful in environments with non-standard images and users roaming freely with USB thumb drives. Since you’re going to Windows 7, don’t forget about “BitLocker to go” for those scenarios!

    Another side note: Make sure you are using latest USMT 4.0 so that you capture Outlook 2010 settings correctly! http://support.microsoft.com/kb/2023591

     

    Until next time.

    - Ned “ends in t” Pyle

  • Get Shiny with USMT: Turning the Aero Theme on During XP to Windows 7 Migration

    Ned here again. Today’s post is quick and dirty… and glossy! One thing users notice immediately about Windows 7 is the default Aero theme. Even if you‘re not a fan of eye candy, your users are – and in the end they run the show in most environments. Especially when their business card says “VP” on it.

    Consider the following scenario:

    1. Windows XP users have the default display theme of "Windows XP".
    2. You use USMT 4.0 to migrate from Windows XP to Windows 7.
    3. After migrating to Windows 7 with USMT – using online or offline migrations -- any migrated users that logon have the old and busted "Windows Classic" theme:

    image

    … instead of the new hotness:

    image

    image

    image

    Why this happens

    By default, USMT tries to copy a user’s theme to the target computer. But XP did not have the Aero theme. Worse, XP’s default theme was “Windows XP” which does not exist in Windows 7. The closest possible match is “Windows Classic” so that’s what USMT uses.

    Whether or not this default behavior is desirable is open for polite debate in our comments area. :-)

    How to get Aero by blocking theme migration

    To override the default theme migration, use the following steps:

    1. On your reference XP source computer with USMT 4.0 installed, run:

    SCANSTATE.EXE /genconfig:config.xml /o

    This will create a new customization config.xml file in the scanstate working directory.

    2. Open the config.xml in notepad.exe.
    3. Locate the following line in the config.xml file (this is wrapped for readability):

    <component displayname="Microsoft-Windows-themeui-DL" migrate="yes" ID="http://www.microsoft.com/migration/1.0/migxmlext/cmi/microsoft-windows-themeui-dl/microsoft-windows-themeui-dl/settings"/>

    4. Change the "yes" to "no", so that the line now is:

    <component displayname="Microsoft-Windows-themeui-DL" migrate="no" ID="http://www.microsoft.com/migration/1.0/migxmlext/cmi/microsoft-windows-themeui-dl/microsoft-windows-themeui-dl/settings"/>

    5. Save the config.xml file.
    6. Run your usual scanstate and loadstate syntax, making sure to also include your new config file on the scanstate:

    /config:config.xml

    Disabling this component prevents the following registry keys from migrating (under the covers):

    HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Themes\LastTheme
    HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Themes

    And that’s it – now you have Aero.

    image

    Until next time.

    - Ned “London Dungeon” Pyle

  • Strict Replication Consistency - Myth versus Reality

    Hi, David here again. Having worked numerous lingering object cases, I find a common misunderstanding about Windows Server 2003 (or later) and its ability to automatically enforce Strict Replication Consistency. Strict Replication Consistency is a registry value that prevents destination domain controllers (DC) from replicating in lingering objects. Lingering objects are objects that have been deleted on one DC but replication failures prevent a partner DC learning of the deletion. The result is those deleted objects remain “live” on the replication partners. If the replication failure persists for longer than tombstone lifetime but is later corrected, the DC that failed to inbound replicate the deletions will continue to have “live”/lingering objects in its copy of the AD database. When one or more attributes are modified on these “live” objects, that object must replicate outbound. DCs that don’t have Strict Replication Consistency enforced will replicate in these formerly deleted objects, re-animating them.

    Note: for the sake of readability, when I say “Windows Server 2003” in the document, I mean Windows Server 2003 or later. Windows Server 2008 and Windows Server 2008 R2 behave the same in this respect.

    A Windows Server 2003 server can automatically configure Strict Replication Consistency during the domain controller promotion process certain conditions have to be true. There are some myths and badly worded documents out there that imply Windows Server 2003 DC’s always configure themselves for strict replication , so this blog post aims to set the record straight.

    Forests originally created on Windows 2000 Server - but later upgraded to Windows Server 2003 require an additional step to automatically enable strict replication consistency on newly promoted domain controllers. While Windows Server 2003 DCs tend to quarantine themselves if they have not replicated for greater than Tombstone Lifetime, failure to implement this step will leave all newly promoted Windows Server 2003 DCs configured for Loose Replication Consistency; leaving them at risk of re-animating lingering objects from Windows 2000 replica DCs or Windows Server 2003 DCs that have had Allow Replication With Divergent and Corrupt Partner set but have not been cleaned of lingering objects first.

    If you want new domain controllers added to the forest to have strict replication consistency automatically enabled, you can import the Operational GUID cited below to the Configuration directory partition using Ldifde.exe. The presence of this object in the Configuration partition causes dcpromo.exe to enable Strict Replication Consistency on any Windows Server 2003 domain controller that is promoted into the forest.

    1. Start notepad.exe and copy in the sample LDF text below. Edit both lines containing DC=<domain>,DC=<com> to match your forest root domain.

    Example: If your forest root domain were contoso.com the DN would be DC=contoso,DC=com

    dn: CN=94fdebc6-8eeb-4640-80de-ec52b9ca17fa,CN=Operations,CN=ForestUpdates,CN=Configuration,DC=<domain>,DC=<com>
    changetype: add
    objectClass: container
    showInAdvancedViewOnly: TRUE
    name: 94fdebc6-8eeb-4640-80de-ec52b9ca17fa
    objectCategory: CN=Container,CN=Schema,CN=Configuration,DC=<domain>,DC=<com>

    2. Once the DN has been modified correctly save the file as StrictRepl.ldf.

    3. Open a command prompt with Enterprise Administrator credentials and issue this command:

          Ldifde.exe -i -f StrictRepl.ldf

    It’s important to note that just adding this object to the CN=Operations container will not cause existing Windows Server 2003 DCs to configure themselves for Strict Replication Consistency. For existing DCs you must add the Strict Replication Consistency registry value and set it to 1 before they will enforce strict replication. The good news is this can be done remotely using repadmin.exe from a command prompt opened with Enterprise Admin credentials. The command to run is:

    repadmin.exe /regkey <dcname> +strict

    If you want to make certain this is configured on all DCs in the forest you can pass a wildcard to repadmin.exe like this,:

    repadmin.exe /regkey * +strict

    If you are running this against all DCs in the forest you should pipe this out to a text file and verify all DCs have been contacted and the value has been set correctly.

    Warning: Before you implement this change forest-wide it is important to understand that all replication between the source DC and the destination DC will stop for any partition that has a lingering object in it. Replication will only be restored once the lingering object is removed. This could cause forest-wide authentication issues until replication is restored.

    For more information about this operational GUID see Technet.

    Hopefully this will clear up the common misconception that all Windows Server 2003 domain controllers will always enforce strict replication consistency and will lead to closer examination of current configurations before lingering objects start causing issues.

    David “buster” Everett

  • USMT 4 and WinPE: Common Issues

    Ned here again. As odd as it sounds, when you call Microsoft for USMT 4.0 support, you will be sent to the Directory Services team. This is because we support user profiles loading and unloading, as that is a core piece of interactive logons. It’s a tangled web.

    Believe it or not, I support more than just DFSR. Really!

    DS engineers at Microsoft also have to know about other dependencies USMT can take, such as the Windows Preinstallation Environment (WinPE). You can run USMT 4.0 within a WinPE session so that your scanning process takes place with the source OS completely offline. It’s a pretty slick new feature and we have some straightforward steps in the “Offline Migration with USMT 4.0” guide.

    As with any new feature, some common questions start cropping up. Because Windows 7 is being deployed at insanely high rates USMT is becoming a hot topic. Below are some errors people frequently see with USMT running offline migrations in WinPE.

    Error: “The subsystem needed to support the image type is not present”

    This error is caused by running a 32-bit version of USMT within a 64-bit WinPE image. There are two rather obvious solutions to this:

    1. Use an x86 version of WinPE when running an x86 version of USMT.

    2. Use an x64 version of WinPE when running an X64 version of USMT.

    It’s fine to use x64 WinPE and USMT to run a scanstate on an x86 offline OS. Always remember to specify the environment variable for the offline OS architecture being scanned. This variable is:

    MIG_OFFLINE_PLATFORM_ARCH=<32 or 64>

    For example, if the offline OS being scanned was 32-bit Windows XP, you would specify the following in the WinPE command prompt that is about to run scanstate:

    SET MIG_OFFLINE_PLATFORM_ARCH=32

    Return Code 29: “A minimum of 250 MB of free space is required for temporary files”

    You see this error even if you have forty bajillion GB free. When you examine your scanstate log you will see something to the order of:

    2009-12-01 15:04:25, Info  [0x000000] USMT Started at 2009/12/01:15:04:25.187
    2009-12-01 15:04:25, Info  [0x000000] Command line: scanstate z:\store /offline:c:\usmt\offline.xml /i:migapp.xml /i:miguser.xml /o /config:config.xml /v:5 /encrypt /key:****
    2009-12-01 15:04:25, Status  [0x000000] Activity: 'MIGACTIVITY_COMMAND_LINE_PROCESSING'
    2009-12-01 15:04:25, Status  [0x000000] Activity: 'MIGACTIVITY_AUTO_GENERATE_OFFLINE_VERSION'
    Info  [0x000000] Drive X:\ has 30 MB free; a minimum of 250 MB is required[gle=0x000000cb]
    Info  [0x000000] Failed.[gle=0x000000cb]
    Info  [0x000000]   A minimum of 250 MB of free space is required for temporary files[gle=0x000000cb]
    Info  [0x000000] USMT Completed at 2009/12/01:15:04:26.078[gle=0x000000cb]
    Info  [0x000000] Entering MigShutdown method
    Info  [0x000000] Leaving MigShutdown method

    This error will appear even if you are running scanstate.exe on the source computer’s local hard drive. The problem is that you’re missing a required environment variable that must be set in the WinPE CMD prompt in order to execute an offline migration. Use the following environment variable:

    USMT_WORKING_DIR=<some non-WinPE path>

    For example, to specify the physical computer's C:\temp path to act as the temporary storage space, run:

    SET USMT_WORKING_DIR=C:\temp

    Naturally, you would also see this error if not using an Offline migration and the source computer hard drive really was out of disk space, or was exceeding disk quotas.

    Return Code 36: “Use /offline to run gather on this platform”

    This error is especially weird because the /offline switch is only available within scanstate.exe, not loadstate.exe. If you try to follow the error’s suggestion to use /offline in the loadstate, you get a new error:

    An error occurred processing the command line.
    loadstate ##ERROR## --> /offline
    Undefined or incomplete command line option
     

    The real problem is you’re running loadstate within the WinPE session, which is not possible. Loadstate must always be run within a running target OS, never in WinPE.

    Scanstate /UI /UE rules with user names are ignored

    Consider the following command being run in an offline migration:

    Scanstate.exe c:\store /offline:offline.xml /hardlink /nocompress /efs:hardlink /ui:contoso\nedpyle /ue:*\* /i:migApp.xml /i:MigDocs.xml /v:5

    You would expect that /ui:contoso\nedpyle and /ue:*\* would cause the Contoso domain user NedPyle to migrate, and all other user profiles to not migrate. When run, no user profiles are migrated at all. The same exact migration run on the same computer in an online migration works perfectly and only NedPyle is migrated.

    This behavior is both expected and unavoidable. The WinPE computer has no knowledge of the domain and no way to perform SID translation. Specifying /ui:domain\username or /ui:*\username is therefore an invalid command. If only the scanstate tool would tell you this!

    To work around the limitation, you must use a SID in your /ui or /ue arguments. All profiles are defined in the registry as SIDs, and without the need to translate a user name USMT will work fine. So this would give the desired result:

    1. The SID of my contoso\nedpyle user is S-1-5-21-1405795100-2172710363-725018148-1112.

    2. My command-line to include this user but exclude all other users in an offline migration would be:

    Scanstate.exe :\store /offline:offline.xml /hardlink /nocompress /efs:hardlink /ui:S-1-5-21-1405795100-2172710363-725018148-1112 /ue:*\* /i:migApp.xml /i:MigDocs.xml /v:5

    To figure out SIDs I recommend the old reliable psgetsid.exe tool.

    Wrap up

    Want to know what all these cryptic error codes mean? Make sure you book mark the USMT 4.0 return codes matrix.

    Hopefully the info above saves you a headache someday in your quest to deploy Windows 7, the greatest OS we have ever made. :-)

    - Ned “two trick pony” Pyle

  • USMT Custom XML the Free and Easy Way

    Ned here again. XML is used to configure all aspects of USMT 4.0 migration and is especially important when customizing. Unfortunately most IT staffers are not familiar with XML – why should they be? It’s barely used within Windows and is mainly an applications-specific file store. Maybe you noticed that group policy ADMX files are XML – did you care, since you were using the GP editor to make changes?

    Unfortunately, there’s no USMT XML editor. What’s more, XML follows programming conventions– tags must be closed; nesting must be complete; rules are case-sensitive. And any mistake in the XML will cause the migration to fail or skip crucial steps.

    XML, like any programming file format, has rules. This means that there are tools that can examine that file and see if the rules are being broken – programmers are not super human. One such tool is Visual Studio 2008 Express. It has an excellent suite of XML authoring and validation options – and it’s free :).

    Let’s come up with a scenario that requires custom XML, create our file, and then validate it.

    The Scenario

    End users love wallpaper. I mean they love it, in a manner usually reserved for grandkids, puppies, and their first car. So when you find that USMT doesn’t migrate wallpaper from XP to Windows 7, you know they won’t be pleased.

    After some Internet exploration, you find all the wallpaper settings from XP and the default wallpaper locations are made up of a few registry settings and folders. You have examined our various references, including:

    Now you want to bang out a custom XML file. Most folks are not familiar with VS2008 so I’ll go through this step-by step.

    Authoring

    1. Download and install Visual Studio 2008 Express Edition (.Net), then start it up.

    image

    2. Choose File, New Project, and select “Empty Project”. Call it USMT.

    image

    3. In your new project, select Add then New Item.

    image

    Select XML file and give it a name like “wallpaper.xml”, then add it.

    image

    Click File menu, then Save All. Save your project and files somewhere.

    4. Paste in the following sample that migrates wallpaper and background settings. It’s wrapped for readability:

    <migration urlid="http://www.microsoft.com/migration/externalUserDocs">

      <!-- This component migrates wallpaper settings -->
      <component type="System" context="User">
        <displayName>Wallpapers</displayName>
        <role role="Settings">
          <rules>
            <include>
              <objectSet>
                <pattern type="Registry">HKCU\Control Panel\Desktop [Pattern]</pattern>
                <pattern type="Registry">HKCU\Control Panel\Desktop [PatternUpgrade]</pattern>
                <pattern type="Registry">HKCU\Control Panel\Desktop [TileWallpaper]</pattern>
                <pattern type="Registry">HKCU\Control Panel\Desktop [WallPaper]</pattern>
                <pattern type="Registry">HKCU\Control Panel\Desktop [WallpaperStyle]</pattern>
                <pattern type="Registry">HKCU\Software\Microsoft\Windows\CurrentVersion\Themes [SetupVersion]</pattern>
                <pattern type="Registry">HKCU\Software\Microsoft\Internet Explorer\Desktop\General [BackupWallpaper]</pattern>
                <pattern type="Registry">HKCU\Software\Microsoft\Internet Explorer\Desktop\General [TileWallpaper]</pattern>
                <pattern type="Registry">HKCU\Software\Microsoft\Internet Explorer\Desktop\General [Wallpaper]</pattern>
                <pattern type="Registry">HKCU\Software\Microsoft\Internet Explorer\Desktop\General [WallpaperFileTime]</pattern>
                <pattern type="Registry">HKCU\Software\Microsoft\Internet Explorer\Desktop\General [WallpaperLocalFileTime]</pattern>
                <pattern type="Registry">HKCU\Software\Microsoft\Internet Explorer\Desktop\General [WallpaperStyle]</pattern>
                <content filter="MigXmlHelper.ExtractSingleFile(NULL, NULL)">
                  <objectSet>
                    <pattern type="Registry">HKCU\Control Panel\Desktop [WallPaper]</pattern>
                    <pattern type="Registry">HKCU\Software\Microsoft\Internet Explorer\Desktop\General [BackupWallpaper]</pattern>
                    <pattern type="Registry">HKCU\Software\Microsoft\Internet Explorer\Desktop\General [Wallpaper]</pattern>
                  </objectSet>
                </content>
              </objectSet>
            </include>
          </rules>
        </role>
      </component>

      <!-- This component migrates wallpaper files -->
      <component type="Documents" context="System">
        <displayName>Move JPG and BMP</displayName>
        <role role="Data">
          <rules>
            <include>
              <objectSet>
                <pattern type="File"> %windir% [*.bmp]</pattern>
                <pattern type="File"> %windir%\web\wallpaper [*.jpg]</pattern>
                <pattern type="File"> %windir%\web\wallpaper [*.bmp]</pattern>
              </objectSet>
            </include>
          </rules>
        </role>
      </component>
    </migration>

    It will now look much better, which is one important advantage of using VS 2008 to work with USMT XML: many mistakes can be avoided just by the visual cues of colored letters and proper formatting.

    image

    5. Click the XML menu and choose Schemas… (note that if you don’t select an XML document tab, the menu won’t appear).

    image

    Add in the MIGXML.XSD from your USMT folder. This file defines the schema of USMT XML and will allow Visual Studio to point out further errors.

    image

    6. Open the View menu and select Error List:

    image

    Now you will see all syntax errors in your XML file in real time, both with an underline squiggle, a la MS Word, as well as in the error window below. The sample I gave to paste in is (of course!) perfection, so you have no errors yet.

    Validation

    Let’s create some intentional mistakes to show how Visual Studio can help:

    • Change <component type="System" context="User"> to <component type="System" context="user"> . Note how the lower-case “user” is now underlined and the error list shows “The ‘context’ attribute is invalid.” XML tags are case-sensitive and a lower case user no longer matches the schema defined name of “User”, so an error is raised. Change it back.
    • Change <component type="System" context="User"> to <component type="System" context="SystemAndUser"> . Since the real value must be “UserAndSystem”, the context attribute is invalid. Change it back.
    • Change any <include> tag to <include . The next line will show an underline and “Expecting ‘>’” will appear with a line number. Change it back.
    • Delete any </include> tag. Now you will see an error “Tag was not closed” and “Expecting end tag </include>”. Change it back.

    See how much easier this is than using Notepad? :) The right tool for the job makes all the difference and didn’t cost you a dime. Take a few more of the examples included in Exclude Files and Settings and Include Files and Settings, paste them into some new XML files in your project, and start tinkering. Repetition breeds familiarity. Everything above also works with Visual Studio 2008 Professional of course.

    Save your Wallpaper.XML to the USMT folder and fire up scanstate in a test environment with /i:wallpaper.xml . Your users can now have their old wallpapers migrated free of charge. I’m sure they will thank you. Riiiiiiight.

    Update October 29 2010: If using Visual Studio Express 2010, make sure you click "Tools --> Settings --> Expert Settings" in order to see the XML menu and other feateures described above. We buried a lot in this version...

    Until next time.

    Ned “bliss.bmp” Pyle

  • USMT and OfflineWinOld: Taking XP to Windows 7 in a Hurry

    Ned here again. To paraphrase Mark Twain, reports of XP’s life have been greatly exaggerated. Mainstream support ended in April 2009 and Service Pack 2 support ends on July 13, 2010. With the release - and crazy popularity – of Windows 7, companies are exploring various options to move on from XP.

    But there’s a catch: XP cannot be in-place upgraded to Windows 7. And even if you could upgrade it, most IT admins are wary of in-place upgrades, even if only by reputation. So what do you do? You use USMT 4.0’s new offline migration feature /OfflineWinOld.

    The high level process

    1. Examine your current XP image(s) and inventory what applications are installed.

    2. Make a decision on how you plan to deploy:

    a. Use a Windows 7 DVD and install applications by hand (less than 100 computers – keep reading this post)

    b. Create a new Windows 7 image that contains all your applications and is sysprepped (more than 100 computers – go read all about large deployments and maybe forget about this post).

    c. MDT 2010 or SCCM 2007 (go read Mike and Dan’s blog and definitely forget about this post!)

    3. Install WAIK somewhere and drop the USMT directory onto a network share or thumb drive.

    4. Install Windows 7 on an XP computer using a “Custom (Advanced)” installation and not formatting the hard drive. Add the applications and join the computer back to the domain.

    5. Run USMT scanstate with an offlinewinold hardlink migration.

    6. Run USMT loadstate against that hard link store.

    7. Repeat steps 4 to 6 until done.

    8. Give self a raise - just like Congress.

    The step-by-step process

    Critical note: Everything I describe below is happening in your test environment first. Really.

    Another critical note: There are certain OS customizations that may have made that cannot be migrated offline due to product limitations. Make sure that nothing on this list really bothers you before proceeding:

    • DCOM settings
    • Internet Options
    • Regional Language settings
    • Media Player settings
    • Offline Files settings
    • RAS settings
    • Display settings
    • Speech recognition settings
    • Handwriting recognition settings
    • Phone and modem settings
    • Media Center settings
    • Windows Search settings
    • Mapped network printers <-- probably the one you think is important, but isn’t necessarily

    In addition, offline migrations can’t handle situations where someone has moved the Program Files or Documents and Settings folders to a different drive than the Windows folder. For businesses, many of these settings are unimportant (Media Center!) or are controlled through Group Policy anyways (Offline Files) so most customers don’t get in a twist over them. But if any of these are a show stopper for you, go with a normal online migration. It will be slower but ultimately more capable.

    Good to go? On with the show.

    1. Begin by inventorying your XP computers. You need to know what applications are installed on them so that they can be reinstalled. Check with your hardware vendors to make sure your computers don’t have any known issues with running Windows 7, or if they have updated firmware if they do. By now any credible vendor can at least tell you if an application is supported or not – even if the answer is “no”. If the vendor doesn’t know, it’s probably time to start looking at competitors – and I bet if you mention that, the vendor will suddenly know in a hurry! :-)

       For more Windows 7 application compatibility info, go to our portal here.

    2. Decide how to migrate and create your image. In this case, you decided to install Windows 7 by hand, add the applications (or create a simple Win7 image with the applications included), then use USMT to offline migrate.

    3. Download the Windows Automated Installation Kit and burn the ISO to a DVD. Install on an administrator computer running Win 2003 SP2, Win Vista, Win 2008, Win 7, or Win 2008 R2 somewhere in the environment. You cannot install it on XP.Inside the (default) location for WAIK you will find the USMT data:

       %programfiles%\Windows AIK\Tools\Usmt\x86

       %programfiles%\Windows AIK\Tools\Usmt\amd64

    Copy those folders to a network share, to a USB thumb drive, etc. You will be using those files a lot for the next few days. If you only have plans to use x86 and not deploy 64-bit Windows 7 then don’t bother with the amd64 folder.

    4. Note the name of your XP computer, and then install Windows 7 over the top of your existing Windows XP installation. In the example below you have booted from the DVD. Click “Install Now”.

    image

    You are prompted to choose “Upgrade” or “Custom (Advanced)”. If you choose upgrade, you get smacked down, so make sure you choose custom:

    image

    Windows servicing then notes that you already have Windows (XP) installed.

    image

    Here’s the critically important part: do NOT delete or format the disk. This sounds obvious, but I’ve already had several people format drives and then ask why USMT didn’t migrate any data. Guess what sort of backups they had taken…

    image

    Now, the setup runs and because it is not an upgrade it installs fast. My Windows 7 installations in a hyper-v guest usually take less than 20 minutes.

    Log on and install any applications you need – at least the ones that were installed on XP for business use. Join the computer to the domain using the same name the XP computer used. This will keep the domain SID, group memberships, etc. that had belonged to this computer when it was XP.

    At this point we have a nice pristine computer that has its old name, is joined to the domain, and has its old applications installed. We’re ready to migrate.

    5. Start Windows Explorer and have a look around the drive. There is now a Windows.old folder. Expand that folder and you’ll find inside that all the user profile data exists – this includes files in My Documents, data on the Desktop, the user’s registry settings etc. The Windows.old folder also contains the system’s registry and file structure, under Windows and Program Files. Other folders that were not in the users’ profiles, Windows, and Program Files directories will still be on the drive in their original location. Everything has been preserved.

    image

    Open a CMD prompt as an Administrator.

    image

    Navigate to wherever you decided to store the USMT files. In the examples below they were copied from a network location to a local folder c:\usmt on the computer being migrated.

    Run your scanstate operation. This will examine and gather up all the data that was preserved in the Windows.old folder. This scanstate should include the following commands at a minimum:

    <store path>
    /offlinewinold:<path to your old XP Windows folder>
    /i:migapp.xml
    /i:migdocs.xml
    /hardlink
    /nocompress

    For example:

    image

    The store path will be used to store migration information about settings and what are called “hard links”. These are pointers to files that allow USMT to migrate those files without having to copy the files into the store or duplicate files that are identical. It saves a huge amount of time and space during the migration.

    Note: If you have users that are specifically added to local groups on the XP source computer you will need to use a custom /config:config.xml file to specify that users should be added into specific groups. Usually this is for users that are members of the Administrators group in XP – something which you should not continue to do!

    Here is a sample config.xml that makes all migrated users members of the Administrators group. Which again, you shouldn’t do! I cannot emphasize that enough, but I get asked about it frequently. Just remember that if you make your users administrators all the other security best practices in the world will not save them. This goes for every operating system ever made.

    <Configuration>
    <ProfileControl>
        <localGroups>
          <mappings>
             <changeGroup from="*" to="Administrators" appliesTo="MigratedUsers">
                <include>
                   <pattern>*</pattern>
                </include>
             </changeGroup>
          </mappings>
        </localGroups>
    </ProfileControl>
    </Configuration>

    Genconfig has other good reasons to be run, make sure you look into it.

    The scanstate runs, examining all the profiles and the computer configuration. All the data and settings are gathered up into migration units that are saved or hard-linked in the store.

    image

    image

    Note: If you are interested in seeing what I mean about hard links, look at this sample output using FSUTIL.EXE. Here I dump out the hard link info from a file that actually exists in the backed up Windows.old folder. The hard link allows it to simultaneously “exist” in both spots at the same time.

    image

    Note: If you run the hard link scanstate successfully, you can get an error if you try to run it again:

       scanstate.exe c:\store /o /hardlink /offlinewinold:c:\windows.old\windows /nocompress

       Failed to delete 'c:\store\usmt'. Windows error 18.

       Failed.
         An error occurred processing the command line.
         Invalid store path; check the store parameter and/or file system permissions

       Scanstate returned code: 27

    This is expected. Use the usmtutils.exe tool included with USMT to delete the hard link store, then you can run the scan again. This in no way affects the real data living in windows.old; it’s just deleting the hard links.

    6. Now you run the loadstate operation against the same store you just created – there’s no need to reboot or so anything fancy. The loadstate should include the following commands at a minimum:

    <store path>
    /i:migapp.xml
    /i:migdocs.xml
    /hardlink
    /nocompress

     

    image

    All the files, settings, profiles, application configurations, etc. are restored to their original spots on the hard drive. If any of the migrated users logon now they will find their files and settings haven’t changed – for the most part; it is a new operating system after all.

    7. Repeat for all the other computers in your test environment. Then evaluate the results, have some users try things out, and consider going for it in production, repeating steps 4-6.

    8. Chuckle at all the people that complained about the lack of an XP upgrade, secure in the knowledge that you now have a much more stable system with all the data and settings your users care about.

    And yes, we do have this all publically documented in TechNet. But it’s buried pretty well in the appendix of this article and doesn’t have my pretty pictures. :-)

    Until next time.

    - Ned “no longer a DFSR SME” Pyle