The USMT team blog

Next USMT release

tdolan — Wed, 06 Aug 2008 14:43:38 GMT

I wanted to take a moment here on the blog to let everyone know that we are continuing to invest in USMT and have aligned the next USMT release to coincide with the next release of Windows. Please be assured that all of you will be the first to know when we have more information to share with you.

Thanks

tdolan — Mon, 21 Jul 2008 21:04:27 GMT

Thanks for all of your suggestions in the comments on the post "What do you want to see here? Any ideas?" that went live on July 3rd. I am working on a post to address some of the questions in the comments regarding the config.xml file. Hopefully I will be able to get this up for you in the next week or so. In the meantime, if you have any other post suggestions, please drop us a comment.

What version of USMT can be installed on Windows Server 2003 or 2008?

tdolan — Thu, 10 Jul 2008 16:30:00 GMT

Unfortunately, there isn't a version of USMT that can be installed or executed on Windows Server 2003 or 2008. As noted on the download page and in the user documentation, USMT is only supported on client versions of Windows.

What do you want to see here? Any ideas?

tdolan — Thu, 03 Jul 2008 19:14:00 GMT

Is anyone out there? Is there anything you'd like to see a blog post about? I can’t guarantee that we will answer your question but I can promise that we will try. Drop a comment and let us know.

Increasing MigXML efficiency

tdolan — Fri, 27 Jun 2008 01:37:00 GMT

Increasing the efficiency of MigXML is all about making each individual include or exclude pattern as specific as possible and ensuring that the component operates in the proper context. For example, the following component is extremely expensive to include in a migration:

<component type="Documents" context="UserAndSystem">
<displayName _locID="All Word Documents">User Data</displayName>
<role role="Data">
    <rules>
      <include>
        <objectSet>
          <script>MigXmlHelper.GenerateDrivePatterns ("* [*.doc*]", "Fixed")</script>
        </objectSet>
      </include>
    </rules>
</role>
</component>

This component searches the entire directory structure of each fixed drive on the system for Word documents to include in the migration, in both user and system context (as covered in the post on context, this search will be executed n+1 times where n is the number of users on the machine selected for migration). How expensive is it? Well, lets try running Scanstate a few times to find out.

I added this component to the file expensive.xml and built the following command line:

ScanState.exe /i:expensive.xml /all c:\perfStore /c /o /i:miguser.xml /v:13

I then executed the above three times with the following variations:

As above, no variations
With the context of the All Word Documents component changed to be only System
Without expensive.xml on the command line

On the same Vista SP1 machine with five user accounts that I ran the examples for the context migration post on. I found the following results:

Test	ScanState Runtime (sec)
(1)	264.5
(2)	217.6
(3)	186.1

The reduction in total runtime from (1) to (3) works out to be 30%! Although I have not run this test on another machine, I can reasonably assure you that the runtime difference between each test will be even greater on a machine that has more user data than this one. Also, in the case of the machine that I ran these tests on, the exact same set of files and settings migrate in each the these three cases despite the differences in XML and overall runtime. This is because this particular machine only contains Word documents within each user's profile (eg, c:\users\tdolan\*). Since MigUser.xml migrates all data under user profiles, the addition of the All Word Documents component won't add any files to the migration. All adding it does is require ScanState to do more searching before determining that it has a complete set of files for migration.

So, what should we take away from this? Writing efficient MigXML rules can increase migration performance. This comes in the following forms:

Selecting the proper context for each component
- For the most part, only components with user specific environment variables should be user context (eg, %CSIDL_MYMUSIC% and others as seen in MigUser.xml)
- For the most part, only components that need to search the whole system should be in system context
- UserAndSystem context should only be used when you are sure that User or System context alone isn't appropriate
Reducing the overlap among components. Overlap among components (having the same files migrate from more than one component) reduces performance without impacting the migration. This happened in the above example in two ways:
- Placing the All Word Documents component in anything other than System context is a waste since doing so only results in the same search being executed multiple times
- The All Word Documents component was unnecessary on my test system since the only Word documents it found were already migrating per MigUser.xml
Making patterns as specific as possible. For example:
- <pattern type="File">C:\* [*doc]</pattern> : This is about as bad as it gets (short of using the GenerateDrivePatterns helper as above). ScanState must search the drive for ONLY .doc files and discard everything else.
- <pattern type="File">C:\Data\* [*doc]</pattern> : This is much better since we have restricted the search from above to only be under C:\Data and its subdirectories.
- <pattern type="File">C:\Data\* [*]</pattern> : From a processing perspective, this is the best. There isn't any searching required; ScanState just knows that the entire C:\Data directory must migrate.

Following these guidelines should help you write more efficient and MigXML rules that will yield better performance.

Why you need to be careful with /c

tdolan — Thu, 19 Jun 2008 10:52:45 GMT

Scanstate and Loadstate both allow the /c switch to be used to skip non-fatal errors. However, due to the impact that using it can have on a migration, great caution must be used when deploying Scanstate and Loadstate with /c. To get started, here is what the help content has to say:

When specified, ScanState will continue to run even if there are nonfatal errors. Without the /c option, ScanState exits on the first error. When you specify this option, any files or settings that cause an error and are ignored will be logged in the progress log. For example, if there is a large file that will not fit in the store, ScanState will log an error and will continue with the migration. In addition, if a file is open or in use by an application, USMT may not be able to migrate the file and will log an error. (emphasis mine) Link

When specified, LoadState will continue to run even if there are nonfatal errors. Without the /c option, LoadState exits on the first error. When you specify this option, any files or settings that cause an error and are ignored will be logged in the progress log. For example, if a file is open or if there is a large file that will not fit on the destination computer, LoadState will log an error and will continue with the migration. (emphasis mine) Link

So, what can we take away from this? Bottom line, when specifying /c on the command line you are telling Scanstate and Loadstate that it is ok for them continue even when they can't migrate any number of files or registry keys. Also, this behavior is not configurable in any way. That is to say that both executables will continue no matter the importance or quantity of the data left behind.

However, this is not to say that /c should't be used. Each time a file is skipped with /c a note is made in the log indicating so. This enables scenarios in which these files are gathered separately, if important, either manually or with a script. We have talked to folks who have done both. The important thing here is to understand exactly what /c does and ensure that it fits into the scenario in which you use USMT.

About context in MigXML

tdolan — Thu, 12 Jun 2008 12:30:00 GMT

If you have written a fair amount of migration MigXML you have probably noticed that there are a number of elements that take a context parameter. Do you understand what this parameter controls? Do you understand the impact that it can have on your migration? We will work through a couple of examples here to address these questions.

Before we get started, lets talk about some background info. Although context is a parameter in a number of elements, lets focus this discussion on the impact of the context parameter in the <component> element. This element is commonly used to encapsulate logical units of file or path based <include> and <exclude> rules. So, as an example, if I stored a bunch of files located on the root of my C drive, I would author a single include rule that would be encapsulated in a <component> tag.

However, the context parameter accepts the following values:

User
System
UserAndSystem

Which one should I use? Lets try each value and see what the result is.

To setup this experiment I made sure that my Vista SP1 machine had multiple (five) total users and I placed a single MP3 file (PowersCombine.mp3) on the root of my C drive. So, I wrote the following MigXML to migrate the PowersCombine.mp3:

<component type="Documents" context="System">
<displayName>Component to migrate mp3 file on the c drive</displayName>
<role role="Data">
    <rules>
      <include>
        <objectSet>
          <pattern type="File">C:\ [PowersCombine.mp3]</pattern>
        </objectSet>
      </include>
    </rules>
</role>
</component>
</migration>

and ran Scanstate including all users on the machine and the rule outlined above:

ScanState.exe /i:mp3.xml /o /c /all c:\RuleTest /v:13

After Scanstate finished running I opened up the log to take a look at how the migration went. After a little searching I found the part that references the processing of migration rule that I wrote:

> Processing GATHER for migration unit: <System>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
> Argument 1: 'Component to migrate mp3 file on the c drive'

Next, I modified the MigXML rule such that it would operate in User context and ran the migration again. I found the following in the Scanstate log:

> Processing GATHER for migration unit: <test2>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
>   Argument 1: 'Component to migrate mp3 file on the c drive'

> Processing GATHER for migration unit: <test1>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
>   Argument 1: 'Component to migrate mp3 file on the c drive'

> Processing GATHER for migration unit: <tdolan>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
>   Argument 1: 'Component to migrate mp3 file on the c drive'

> Processing GATHER for migration unit: <Administrator>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
>   Argument 1: 'Component to migrate mp3 file on the c drive'

> Processing GATHER for migration unit: <tjdolan>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
>   Argument 1: 'Component to migrate mp3 file on the c drive'

Finally, I modified the MigXML rule such that it would operate in UserAndSystem context and ran the migration again. I found the following in the Scanstate log:

> Processing GATHER for migration unit: <test2>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
>   Argument 1: 'Component to migrate mp3 file on the c drive'

> Processing GATHER for migration unit: <test1>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
>   Argument 1: 'Component to migrate mp3 file on the c drive'

> Processing GATHER for migration unit: <tdolan>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
>   Argument 1: 'Component to migrate mp3 file on the c drive'

> Processing GATHER for migration unit: <Administrator>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
> Argument 1: 'Component to migrate mp3 file on the c drive'

> Processing GATHER for migration unit: <tjdolan>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
> Argument 1: 'Component to migrate mp3 file on the c drive'

> Processing GATHER for migration unit: <System>\Component to migrate mp3 file on the c drive (CMXEAgent)
> Activity: 'MIGACTIVITY_MIGUNIT_GATHER'
> Argument 1: 'Component to migrate mp3 file on the c drive'

In comparing the context that the rule ran in, and the number of times the log notes the rule was processed we can understand how the context parameter impacts a migration. Let's break down the results for each log sample:

In system context the rule was only processed only once, for the system
In user context the rule was processed five times, once for each user on the system
In userandsystem context the rule was processed five times, once for each user on the system and once for the system its self

Now, you may be wondering, why do we need the rule to be processed this many times? Does the number of times the rule is processed impact the number of times the file is migrated?

To answer the easy question first, no, the number of times the rule is processed has no bearing on the number of times the file is migrated. When processing a rule multiple times results in the same files being migrated the migration engine ensures that each files only migrates once. Well, what about the case when processing the rule multiple times results in different files migrating? The answer to this question actually answers the earlier question on why we might want these rules to be processed multiple times. Lets take a look at a sample from MigUser.xml:

<component type="Documents" context="User">
    <displayName _locID="miguser.mymusic">My Music</displayName>
    <paths>
        <path type="File">%CSIDL_MYMUSIC%</path>
    </paths>
    <role role="Data">
        <rules>
            <merge script="MigXmlHelper.DestinationPriority()">
                <objectSet>
                    <pattern type="File">%CSIDL_MYMUSIC%\ [desktop.ini]</pattern>
                </objectSet>
            </merge>
        </rules>
    </role>
</component>

By operating in user context and using the %CISDL_MYMUSIC% environment variable this single section can migrate the My Music folder for every user on the machine. Pretty cool, huh? So, the context that is attached to a component should be chosen such that the component is only processed as many times that it needs to be processed. In the example we stepped through earlier in the post, the appropriate context to choose would have been System.

Why you want (and need) to run USMT from an account directly associated with the local administrators group

tdolan — Thu, 12 Jun 2008 03:18:17 GMT

UPDATE: a post regarding this topic originally went live on 5/29/2008. This was written to clarify some questions that come up in comments I received both online and offline.

Although there are a very limited number of scenarios in which USMT can be run without local administrator privileges, doing so ends up not being the most ideal situation. There are a couple of reasons why:

USMT needs local administrator privileges to migrate many (if not all) operating system settings
Without local administrator privileges USMT can only migrate the currently logged in user
A bug in Vista SP1 prevents USMT from ever running as a local user on a Vista SP1 machine (KB article here)
The LoadState and ScansState documentation recommend that both tools be run from an account with local administrator privileges

Due to all the reasons above, as the product team, we strongly recommend that USMT only be executed from a user account that has local administrator privileges (and, if necessary, from an elevated command prompt).

However, there is an additional complication that must be considered in some cases. A bug in USMT 3.x prevents it from recognizing users who have indirectly inherited administrator privileges as actually being an administrator. Although the rest of the system will consider these users to be an administrator, USMT will not. This will result in USMT either failing or completing the migration without gathering all the users, files, and settings as it was told to.

How can you tell if you will encounter this problem? Is there a work around?

You can tell if you will encounter this problem by checking the computer management counsel to verify that the user account you will be running USMT from is a direct member of the local administrators group:

Windows Key + R
Type compmgmt.msc and press enter
Double click on "Local Users and Groups"
Click on "Groups"
Double click on "Administrators"
Check to see if the user that will be executing USMT appears in the "Administrator" membership list

If the user account you plan on running USMT from is directly on this list you should be good to go. However, if it is not, you will likely encounter this bug. As noted earlier, this bug occurs whenever the user is indirectly a member of the administrators group. Once of the scenarios in which this is the case in when the user executing USMT is a member of a group that is listed as a member of the administrators group. In this case, the membership tree might be something like so:

Administrators
- User1
- User2
- GroupA
  - User3

Although users 1-3 are all administrators, user 3 won't be recognized as such by USMT due to this bug.

One work around is to temporary make the user executing USMT directly a member of the administrators group (add them to the group before executing Scanstate or Loadstate and remove them immediately afterwards). This can be accomplished with a script in numerous ways. For example, there are many cases where two net localgroup commands should do the trick. However, there are many other valid solutions.

If you are working with localized machines and choose to develop a script, keep in mind that the well known groups such as Users and Administrators will appear in the associated language (rendering workarounds that depend directly on the group name to fail).

Although we don't like it when a work around is necessary to complete a migration, there are times when it is necessary that they be utilized. Unfortunately, this is one of those times.

Why Scanstate and Loadstate require a top level directory path to a migration store

tdolan — Thu, 05 Jun 2008 16:56:00 GMT

If you have played around with USMT a bit you may have noticed that both Scanstate and Loadstate require that StorePath (the path provided on the command line to where you'd like the migration store placed) be simply a path. However, when creating a compressed migration store (the default behavior) Scanstate places the .mig migration store in a subdirectory of StorePath. For example, if run the following command:

Scanstate c:\test_store [... lots of other switches]

When complete, my directory structure will be as follows:

Directory of C:\test_store\USMT3

03:19 PM    <DIR>          .
03:19 PM    <DIR>          ..
03:20 PM         7,833,045 USMT3.MIG
               1 File(s)      7,833,045 bytes
               2 Dir(s)

More curiously, Loadstate also expects that StorePath be simply a path to a top level directory -- in fact the same path that was provided to Scanstate. So, in the above example, my Loadstate command line will need to look like:

Loadstate c:\test_store [... lots of other switches]

It can't look like:

Loadstate c:\test_store\USMT3\USMT3.MIG [... lots of other switches]

(this will produce an error regarding StorePath being invalid)

Why doesn't Scanstate or Loadstate allow you skip this forced directory hierarchy and directly handle the .mig file? The reason is simple but subtle. As discussed in a previous post, USMT allows two different kinds of migration stores. The first is the compressed migration store covered above, and the second is the uncompressed migration store. Lets come back to this question after we look at the directory structure of an uncompressed migration store. If we run the following command:

Scanstate c:\test_store_unc /nocompress [... lots of other switches]

We get something like this:

Directory of C:\test_store_unc\USMT3

03:46 PM    <DIR>          .
03:46 PM    <DIR>          ..
03:44 PM        36,357,347 catalog.mig
03:44 PM         2,156,848 MIGSTATE.DAT
03:46 PM                 0 status.~lg
               3 File(s)     38,514,195 bytes
               2 Dir(s)

By requiring that StorePath be the top level directory we can essentially abstract away the need to specify different paths to different file formats for our different migration store formats. As we can see above, regardless of whether or not I have created a .mig file or both a .dat and .mig file, all I need to know is the top level directory.

Why you want (and need) to run USMT as a local administrator

tdolan — Thu, 29 May 2008 16:33:00 GMT

UPDATE: an updated post is available here. I'm leaving this post up to preserve the post history as well as comments.

Although there are a limited number of scenarios in which USMT can be run without local administrator privileges, doing so ends up not being the most ideal situation. There are a couple of reasons why:

USMT needs local administrator privileges to migrate many (if not all) operating system settings
Without local administrator privileges USMT can only migrate the currently logged in user
A bug in Vista SP1 prevents USMT from ever running as a local user on a Vista SP1 machine (KB article here)

Have I convinced you yet that you always want to run USMT as an local administrator?

However, if you paid attention to the title of this post, you may be wondering how USMT treats domain administrators. Unfortunately, a bug in USMT 3.0.1 prevents it from running properly when it is executed from a domain administrator account that isn't also a member of the machines local administrator account.

Thankfully, there is an easy work around for this. However, before we get to the workaround lets discuss how this problem might occur.

Let's assume that your organization has an account "DomainAdmin" that is a domain administrator on your network. Most likely this user won't be a member of any of the well known groups on all of your client machines. However, due to the nature of the domain, this user will have administrator level access on any domain joined client machine. As such, it probably makes a lot of sense to run USMT from this user account (whether manually or automatically). However, this is exactly where this bug crops up. Even though USMT should recognize a domain administrator as having administrator level access to any client machine it doesn't. This will likely result in USMT failing.

As noted earlier, there is a workaround. All that we need to do is get "DomainAdmin" added to each machines local administrator account, run USMT, and then remove "DomainAdmin" from the local administrator account (if desired, this removal could certainly be optional). Thankfully, the net localgroup command can be used in scripts to accomplish just this. For example, the following syntax adds a domain user to the administrators group:

net localgroup administrators %domain%\%user% /add

With a small variation we can remove this domain user from the administrators group:

net localgroup administrators %domain%\%user% /delete

By creating a script that calls these commands before and after Scanstate or Loadstate you can easily work around this bug.