I’ve been building SMS and ConfigMgr environments for over 17 years now.  You would think that in that amount of time I would be able to do this without pulling my hair out.  Most of the time that is the case, but not always.  Today, I wanted to do something that I’ve done dozens of times before: set up the ConfigMgr 2012 application catalog and web service.  Simple enough, right?  But it wouldn’t work, “Cannot connect to application server”.

Searching through Bing led me to many of the same web pages I’ve seen in the past:  Make sure ASP.NET is installed, make sure HTTP activation and non-HTTP activation are enabled, etc.  Easy to check, easy to fix, but still “Cannot connect to application server”.  More searching ensued.

Eventually, that led me to this thread:

http://social.technet.microsoft.com/Forums/en-US/configmanagerapps/thread/267352b5-01d2-4c8e-b40b-0a7ab669b65b/

The solution was interesting (manually tweaking the connection string), but not something that should be necessary (or supported).  But it caused me to think for a minute:  What if I didn’t configure a DNS suffix on the site server when I configured the static IP on my standalone server environment, and as a result the server couldn’t resolve its own name?  Sure enough, I didn’t configure one, so I added it and rebooted.  And all my problems disappeared.

Two hours lost, new things learned.  If you are coming to TechEd in New Orleans next week, I’ll be using this site server for some of my demos.

You would expect that having Microsoft in the Seattle area would result in having a plethora of Microsoft-related users groups around.  But over the years, I’ve found that these are somewhat hard to find.  So here’s my compilation:

• Seattle IT Pro User's Group.  This group talks about all sorts of IT pro-related topics, including Windows 8, Windows Server 2012, Windows Azure, System Center, Office, SharePoint, etc. – pretty much anything that an IT pro would care about.  They meet the first Wednesday of every month in Bellevue at the Lincoln Square Microsoft office.  As long as I’m in town, I try to make it to this meeting.  There is a meeting coming up on Wednesday of this week (November 7th) at 6pm – there will be copies of Windows 8 Pro to be given away and free food provided, so RSVP now using the hyperlink above.
• Northwest System Center User’s Group.  This group meets every other month at the Microsoft Civica office in Bellevue to talk about System Center products, and there is ample participation from the System Center product groups (including Wally Mead).  I usually try to make it to this one too.  The next meeting is coming up on November 9th, so sign up now via the LinkedIn hyperlink above.
• Pacific Northwest SQL Server User’s Group.  This group meets every second Wednesday of the month, usually on the main Microsoft campus, to talk about topics related to SQL Server.  By my calculation that means there is a meeting coming up next week, for those interested in SQL Server, so keep checking their web site for details.
• Puget Sound SharePoint User’s Group.  This group meets monthly at the Microsoft Civica office in Bellevue to talk about SharePoint.  The next meeting is coming up on November 15th, from 6-8:30pm.

I’m sure there are more groups out there that I haven’t heard about, so let me know if there are more that I should add to this list.

MDT 2012 includes a task sequence for deploying an operating system into a VHD, setting up the computer for booting from that VHD.  See my previous blog posting about Deploy to VHD for more details on that.  As I mentioned in that blog, setting up a differencing disk to be created during the “Deploy to VHD” task sequence doesn’t make sense, as the parent VHD would be empty.  But it would be useful to be able to do it later, so that you have the already installed and configured OS in the main VHD, and then one or more differencing disks set up with that VHD as the parent.

So how do you actually do that?  The basic steps would be:

• Boot into Windows PE.  (You can’t create a differencing disk off of a parent VHD if that VHD is currently open, as it would be when boot from the VHD.)
• Use DISKPART to create a new differencing disk with the existing VHD as the parent.
• Create a new BCD entry for the differencing disk.

To help with that process, I created a new script called Diff.wsf, attached below, that will perform all of those steps.  Set up a new custom task sequence in your MDT deployment share that has a single step in it that runs the script like so:

Then reboot into your MDT Lite Touch boot image (either from media or from PXE) and run this task sequence.  The command line parameters above tell it what to do:

• /CREATE specifies to create a new differencing disk.
• /CLEAN specifies to remove any existing differencing disks that might be present.
• /STAGE says to copy the Diff.wsf and all related scripts locally onto the disk, in the same folder as the parent VHD file.

The task sequence completes while in Windows PE, and as soon as you click finish on the summary wizard page, the computer will reboot into the new differencing disk, causing all changes to be written into the diff file while the parent VHD remains unchanged (and effectively read-only).

After that initial differencing disk has been created, the locally-staged script can be run directly without even using a task sequence.  Assuming that the drive letter assigned to the physical disk containing the parent VHD (and the differencing disks) is D:, then the script will be located at D:\VHD\Scripts\Diff.wsf.  When you run it from there (initiated from within the currently-running OS), you can specify any of the parameters described above, or you can leave off the parameters and the script will prompt.

Note that the script can’t actually delete the differencing disk that is presently in use because, well, it’s currently in use.  But it can remove the BCD entry for it, and after a reboot it can then be deleted.  So the script will also take care of cleaning up any “orphaned” differencing disks it finds laying around.

A few final comments on scenarios for running Diff.wsf from within the currently-running OS:

• If you specify /CLEAN without /CREATE, it will remove all the differencing disks and boot back into the main VHD file.  This is useful if you want to update (e.g. patch) the installed OS.
• If you specify /CREATE without /CLEAN, you’ll get another differencing disk and another BCD entry – so if you really want to, you can boot between different differencing disks.

In my previous post, I talked about using a small RAMdisk to speed up the process of generating MDT boot images.  But the trial version only supports RAMdisks up to 4GB in size and only on client OSes.  One comment posted to that blog pointed out that StarWind also has a free RAMdisk available, and it supports larger RAMdisks and server OSes.  See http://www.starwindsoftware.com/high-performance-ram-disk-emulator for more information.  (Again, I’m not recommending it, just pointing out its existence.  Registration is required to download.)

This let me test out another scenario: Creating a RAMdisk large enough to hold an entire virtual machine, then seeing how much faster Windows can be deployed into such a virtual machine.  Fortunately, I have a workstation-class machine with enough RAM to do something like that.  I created a 20GB RAMdisk, formatted it as NTFS, then created a new VM where the VHDX file (using Windows Server 2012) was on the RAMdisk.

Using a physical disk array (attached to an Intel RAID controller) on the same system, I can deploy Windows 8 in about 10 minutes (new computer, clean install, no apps, no updates, etc.).  So how much faster is the RAMdisk?  On my machine, it took about eight minutes – noticeable, shaving 20% off of the time, but not as fast as I would have hoped.  That’s another case of shifting the bottleneck: the image is pulled across the (virtual) network, read from a physical disk (VHD), consumes CPU, etc.  Now if I only had enough RAM to load my entire server VHD into a RAMdisk to see what that does.  Last time I checked, the server required about a terabyte of disk space, so going that far is not possible.

But I can do the next best thing:  Create an MDT media ISO, copy that into the RAMdisk, and do a deployment using that instead of over the network.  How long did take?  About 7 minutes and 30 seconds.  Sigh, 30 seconds saved.

One more test:  Give the VM more CPUs.  Instead of the default single core, I increased it to four cores and tried again.  This time, the deployment completed in 6 minutes and 30 seconds.  Another minute shaved off.  (Don’t have lots of RAM?  Well, you probably have more than one CPU in your Hyper-V server, use it when building images.)

So that’s up to a 35% improvement in performance – good overall, but not quite the fantastic results I was hoping for.  Ah well, it was worth a try…

One of the slow operations in the MDT Deployment Workbench is the initial “Update deployment share” process that has to completely generate new Lite Touch boot images.  I always assumed that this was slow due to the amount of I/O being generated by the update process.

Recently, ATI and Dataram released a trial version of their RAMdisk software at http://www.radeonramdisk.com (not that I am endorsing the product – it just happened to come through my Twitter feed and it works on Windows 8), so I had a chance to test the assumption:  What would happen if the temporary storage used by MDT to generate the boot images would be on a RAMdisk?

So I installed the software on my laptop, created a 2GB RAMdisk, and formatted it as an NTFS disk.  First, I “completely regenerated” the MDT boot images without using the RAMdisk.  That process finished in six minutes and 15 seconds (6:15).  Then, to get it to use the RAMdisk, I did the following:

• Start an elevated command prompt.
• Set TMP and TEMP to point to the RAMdisk (E:\ in my case).
• Run “mmc.exe DeploymentWorkbench.msc” from the elevated command prompt, so it inherits the TMP and TEMP environment variable settings.
• “Completely regenerate” the MDT boot images again.

That looks sort of like this:

So what difference did it make?  Well, instead of 6:15, the whole process finished in 4:55.  Not too shabby, about 20% faster, but I expected more.  So why wasn’t it any faster?  Well, it turns out it’s just a case of shifting the bottleneck.  Watching the process using ProcMon and the Windows 8 task manager, I could see that the process was CPU-bound; the RAMdisk utilization was negligible.  Hmm, I guess it’s time for a faster CPU…

The trial software doesn’t support server OSes or more than 4GB of RAM; you have to purchase the full version for that.  Maybe I’ll try that sometime: Imagine a VM where the entire VHD is in a RAMdisk.  I wonder how long that would take…

I was working on a separate blog posting (to be posted soon) that referenced the “Deploy to VHD” support that was present in MDT 2012 and improved in MDT 2012 Update 1 and wanted to include a link to that previous blog.  Except there was no previous blog – I guess I never did one…

So let’s start with what was added in MDT 2012.  Present in this version are two task sequence templates:

• Deploy to VHD Client Task Sequence.  This is the equivalent of the “Standard Client Task Sequence” template, modified to deploy a client OS (Windows 7 or above) into a VHD file.
• Deploy to VHD Server Task Sequence.  This is the equivalent of the “Standard Server Task Sequence” template, modified to deploy a server OS (Windows Server 2008 R2 or above) into a VHD file.

Just to be clear, these task sequences have nothing to do with the creation of virtual machines (although you could use them in virtual machines – there’s nothing preventing a VM from using boot from VHD).  Instead, these are designed to do the following, in the case of a new computer deployment:

• Format and partition the physical disk.
• Create a new, empty VHD file.
• Format and partition the VHD file.
• Apply the specified operating system WIM image to a volume created in the VHD file.
• Set up a BCD entry to boot from the OS volume contained in the VHD file.

Of course all the other standard task sequence components are present too (injecting drivers, installing apps, applying patches, etc.).  Also, in MDT 2012 Update 1, the first step to format and partition the physical disk can be skipped, causing the new OS to be created on the existing disk, not disturbing the existing OS that might be on that disk.  As a result, you could then have a dual-boot computer, choosing the appropriate OS from the boot menu as there would be multiple BCD entries.

MDT 2012 Update 1 also will work with other scenarios, e.g. refreshes, but I’m not exactly sure why would want to do that.  Imagine what happens:  The user state is captured from current OS on the disk, a new VHD is created with the new OS, and then the user state is restored into that new OS in the VHD.  But now the user state is in both places, in the old OS and in the new OS.  (MDT 2012 RTW wouldn’t preserve the existing BCD, so you would always end up with only one BCD entry.  That behavior was fixed in MDT 2012 Update 1.)

If you want to try refreshing from a system already booting from a VHD file into a new OS running from a VHD file (same or different one), don’t.  It won’t work.  The challenge is that MDT really isn’t aware that the existing OS volume is in a VHD, so it might run into a variety of issues (e.g. staging the boot WIM in the VHD, or any operation that requires accessing the old OS files from within Windows PE).

So why is this referred to as “Deploy to VHD”?  It’s called that because MDT is not moving around VHD files (which would be highly inefficient, as these files aren’t compressed in any way).  Instead, it creates a new empty VHD file and deploys the operating system WIM file into it.  This is actually fairly fast and efficient, and surprisingly doesn’t complicate the deployment process very much at all.

Why might you want to do this?  This is primarily useful as a mechanism for setting up an additional OS on an existing computer.  Personally, I prefer using virtual machines for most things, but there could be some valid cases where using an OS running on the physical hardware (which after all is what you are doing, even when booting from a VHD file) is advantageous.

There are a few gotchas in the process though.  Because of some technical issues that could occur when using dynamically expanding VHDs, any time you perform a “boot from VHD” the VHD file will automatically expand to its full size (to make sure that Windows never needs to do this while the OS in the VHD is actively running).  So even if you have a 10GB operating system WIM (expanded from a 3GB WIM), you’ll end up with a much larger VHD file.  By default, MDT will create a VHD file whose maximum size is 80% of the available disk space on the OS volume.  So if you do a bare metal deployment to a computer with a 128GB physical drive, you’ll end up with a 102GB VHD file and about 26GB of free disk space.  You can customize this by editing the task sequence if you don’t want it to be that large:

Another gotcha has to do with the use of differencing disks, which you can see in the above screen shot.  What happens if you configure that?  Well, it will happily create a new empty VHD and then create a differencing disk on that empty VHD.  So then when the operating system image gets deployed, it ends up writing the whole thing into the differencing disk, while the parent VHD stays empty.  That’s not terribly useful.  But there are some additional properties in the underlying MDT ZTIVHDCreate.wsf script that could be leveraged instead to have MDT start by making a copy of an existing VHD file, instead of starting with an empty VHD.  But that’s a more advanced scenario, left for the creative types.

Apparently I was so wrapped up in the MDT 2012 Update 1 development process that I forgot to talk about one of the areas that consumed weeks of my time before the MDT 2012 Update 1 release: improvements to the “Roles and Features” logic.

In MDT 2010 and MDT 2012, we had a task sequence action to “Install Roles and Features” that you could use to install roles and features on various operating systems.  But it didn’t support Windows 8 or Windows Server 2012; it tried (unsuccessfully in some cases) to merge all of the role and feature lists into one single list for the rest of the OSes; it didn’t include a complete list of roles and features for all the OSes (e.g. Windows 7); etc.  Basically, it needed a lot of work.

So in MDT 2012 Update 1, we took the opportunity to do some housecleaning.  First, we changed the “Install Roles and Features” task sequence UI to allow you to display a filtered list of roles and features for the particular OS you are deploying:

We also made sure that the complete lists were present for all the various OSes:  Windows 7, Windows 8, Windows Server 2008 (full install and core install), Windows Server 2008 R2 (full install and core install), and Windows Server 2012 (full install and core install).  (We didn’t change the list for Windows XP and Windows Server 2003.  Given that those are already in extended support and rapidly approaching their end-of-life date, we’re not adding new functionality for them.)  And we verified the lists – something that gave our test team a real workout, as we had to make sure that we properly handled all the dependencies between the roles and features on OSes where that wasn’t handled automatically (e.g. Windows 7).

Then, we added a new Lite Touch wizard pane to let you dynamically choose roles and features.  It automatically displays the right list based on the OS being deployed, allowing you to choose additional roles and features to install later in the task sequence:

What do you need to enable this wizard pane?  Just add a step to the task sequence (somewhere in the State Restore phase; the exact location doesn’t matter, but I typically choose to put it in the “Custom Tasks” group) and the new pane will automatically show up.  If you don’t want the wizard pane, you can turn it off through Custom Settings.ini by specifying “SkipRoles=YES”.

We also added a new “Uninstall Roles and Features” task sequence step that can be used to remove roles and features that you no longer want.  It presents exactly the same list of roles and features inside the task sequence editor:

Notice the red box above, which I highlighted to point out one specific Windows 8 and Windows Server 2012 feature (and only shows up when you choose Windows 8 or Windows Server 2012):  Not only can you uninstall roles and features, but you can also completely remove the components, getting rid of all the files related to that component.  (If you ever want to add those back, Windows can download the components from Windows Update, or you can provide the original media to pull them from the WIM file.  See http://technet.microsoft.com/en-us/library/hh824822.aspx for more details around this, as well as http://technet.microsoft.com/en-us/library/hh825020 for information about how to specify an alternate “repair source” instead of Windows Update.)

It’s worth noting too that we added some extra logic to handle the installation of .NET 3.5 on Windows 8 and Windows Server 2012.  For both of these OSes, the .NET 3.5 feature is not present in the standard WIM file, but the files do exist on the media in the \sources\sxs folder.  So for Lite Touch deployments, we will automatically provide these files to Windows as long as you have imported the full source files into your deployment share.  For ConfigMgr clients, you would need to do a little bit more:

• Set up a network share containing the contents of the “sources\sxs” folder from the original Windows 8 or Windows Server 2012 media.
• Set a task sequence variable (either through CustomSettings.ini or directly in the task sequence) called “WindowsSource” that points to the location of that folder.

Behind the scenes, the script responsible for performing the role and feature work, ZTIOSRole.wsf, figures out the right thing to do for each OS:

• For Windows Server 2003, use SYSOCMGR.EXE.
• For Windows Server 2008, use SERVERMANAGERCMD.EXE.
• For Windows 7 or Windows 8, use DISM.EXE.
• For Windows Server 2008 R2 (non-core) or Windows Server 2012, use PowerShell.
• For Windows Server 2008 R2 (core), use OCSETUP.EXE.

Fun stuff.  (For those of you keeping track, this also means that MDT now actually uses PowerShell itself during a deployment task sequence to install or uninstall roles and features on Windows Server 2008 R2 and Windows Server 2012.  To keep things simple, the ZTIOSRole.wsf script calls the ZTIOSRolePS.ps1 script to take care of this.  Look at the logic in ZTIOSRole.wsf and the new ZTIPSUtility.vbs script to see how that works, in case you ever have the need to do something similar, e.g. do some work in VBScript and some in PowerShell.)

For those of you using ConfigMgr 2012, you may be interested in this.  When ConfigMgr 2012 Cumulative Update 1 was released, it was discovered that you could no longer create a new task sequence using MDT 2012 Update 1 and the “Create MDT Task Sequence” wizard.  When you tried, you would get an error like so:

Microsoft.ConfigurationManagement.ManagementProvider.SmsConnectionException: Failed to validate property Type. ---> System.Runtime.InteropServices.COMException: Failed to validate property Type.
   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
   at System.Management.ManagementObject.InvokeMethod(String methodName, ManagementBaseObject inParameters, InvokeMethodOptions options)
   at Microsoft.ConfigurationManagement.ManagementProvider.WqlQueryEngine.WqlConnectionManager.ExecuteMethod(String methodClass, String methodName, Dictionary`2 methodParameters, Boolean traceParameters)
   --- End of inner exception stack trace ---
   at Microsoft.ConfigurationManagement.ManagementProvider.WqlQueryEngine.WqlConnectionManager.ExecuteMethod(String methodClass, String methodName, Dictionary`2 methodParameters, Boolean traceParameters)
   at Microsoft.ConfigurationManagement.ManagementProvider.WqlQueryEngine.WqlConnectionManager.ExecuteMethod(String methodClass, String methodName, Dictionary`2 methodParameters)
   at Microsoft.BDD.Wizards.SCCM_ImportTaskSequenceTask.DoWork(SmsPageData smspageData, Dictionary`2 data)

A new MDT 2012 Update 1 download has now been released containing a fix for that:

http://www.microsoft.com/en-us/download/details.aspx?id=25175

From the text on that page:

MDT 2012 Update 1 version 6.1.2373.0 (Version 6.1.1 on this page) was made available for download on September 19, 2012 and adds support for System Center Configuration Manager 2012 CU1 and System Center Configuration Manager 2012 SP1 Beta. It can be identified as MDT build 6.1.2373.0 in the MDT Workbench console or in the installer program properties. This is the latest version and we recommend all users run the latest version when they can to ensure the smoothest experience during future upgrades.

So for those of you keeping track, the original MDT 2012 Update 1 release was build 6.1.2369.0, while the new release is 6.1.2373.0.  Other than this wizard fix and some documentation updates for ConfigMgr 2012 SP1 Beta, there are no other changes.  So if you are using ConfigMgr 2012, please download the new version, install it, and then repeat the “Configure ConfigMgr Integration” process to integrate the updated binaries into your ConfigMgr console.  (You don’t need to recreate any task sequences or change the MDT toolkit files package if you are already running MDT 2012 Update 1.)

If you are only using Lite Touch, you don’t need to upgrade, but there would be no harm in doing so.  Note that after doing this you would need to update your deployment shares, boot media, etc.

Inside the new Windows Assessment and Deployment Kit (ADK) that was released back in August is a component called the Windows Performance Toolkit.  I’ve talked about this at various events over the past few months, noting that you can use this kit to help identify what is causing your computers to take so long when they start up before they are usable.  But people always ask “how do I do that myself”.  First, I would suggest reviewing the documentation that is available online:

http://msdn.microsoft.com/en-us/library/hh162945

Some of you might have used a previous version of this toolkit, which included tools called XPERF and XPERFVIEW.  This latest version includes new replacement tools (although the old tools are still around for compatibility, for those die-hard users) called the Windows Performance Recorder, WPR.EXE (command line) and WPRUI.EXE (GUI), and the Windows Performance Analyzer, WPA.EXE.  The Recorder captures the events; the Analyzer displays the results.  There is another utility called XBOOTMGR.EXE that takes care of one of the more complicated processes: setting up the computer so that it captures information at boot time.  That’s the one we’ll want to use here.

So let’s assume that you are experiencing a problem with Windows 7 computers starting up slowly.  You can reproduce this at will, but it’s not immediately obvious why.  So you log on to the Windows 7 computer, install the Windows Performance Toolkit redistributable from a computer that has the ADK installed (so we don’t need to put the whole ADK on the Windows 7 computer), and then run the “xbootmgr.exe -trace boot” command to get everything started:

It will quickly reboot the computer, so be prepared.  As soon as you can, log in to the computer to finish up the process.  By default, it will wait two minutes to let the system settle down, but you can choose to end it sooner by clicking the “Finish” button:

Then you can run the Windows Performance Analyzer to look at the trace; start it using the shortcut on the Start menu.  (Note that it requires .NET 4.0 so if you don’t have that installed it will complain.  You can always analyze the trace remotely if you want, just copy off the *.ETL file.)  Once it launches, have it open the trace file that was created in the same folder we ran XBOOTMGR.EXE from:

Since the Windows Performance Analyzer is very graphically-oriented, you might want to do this using a high-resolution display:

And that’s really where the fun starts: figuring out what graphs to add, what timeframes to focus on, and trying to get to a root cause.  I’m no expert myself (yet), but I can see from the trace above that most of the time was spent in the Winlogon Init process.  And probably not coincidentally, most of that time corresponds to the gpscript.exe and wscript.exe process lifetimes.  And farther below, we can see generic events from Group Policy that confirm group policy processing took about a minute.  So what was going on in this case?  A bad startup script that took a minute to complete, combined with a policy setting that said “don’t run startup scripts asynchronously”.

Now I just need to teach myself how to better analyze these traces.  Let the fun begin…

Traveling from New Zealand to Australia for the TechEd Australia conference, I had some additional time to work on the troubleshooting guide, which is now starting to feel more like a book.  I added two sections to this document, one reviewing all the possible return codes from the MDT scripts (yes, there are lots of them, over 30 pages worth) as well as from USMT 5.0.

As you can probably tell, my focus is still on reference material.  See the attached file below.