Chris's Blog - Just weedin around Microsoft

Failure during Windows Server 2008 System State Backup & System Writer Missing

For some of those new to DPM 2007, you might find yourself in a situation where a Windows Server 2008 (and R2) is failing during backups consistently.  The consistency part is the ironic is that it is always failing and might lead you to review a bit more what is going on.  This problem, as outlined in this post, is often related to missing the Windows Backup feature on Windows Server 2008.

However, there are cases where some servers will fail unless you reset permissions for the System Writer.  This was outlined in this forum post yet no one actually batched this up and shared and since I had quite a few servers that I needed to run this on I batched this up and it corrected the issue.

To correct this problem, simply copy this batch file and execute on your Windows Server 2008 servers in question -

REM  Set ACLs correct for System Writer
REM  *************************************
Takeown /f %windir%\winsxs\filemaps /a
icacls %windir%\winsxs\filemaps  /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps  /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps  /grant "BUILTIN\Users:(RX)"
icacls %windir%\winsxs\filemaps  /grant "Administratoren:(RX)"
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.*  /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.*  /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.*  /grant "BUILTIN\Users:(RX)"
icacls %windir%\winsxs\filemaps\*.*  /grant "Administrators:(RX)"
Takeown /f %windir%\winsxs\temp\PendingRenames /a
icacls %windir%\winsxs\temp\PendingRenames  /grant "Administrators:(RX)"
icacls %windir%\winsxs\temp\PendingRenames /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\temp\PendingRenames /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\temp\PendingRenames /grant "BUILTIN\Users:(RX)"
Takeown /f %windir%\winsxs\temp\PendingRenames\*.* /a
icacls %windir%\winsxs\temp\PendingRenames\*.*  /grant "Administrators:(RX)"
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "BUILTIN\Users:(RX)"

REM  Restart the Cryptographic Services
REM  **********************************
net stop cryptsvc &;& net start cryptsvc

After executing this, you will be set and ready to re-run your job and off you go.  This is the smallest post in history but it solved a big problem of mine on a couple of servers.

-Chris

Deploying Windows 7 in the Enterprise – A Blueprint for Success

It’s taken nearly a year and four developers, 2 QA, and 3 PMs to build the solution to deploy Windows 7 at Microsoft & other customers.  Code-named “Modena”, this solution offers customers who utilize System Center Configuration Manager 2007 to deploy Windows 7 easily, with little headaches, and hopefully rapidly.  My team was responsible for building the solution and we are putting the finishing touches on Modena and about to release RC3 to Microsoft’s Connect site for our customers to download and begin using.

In the meantime, I wanted to share an update about Modena as we recently released an article titled “Windows 7 and System Center Configuration Manager:  Your Windows 7 Deployment Guide” in the November 2009 edition of TechNet magazine.

What does Modena offer Configuration Manager customers?

A lot of questions are swirling around about ConfigMgr’s OSD feature, the Microsoft Deployment Toolkit (MDT) 2010, and now Modena.  We hope to update folks on the guidance as there are a lot of customers frustrated by not hearing from Microsoft a definitive answer around deploying Windows.

Nonetheless, the purpose of Modena from the outset was to expand on the functionality already offered in OSD as out-of-the-box features simply were not enough for the customers we were responsible for delivering the solution too.  We built a very customizable end-user experience that asks users a set of questions to prepare the system for the migration to Windows 7 and then they walk away and when they return their system has been migrated to Windows 7 with all their data intact.  For administrators, we also spent a lot of time ensuring that we prepared for the unhappy path scenario so that we protect user data and also provide help desk staff enough logs to determine the root cause of the issue.

Modena offers the following:

A end-user experience wizard easily configurable    
  Robust task sequence with built-in error handling that is snap-in and go  
  Key functionality needed to deliver wizard to end-users in their context (ServiceUI.exe)  
  Easily extensible Pre-Flight checks to verify systems are ready for Windows 7, prior to migration  
  Application Discovery Pre-Flight to discover applications already installed and set Modena to re-install them (ConfigMgr packages only)  
  Application selection capabilities that supports local configuration or dynamic Web service to simplify updating of application repository  
  A simple, configurable application called OSD Results that communicates success or failures to end-users at completion of migration

How do I learn more about Modena & Deploying Windows 7?

After you’ve reviewed our article above, then turn your attention to my team’s blog at http://blogs.technet.com/osd where we will continue to share details about our release as well as teach you how to utilize Modena.  We’ve been very, very busy the past few months and haven’t focused as much on sharing information since our last release but expect this to change over the coming months as we turn our attention to helping you successfully deploy Windows 7 at your company.  This is our goal – helping you adopt Windows 7 faster, more reliably, and with confidence. 

If you have any questions about Modena, please do not hesitate to drop me a line!

Thanks,

-Chris

Unable to Extend System Volume & DPM Backup Failure: Change the System State backup location

I recently had a couple of servers where they had rather small system partitions, one was Windows 2003 while the other was Windows 2008 RTM and DPM continued to fail with replica is inconsistent errors.  The DPM documentation and event errors were referencing the replica and volume partition size(s) and all checked out with plenty of space according to the current data size. 

Fix #1:  Expand the available space for the system partition

As a test, I thought about checking the actual system partition size on the protected servers and noticed small free space available on these virtual machines.

For testing purposes, I shut the virtual machine and expanded the system partition virtual hard disk.  To do this, I simply did the following:

1. Shut the virtual machine down in the VMM Administrator Console
2. Right-click and selected properties on the virtual machine
3. Highlighted IDE disk 1 and checked compact and expand disk and selected the expanded space size
4. Click OK
5. Restarted the virtual machine upon the job completion
6. Opened Server Manager (Server 2008 only), elected disk storage, and right-clicked and chose Re-scan disks
7. Right-clicked the system partition and selected to Extend volume

After the volume free space is sufficient (greater than 30+ gigabytes was my goal), then I re-run the consistency check for the replica partition for the server(s) that failed.

To re-check the consistency check, you do the following:

1. Open the DPM Administrator console
2. Click the Protection Tab
3. Locate the protection group, and expand if not already expanded
4. Locate the protected server and expand it to see the current volumes and system state backup
5. Right-click on the failed state item and select “Perform Consistency Check”

Fix #2:  Re-locate the location for the System State backup location on the protected server

The one problem that many might run into is the system partition isn’t expandable using the “extend volume” as it is either not supported (Windows Server 2003) or is a physical disk limitation.  In either case, the choice at this point is less related to creating the relevant space in order to successfully get system state backups to succeed and more pointed at moving the system state backup to another location. 

NOTE:  If you have limited space for your system partition and no other volume with significant space available to host the backup, you have bigger problems than the replica backup failing.  You should fix the limited free space first for this server and then focus on the following fix.

As I wasn’t familiar with how to successfully re-locate system state backup location, I used Bing search to see if others had figured this out already.  I came across this small blog post that was very helpful and for the most part was completely accurate thought I did have to make one minor tweak to finally get things happy (e.g. no failures or errors, just the big old Green OK check mark).

Follow these steps outlined in the post above -

1. Login to the server with credentials that is an Administrator
2. Navigate to the installation directory for the DPM agent, typically %programfiles%\dpm
3. Double-click datasources directory
4. Right-click on the PSDataSourceConfig.xml file and select open with and choose Notepad (or your preferred text editor)
5. Locate the <FilesToProtect> element
6. Change the current path from %systemdrive%\WindowsImageBackup\* (if Server 2008) or %SystemDrive%\DPM_SYSTEM_STATE\%computername%.bkf (if Server 2003) to the location you would like it to reside in (for example, H:\WindowsImageBackup)

As mentioned in the blog post above, DPM’s protection group will now go south on you and it will seem permament.  This is expected since the location of the system state configuration has changed.  The protection group needs to be modified through the wizard and as mentioned it should do so with no changes.  This, unfortunately, didn’t do the case every time and some protection groups were still failing.  This was easily determined using the DPM Events in the event viewer for the server with the following:

“DPM has detected changes in the file locations or volume configurations of protected objects…”

In order to correct this failure, I had to use the modify protection group wizard and remove the System State from the protected server and add it back.  After doing this, the replica consistency started working as planned.

Summary

In the short time I’ve started using DPM 2007, I’ve found it to be rather straight forward to troubleshoot failures with replica backups.  The issue is usually related to volume space not large enough, in my experience thus far.  With some quick cleanup, you can easily turn things around assuming you have the disk space (got to love SAN’s and DAS!) and the DPM console.

Hope this helps,

-Chris

Data Protection Manager 2007: Replica is inconsistent on Windows Server 2008

I recently took some time to learn more about Microsoft’s System Center Data Protection Manager 2007 since it was again a product in my division that I knew very little about.  Beyond that, I had quite a bit of incentive to get our engineering lab which is handled by me as a “side job” besides my Lead PM role for MPSD Engineering.  We recently had a Hyper-V Failover Cluster slight meltdown that was single-handedly caused by my mistake.  The good thing is that I’ve since recovered with less than a day’s worth of productivity lost by Dev, Test, and PMs.

As I continue to post more about DPM 2007, I thought I would kick-off what I noticed was the first glaring thing that showed up once I created my Protection Groups.  For each computer running Windows Server 2008 or above, I noticed that every single System State backup was failing.

Windows Server 2008 Protected Computers Requirements

The first thing I learned quickly was that right in the event viewer was a suggestion that sticks out like a sore thumb.  For any system state to get backed up on Windows Server 2008 (or R2), the Windows Backup Services feature must be installed.  If not, DPM will successfully backup everything other than System State.  This is not a feature that is often enabled on Windows Servers that I’m aware hence the reason it caught me off-guard.

My second “assumption” was that the DPM protection agent would have installed any pre-reqs and this was in fact not the case as it didn’t enable Windows Backup nor did it tell me to.  So off I go to enable this service so that in the case of a disaster recovery, I can restore the server(s) in question.

Batch File to enable Windows Server 2008 Backup Features

To simplify things, I made this quick batch file that is simplified deployment to all my servers that needed backed up.  This is so simple it is scary but I tend to do things simple as they reduce time and complexity.

REM  Install Windows Server Backup
REM  *****************************
ServerManagerCmd -install Backup-Features

This will install the service needed to effectively backup the System State on Windows Server 2008.

Summary

Data Protection Manager 2007 is a pretty slick piece of software.  The primary piece I like the most is the ability to do disk-based backups which is super efficient.  The learning curve seems to be low while the upside is great from a CYA perspective.  The above isn’t the only issue I’ve encountered and you should see some more posts this week outlining what those are until I have this ship running nice and smooth.

Adding IIS Host Headers in DNS using Netdom utliity

An age old question for IIS administrators is how to effectively manage small, medium, or even large Web servers with a minimal set of IP addresses.  The network team often manages layer 3 which includes the IP addresses that are available for Web servers.  If an unlimited supply of IP addresses existed, there would be no need for host headers.

IIS provides a mechanism that supports the creation of multiple Websites (often referred to as virtual servers) using a single IP address through the use of Host Headers.  Host headers are supported in HTTP version 1.1 and more is included in the following Microsoft Knowledge Base article http://technet.microsoft.com/en-us/library/cc753195(WS.10).aspx which isn’t the purpose of my post. 

Instead, the often difficult aspects of the use of host headers in the enterprise is for a Web admin (or any admin that is the non-AD\DNS administrator) to get the host header created.  I was recently in this dilemma and I was completely taken aback when I learned about functionality included in the netdom.exe utility.

Creating Alternate Identities for a Computer using Netdom

By default, any domain-joined Web server already has a single FQDN relative to the actual computer name.  This is created in the DNS server using a A record utilizing dynamic registration.  However, the problem begins when one needs the ability to have an alternate name for the server that doesn’t match the computer name hence creating an alternative DNS A record.

Little did I know that this is possible using a utility on the Server (Windows Server 2008/Windows Server 2008 R2)

Using Netdom /Add to Configure Alternate Identity (Host Headers)

To add the alternate name for the computer name, do the following:

NOTE:  In the following command, the IIS Web Server name is IIsWebServer name and the IIS Host Header desired is hr.contoso.com

1. Open an Elevated Command-Prompt (e.g. Administrator)
2. Type Netdom /? to see the syntax available or netdom /add /? to see more
3. To add the alternate name, type the following:

Netdom computername IIsWebServer /Add:hr.contoso.com

After using this command, reboot the server for the change to successfully complete.

Summary

Amazingly, this is extremely easy and completely successful.  If there are others out there that were aware of this then shame on me but I couldn’t believe I missed this.  Nonetheless, I hope someone else out there finds this as helpful as I did!

-Chris

Failover Cluster Snafu – Forcibly removing Failover Cluster Feature after Cluster Failure

My secondary job is to maintain our engineering lab (part of the mantra of “Do More with Less”) as we don’t have anyone dedicated to this role.  This lab is running completely virtualized minus our SQL infrastructure which is running on a 2-node Failover cluster since so much of our infrastructure relies on SQL.  In our case, we have a 7-node Failover Cluster that utilizes R2’s Clustered Shared Volumes (CSV) and I recently took a vacation.  Whoa, I bet you didn’t see that coming.  What does a 7-node cluster and your vacation have anything to do with each other.

I’m glad you ask … it is completely related to the fact that the 7-node cluster is conveniently triggered to “fail” during my vacation causing me to stop my vacation and take a look.  Recently, I had a node that simply went south during my vacation and I was super-high on the frustrated level as this lab isn’t my primary focus – though it seems to occupy me way too much lately!

What I thought I would share today is completely unsupported I’m certain but luckily you can take my gossip & rants on this blog as “Well, he doesn’t usually do things in the supported fashion anyhow…”

Dang it… I can’t remove the Failover Cluster Feature because it is still a part of a Cluster

What a “cluster” you might have on your hand.  No Pun Intended.  This is exactly the scenario I had.  I had a host go down that, unfornately, didn’t have access to the cluster any longer since it was evicted.  However, the node itself was seriously convinced that it was in fact still a vital part of the family.  I got high on the level of frustration and decided to start the digging process…

NOTE:  DIGGING IN THE REGISTRY FOR LITTLE JEWELS ISN’T RECOMMENDED NOR THE RIGHT IT APPROACH.  IT’S AN APPROACH FOR THOSE WHO ARE WILLING TO GAMBLE EVERYTHING AND CAN SAFELY CYA THEMSELVES IF THE GAMBLE DOESN’T PAY OFF.

<notFortheFaintofHeart> 

How to Force Failover Clustering Feature to be available to Remove

Now you know the warning.  Let me share how I just came across this way to force R2’s Server Manager feature wizard to again forget about Failover Cluster and allow me to move forward.  To do this, go to your broken node and open the Registry.

Backing up your registry right now is a great idea… do it and return.

1. Open Regedit
2. Locate HKLM\CurrentControlSet\Services\ClusDisk & ClusSvc
3. Delete these keys

You have now royally ticked off your R2 server though it is only for a brief moment.  Move to the next step…

Uninstalling Failover Cluster when cluster is unavailable

The next step is to open Server Manager and to remove the Feature for Failover Cluster.  When you do this, Server Manager will remind you that you shouldn’t move forward unless you know that all the services are moved off this cluster.  It is…so choose Yes and move on.

After the removal, it will likely ask for you to reboot which is a pleasant idea.  After rebooting, you can now safely add the feature back and now re-connect to the cluster and start the rebuild process.

</notFortheFaintofHeart> 

Simple.  Easy.  Not recommended…but if you are like me then time sometimes is worth the risk.  If you screw up, you can always rebuild your server. <grin>

Thanks,

-Chris

Differencing Disks & Merging: 80070005 Error … Just one persons Lesson Learned!

Recently, I had a rather simple demo for an audience that required me to have multiple systems running simultaneously and at different phases of a 2 hour migration to Windows 7.  The original plan of attack was to use a single base image, Windows XP SP2, and create differencing disks for each of the 3 virtual machines.  This is seemed to be the most reasonable approach.

In today’s post, I’m going to talk quickly about how I rather abruptly decided to get away from this model (single source, multiple differencing disks) and move to multiple sources using snapshots.  There are most experts that would argue that this is splitting hairs (or tomato vs. tomatoe) as snapshots utilizes the same principles of differencing disks.

When moving to this model, I had to make duplicate copies of the original source VHD (also known as the parent) and then use the Edit Virtual Disk wizard to “merge” the differencing disks (including some snapshots) until I ended up with a single VHD.

Attributes of a Differencing Disk

The one thing that many might miss about differencing disks is that the source disk is marked by the system as Read-only.  This is to avoid any “clobbering” of the file and causing all virtual machines based on the source from self-destructing.  This little tid-bit of information can save a lot of your time when you are attempting to merge to the parent and the system (e.g. Hyper-V) believes there to be additional virtual machines to still be using this source disk.  (This is my theory because I don’t know at what point Hyper-V removes the read-only attribute or if it ever does – I’m trying to follow up with folks internally to see if I can get an answer).  I digress…

Checking Source Disk Attributes

For those not familiar, to check to verify that the Read-only attribute is not enabled, do the the following:

1. Locate the source parent file (NOTE:  Not the snapshot\child file)
2. Right-click on the file, select Properties

If the Attribute for Read Only, un-check. 

It is important to note that when using Snapshots the parent file does not have the Read-only attribute.

Ouch:  General Access Denied Errors (0x8007005) during Merge

So here goes… you are attempting to merge your differencing disk to the parent when Hyper-V throws you a nice little error that says “General Access Denied” which leads most Admins to run to the file system directory and start reviewing the NTFS permissions.  If you were like me, you didn’t panic at all and then checked the permissions.  Wow, the permissions are fine and the panic sets in….

The first steps are to go through the process of merging…

1. In Hyper-V Administrator Console, right-click on the Server and choose Edit Disk
2. Locate the VHD\AVHD you wish to merge (e.g. d:\VM\MyTest.VHD)
3. It will then ask you whether you would like to compact or merge…
4. Select Merge
5. Click Next
6. Select to Merge with Parent
7. Click Finish

…and then it Happens!

Resolution

For protection, some folks set the source VHDs to read-only to avoid anyone using them incorrectly.  As mentioned, I haven’t found a solid repro whereby the read-only attribute is done by the system (aka – Hyper-V\Windows) so for now I’m going to assume humans were involved in the conspiracy.  Nonetheless, the general 8007005 error seems related to the inability of Hyper-V to merge the disks to a single file is the read-only for the source.  This problem doesn’t occur when the merge takes place to another filename whereby Hyper-V copies the two files to a new file.

Remove the read-only and you are in business…

Enjoy!

-Chris

[Info] – SCVMM R2 Documentation Live on the Web

It was recently announced internally at Microsoft that the VMM documentation team has made available via TechNet the documentation to go along with VMM 2008 R2.  For those of us who spent a great deal of time digging, playing, and mostly in “Fire first, Aim second” mode due to some minimal documentation for RTM this is a welcome site.

VMM R2 Product Docs:  Technical Library

*  Courtesy of Microsoft TechNet

Enjoy!

-Chris

Virtualization Tip: Migrating to SAN’s “after the fact” with SCVMM using R2’s Migrate Storage

In today’s post, I thought I would share some insight into how to effectively migrate to Storage Area Network’s (SAN) after you’ve already got an SCVMM environment up and running.  You found yourself with several Windows Server 2008 Hyper-V hosts and you were moving along with very little issues; though, you recently have noticed that downtime is unavoidable if you don’t have your backend storage running on SANs.

You will get overwhelmed when researching the issue and I just thought I would share one person’s perspective who had no SAN, lot’s of physical servers running Hyper-V, and decided to learn second, execute first.  Typical for my personality type…

Existing Environment

Prior to migrating to a SAN, each server had local drives in a RAID 5 configuration.  The volume was dedicated for Virtual Machines and Scratch directories.  Migrations between hosts utilized network transfer and was using BITS and at minimal required that the VMs were in a saved state.  The average transfer time was around 10 minutes.

Preparing for the SAN

The SAN was installed (EMC Clarion AX4-5 with two shelves) and utilizes Fiber to connect to the hosts.  The EMC Clarion is a 3U unit and is connected to all our Hyper-V servers along with our SCVMM server.

After installation, you have to utilize the NaviSphere Express software to configure your Disk Pools and volumes.  This was obviously done prior to the connection to the servers.

For our environment, we have the following configuration:

VM Source Volume – This volume has our read-only source VHDs and is utilized by using Differencing disks to never alter the actual source.  The volume is small with 1 TB usable space in a RAID 5 (optimized for read-only).

VM Storage Volume – This volume is in a RAID 10 configuration and has all the differencing disk(s) for our virtual machines.  Since using RAID 10, this is optimized for Read\Write.  This volume is ~4 TB in size and currently supports about 30 virtual machines with 53% utilization.

VM Library – This volume is large, SATA drives (1 TB each), in a RAID 10 configuration.  This is the VMM library resources share.

Decisions to make prior migration to the SAN

Prior to migrating to the SAN’s, you will need to make some decisions that are very, very important.  Those decisions are the following -

1. Do I utilizes Windows Server 2008 R2’s Clustered Shared Volumes (CSV) to take advantage of simplified management of LUNs?
2. If answer to number 1 is No, then you need to determine your LUN strategy to work around the 1 VM to LUN limit

The decision I made, since I could, was to run all our VMs as Highly Available and utilize CSV so that I can reduce my time in managing the physical disks and volumes in NaviSphere.  As you can see above, I created one single LUN (~4 TB) that would house all of our virtual machines.  In a later post, I will work through everything I did to get clustering up and running and how I enabled CSV’s.  For the purposes of this exercise, though, lets assume that I have a single LUN that is presented to multiple hosts running in a cluster.

Migrating to the SAN

The first thing I was excited to see was System Center Virtual Machine Manager R2’s behavior after each of the servers were a part of a cluster.  Rather than having to break down SCVMM by removing the hosts, VMM quickly realized that the hosts were now “clustered.”  (I lost for a brief moment connectivity and the hosts were in a warning state during this period.

The VMs, though, were not running at the time on the “shared storage” hence each physical host was in the cluster though not utilizing the resources of the cluster.  Prior to the release of SCVMM R2, I would have been required to rebuild my VMs and place the VHDs on the SAN.  SCVMM’s Migrate Storage feature (outlined later), though, was the magic that turned this process into a very simple migration.  Let me explain what I did…

Verifying that the shared storage drives (or mounts) are ready…

This is the first step as the process is required because the physical host where the VM lives already is required to have access to the storage.  To verify this, connect to the physical host that is currently running the VM -

1. Open Server Manager
2. Click Disk Management
3. Locate the volume for the shared storage, verify it is online and initialized

Migrating a Virtual Machine (One at a time)

Because of the sensitive nature, I started off by doing each migration one-by-one as I wasn’t sure of the “outcome” but as I became more comfortable I simply kicked off the process utilizing VMM’s PowerShell interface which made the migration move much quicker.  For now, I will step through the process using the VMM Administrator’s Console -

1. Open the SCVMM Admin Console
2. Locate the Virtual Machine you want to migrate to the SAN (currently running on the local physical host)
3. Right-click on the VM, select Migrate Storage
4. Utilizing the Migrate Storage Wizard, select the CSV volume (C:\ClusterStorage) on the physical host and complete the wizard

To validate the the Virtual Machines are actually utilizing the CSV storage, use the Failover Cluster Admin console and under Services and Applications you will see it listed.  The part I loved about this process is that VMM was intelligent enough to realize this was a clustered shared volume and during the migration *automagically* made the Virtual Machine(s) highly available.  This was verified in the SCVMM Admin Console by doing the following:

1. After migration completes, highlight the virtual machine in SCVMM Admin Console
2. Right-click the VM, select properties
3. Click Hardware configuration tab
4. Scroll to the bottom to the Availability section, validate it lists as “High”

That was it.  As I said, after a couple of these using the manual process it is rather easy to steal the PowerShell code and customize it and poof, your VMs are highly available.

Enjoy migrating!

-Chris

Real World Example of Troubleshooting R2 Live Migration using CSV’s

As I’ve mentioned several times, we run our engineering lab on much of the latest & greatest.  There are many who would argue that this is a luxury (one afforded to those who work at a Software company such as Microsoft) and I wouldn’t have a solid argument against their argument.  However, it doesn’t change the fact that often we are so cutting edge that when “issues” arise we don’t get a solid response internally without development debugging.  There is a challenge to often getting product group developers on tap to help and this leads to “fiddling” around with things which is where I found myself this past week.

We run our lab on a 7-node Hyper-V cluster that has each node attached to our EMC Clarion AX-4 SAN.  This cluster is in a node-majority setup and recently I found that R2’s new Live Migration functionality wasn’t working as designed.  After a little bit of investigation, I determined that only 2 of the 7 hosts were unable to migrate and each time a migration was attempted (for any VM), the following error was thrown in the event viewer -

'Machine name’ live migration did not succeed at the source.

Failed to get the network address for the destination node 'server a': A cluster network is not available for this operation. (0x000013AB)

Unlike many event error messages, the error “A cluster network is not available for this operation” is literally the value for the Win error 000013AB.

Quick Review of the Infrastructure

Each member of the cluster has two physical NICs, one for management and one that is dedicated to Hyper-V and represents multiple networks through VLAN Trunking.  The first thing that most folks, out-of-the-box, will see happen with clustering and networking is the lack of “exact match” when using Virtual Machine Manager (VMM) to manage your cluster.  Basically, the following two items must be identical in order for you to manage any machines in the cluster -

1. Network Name (on the physical NIC)
2. Network Tag

Depending on the number of Nodes in your cluster, this can be a pain in the you know what to determine.  It is very easy though utilizing the following PowerShell script and execute on your VMM server -

#####################################################################

function DisplayNicInfo($VMHostName)

{

  $yy= get-VirtualNetwork -VMHost $VMHostName; 

  $yy | ForEach-object {write-host "    Name      " $_.Name;

                        write-host "    Locations " $_.Locations;

                        write-host "    Tag       " $_.Tag;

                       }

}

#####################################################################

$clusname = read-host "Host Cluster name to check"

Write-Host ""

$VMMServer = get-vmmserver -computername localhost

$Cluster = get-vmhostcluster -name $clusname

$VMHosts = get-vmhost -vmhostcluster $Cluster

$VMHosts | ForEach-object {Write-Host "VMHost: " $_.Name;

  DisplayNicInfo($_.Name);

  Write-Host ""}

NOTE:  This PowerShell script is going to require you to lower your script in order to execute it.  To do this, do the following -

Set-ExecutionPolicy Unrestricted

To execute, you would then just open PowerShell, navigate to the PS1 file, and execute it.  Enter the Cluster Network Name (e.g. cluster.contoso.com), and it will display all nodes current Network Name & Tag configuration.  Fix whatever doesn’t match…

For more information, see the following blog I did about Network Name & Tag.

How do I do “Live Migration” when using Cluster Shared Volumes?

The first thing to note is that Live Migration is possible without the use of Clustering or Clustered Shared Volumes (CSV).  It is accomplished in Virtual Machine Manager (VMM 2008 R2) by right-clicking the virtual machine in the SCVMM Admin Console, select Migrate.  This you are clustering your Virtual Machines in order to get High-Availability then this changes thing.

To successfully migrate, you now move out of the SCVMM Admin Console and find your new best friend called the Failover Cluster Admin Console.  On your VMM server (if not one of your hosts), you can add this using Server Manager and adding it under the Remote Administration Tools.  You will see your highly available virtual machines listed in the console under Services and Applications, and you can right-click any VM and you will see the following option:

Resolving the Live Migration “Cluster Network is not Available”

Now that you are crystal clear on how to “migrate” between nodes let’s talk about how to troubleshoot a bit when you aren’t successfully able to migrate.  This was the case for me.  The first tool in your toolbox is the trusty Event Viewer which is where all events are targeted in case of failures for Hyper-V and High Reliability.  As shown above, you might see the following error screen in your event viewer -

As you can see, the source is Hyper-V-High-Availability and in this case you will see that the failure occurred for the “designation node” which is important as the error tells you whether it was destination or source.  I’ve blanked the Computer name but in this case I was on the source computer and located this error message thus indicating that it was “unable” locate the destination computer. 

This was troublesome after reviewing the cluster networks configuration as all “looked” well as indicated in this screen shot -

Furthermore, if you highlight the network and look at all the servers in the cluster you see the following -

Um, I learned that this screen might “lie” to you every once in awhile and things are not all happy and content in Cluster Live Migration land.  No fear, a little digging around in the network bindings clears life up a bit.

The following steps cleared up my headaches and by no means are they guaranteed to make your life happy.  I do, though, hope that it does save someone else a lot of time as this took a few hours to dig into and determine what was causing the problem.  At least, what I believe was causing the issue.

There are a lot of folks out there who are unclear on how to change the binding order when you have multiple NICs in your workstation or server.  In my case, I didn’t want the first binding to be the unconfigured VLAN NIC as this would certainly cause problems.  Thus, I went digging and ensured that all seven nodes in the cluster had the exact same binding orders and the first in the list was the domain and cluster management network card.

To do this, you do the following:

1. Click Start, locate Networks, and right-click and select Properties
2. In the Network and Sharing Center, click Change adapter settings
3. Hit the ALT key (yes, that’s it)
4. Click Advanced
5. Click Advanced Settings…
6. In the Adapters and Bindings Tab, ensure your Cluster Network "Connection” is the first listed (see screen shot)

After doing this on the node’s that are failing, the last step was to stop the cluster service on these nodes.  This will force a “Quick Migrate” but not a Live migration so there is a possible outage so do so in your outage window.  Keep in mind for those new to clustering (or those like me who haven’t used it since the late 90’s), the cluster service is stopped not in the Services but in the Failover Cluster Manager.

1. Under Nodes, right-click the Nodes
2. Select More Actions…
3. Click Stop Cluster Service

Give it a minute… If there a lot of VMs on the host, they are all going to Save state and then migrating to the new host.  After several minutes, you can go back in and select Start Cluster Service.  You should now (if they same problem as mine) be able to live Migrate back to the broken hosts.  Happy Clustering of VMs are here again!

Enjoy your Labor Day holiday weekend (if you celebrate!)

-Chris

Dynamic Provisioning for ConfigMgr hits the Shelf – TechNet Magazine Sept 2009

I promised early on to not use “marketing” as a ploy in my blog and trust me this is painful to write.  However, if it wasn’t for the fact that many of the blog posts here are directly related, I decided to go ahead with a little bit of selfish plugging.  In a story format, I recently wrote a TechNet Magazine (September 2009) article entitled Dynamically Provision Configuration Manager Roles using VMM and Operations Manager that focuses on much of the topics I outline here in my blog.  The primary difference is that it is a beginning to end approach and lot less scattered than my blog posts.  For those who have attended my presentation  (MMS, etc.) of the same title, this is the “word” format of that same presentation though you get to see all the pieces and visuals of it working.

The article focuses on the following:
   VMM:  Building your Nucleus
   VMM Server Installation
   VMM Agent Installations
   VMM Library:  The Building Blocks of Dynamic Provisioning
   Don’t Send in the Clones
   Working with Profiles and Templates
   WAIK Up
   From Static to Dynamic
   Test and Verify

This article will give you the baseline to understand the concept of dynamic provisioning of Configuration Manager 2007 client-based roles such as MP’s, DP’s, and SUP’s.  This greatly improves your scale unit and ensures that your infrastructure handles the client users in real time instead of running services all the time costing power and management cycles.

To summarize, let me share the “At a Glance” with you -

   Save time with Unattended Installs
   Understanding VMM Libraries
   Dynamic versus Static Provisioning
   Integrating VMM with Operations Manager

I want to give a shout out thanks to Hector Linares (PM on VMM team) as he was the technical reviewer for the article.  From my team, a thanks to Jason Githens, Catherine Campbell, and Angela Schmeil who reviewed the article and “cleaned it up” for me.

Enjoy!

-Chris

Fighting Hyper-V Physical Host Time Drift – Utilizing NTP to Synchronize Clocks

My team at Microsoft has an engineering lab, one of which, I’ve spent a lot of time designing and building that provides our team some robustness.  We run our own “mini” lab for our engineering purposes that runs very much like a large number of enterprises.  By this, I mean we have our own self-hosted domain, DNS, DHCP, and application services such as ConfigMgr 2007, PKI, IIS, and SQL.  This lab started off running Windows Server 2008 Hyper-V physical hosts (Dell PE2950) that were then upgraded (in-place) to Windows Server 2008 Beta and RC1.  Recently, these physical hosts were cleaned and upgraded to a fresh Server 2008 R2 RTM and we used VMM to migrate our VMs off the hosts prior to doing so.  The upgrade went well though there were some snags that I thought I just might share…

SCENARIO

You have n number of Hyper-V hosts running Windows Server 2008 with various VMs living on them that are “production” like.  Because RTM doesn’t offer Live Migrations, you will fail to be able to migrate VMs off the hosts without some amount of downtime though SAN transfers are much quicker.  When I can’t purchase new hardware but I want to take advantage of R2 features, how might I do this?

ANSWER

Over the next few days, I will spend sometime talking about this very scenario and how I did it as a way to maybe share with folks some coolness…

Kerberos & Time Drift – Like mixing Oil & Water

Issue #1:  I’ve started the migration and I’m starting to get complaints that virtual machines are offline or unavailable.  However, when checking the VMs on the physical hosts they are up and running and seem alive.  They can ping, etc. but no clients can utilize services offered on these machines.

We ran into this issue and the root cause was due to the fact that the Hyper-V hosts time was drifting.  The physical hosts are a member of a domain that is comprised of virtual machine’s that are the DCs which isn’t reality in organizations but due to this they (the DCs) didn’t have good time.  This problem occurs due to the default settings for the integration services that have the virtual machines get their time from the physical hosts where they are running. 

Due to this, you can disable this if you like but not one that I would recommend.

W32TIME SERVICE & AUTHORITIVE TIME SERVER

As many should already know, Kerberos authentication’s documentation for Windows says that a difference of 5 minutes by any two machines who attempt to authenticate using    Kerberos will fail.  The hosts were drifting utilizing the default settings for the W32time service is to use NT5DS (Active Directory).  This causes issues in some cases, as we found out.

This issue can be fixed with these steps on your physical hosts:

1. Open Notepad on the server
2. Copy the following into Notepad -

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time]
"DisplayName"="@%SystemRoot%\\system32\\w32time.dll,-200"
"ImagePath"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\
  74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,73,\
  00,76,00,63,00,68,00,6f,00,73,00,74,00,2e,00,65,00,78,00,65,00,20,00,2d,00,\
  6b,00,20,00,4c,00,6f,00,63,00,61,00,6c,00,53,00,65,00,72,00,76,00,69,00,63,\
  00,65,00,00,00
"Description"="@%SystemRoot%\\system32\\w32time.dll,-201"
"ObjectName"="NT AUTHORITY\\LocalService"
"ErrorControl"=dword:00000001
"Start"=dword:00000003
"Type"=dword:00000020
"ServiceSidType"=dword:00000001
"RequiredPrivileges"=hex(7):53,00,65,00,41,00,75,00,64,00,69,00,74,00,50,00,72,\
  00,69,00,76,00,69,00,6c,00,65,00,67,00,65,00,00,00,53,00,65,00,43,00,68,00,\
  61,00,6e,00,67,00,65,00,4e,00,6f,00,74,00,69,00,66,00,79,00,50,00,72,00,69,\
  00,76,00,69,00,6c,00,65,00,67,00,65,00,00,00,53,00,65,00,43,00,72,00,65,00,\
  61,00,74,00,65,00,47,00,6c,00,6f,00,62,00,61,00,6c,00,50,00,72,00,69,00,76,\
  00,69,00,6c,00,65,00,67,00,65,00,00,00,53,00,65,00,53,00,79,00,73,00,74,00,\
  65,00,6d,00,54,00,69,00,6d,00,65,00,50,00,72,00,69,00,76,00,69,00,6c,00,65,\
  00,67,00,65,00,00,00,00,00
"FailureActions"=hex:80,51,01,00,00,00,00,00,00,00,00,00,03,00,00,00,14,00,00,\
  00,01,00,00,00,60,ea,00,00,01,00,00,00,c0,d4,01,00,00,00,00,00,00,00,00,00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Config]
"FrequencyCorrectRate"=dword:00000004
"PollAdjustFactor"=dword:00000005
"LargePhaseOffset"=dword:02faf080
"SpikeWatchPeriod"=dword:00000384
"LocalClockDispersion"=dword:0000000a
"HoldPeriod"=dword:00000005
"PhaseCorrectRate"=dword:00000001
"UpdateInterval"=dword:00007530
"EventLogFlags"=dword:00000002
"AnnounceFlags"=dword:0000000a
"TimeJumpAuditOffset"=dword:00007080
"MinPollInterval"=dword:0000000a
"MaxPollInterval"=dword:0000000f
"MaxNegPhaseCorrection"=dword:ffffffff
"MaxPosPhaseCorrection"=dword:ffffffff
"MaxAllowedPhaseOffset"=dword:0000012c

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Parameters]
"ServiceDll"=hex(2):25,00,73,00,79,00,73,00,74,00,65,00,6d,00,72,00,6f,00,6f,\
  00,74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,\
  77,00,33,00,32,00,74,00,69,00,6d,00,65,00,2e,00,64,00,6c,00,6c,00,00,00
"ServiceMain"="SvchostEntry_W32Time"
"ServiceDllUnloadOnStop"=dword:00000001
"Type"="NTP"
"NtpServer"="time.windows.com,0x9"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Security]
"Security"=hex:01,00,04,80,84,00,00,00,90,00,00,00,00,00,00,00,14,00,00,00,02,\
  00,70,00,05,00,00,00,00,00,14,00,fd,01,02,00,01,01,00,00,00,00,00,05,12,00,\
  00,00,00,00,18,00,ff,01,0f,00,01,02,00,00,00,00,00,05,20,00,00,00,20,02,00,\
  00,00,00,14,00,8d,01,02,00,01,01,00,00,00,00,00,05,04,00,00,00,00,00,14,00,\
  8d,01,02,00,01,01,00,00,00,00,00,05,06,00,00,00,00,00,14,00,9d,01,02,00,01,\
  01,00,00,00,00,00,05,13,00,00,00,01,01,00,00,00,00,00,05,12,00,00,00,01,01,\
  00,00,00,00,00,05,12,00,00,00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\NtpClient]
"DllName"=hex(2):25,00,73,00,79,00,73,00,74,00,65,00,6d,00,72,00,6f,00,6f,00,\
  74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,77,\
  00,33,00,32,00,74,00,69,00,6d,00,65,00,2e,00,64,00,6c,00,6c,00,00,00
"Enabled"=dword:00000001
"InputProvider"=dword:00000001
"AllowNonstandardModeCombinations"=dword:00000001
"CrossSiteSyncFlags"=dword:00000002
"ResolvePeerBackoffMinutes"=dword:0000000f
"ResolvePeerBackoffMaxTimes"=dword:00000007
"CompatibilityFlags"=dword:80000000
"EventLogFlags"=dword:00000001
"LargeSampleSkew"=dword:00000003
"SpecialPollInterval"=dword:00000900
"SpecialPollTimeRemaining"=hex(7):00,00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\NtpServer]
"DllName"=hex(2):25,00,73,00,79,00,73,00,74,00,65,00,6d,00,72,00,6f,00,6f,00,\
  74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,77,\
  00,33,00,32,00,74,00,69,00,6d,00,65,00,2e,00,64,00,6c,00,6c,00,00,00
"Enabled"=dword:00000001
"InputProvider"=dword:00000000
"AllowNonstandardModeCombinations"=dword:00000001
"EventLogFlags"=dword:00000000
"ChainEntryTimeout"=dword:00000010
"ChainMaxEntries"=dword:00000080
"ChainMaxHostEntries"=dword:00000004
"ChainDisable"=dword:00000000
"ChainLoggingRate"=dword:0000001e

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\VMICTimeProvider]
"Enabled"=dword:00000001
"InputProvider"=dword:00000001
"DllName"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\
  74,00,25,00,5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,76,\
  00,6d,00,69,00,63,00,74,00,69,00,6d,00,65,00,70,00,72,00,6f,00,76,00,69,00,\
  64,00,65,00,72,00,2e,00,64,00,6c,00,6c,00,00,00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\VMICTimeProvider\Parameters]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TriggerInfo]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TriggerInfo\0]
"Type"=dword:00000003
"Action"=dword:00000001
"GUID"=hex:ba,0a,e2,1c,51,98,21,44,94,30,1d,de,b7,66,e8,09

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TriggerInfo\1]
"Type"=dword:00000003
"Action"=dword:00000002
"GUID"=hex:6e,51,af,dd,c2,58,66,48,95,74,c3,b6,15,d4,2e,a1

1. Save it as useNTPTime.reg
2. Double-click on the registry file to import
3. Open a CMD prompt
4. Type net stop w32time && net start W32time

NOTE:  This step modifies the registry.  Please assume all normal precautions such as backing up the registry or exporting the original values as a method to roll back if something drastically fails.

Woo.. Life is good again.

-Chris

Dynamic Provisioning with VMM: Proxy, Windows Updates, and Scripts

In our environment, there are two things that are critical to success of an environment that is dynamically built from scratch – Updates & Internet connectivity.  This might seem odd since most would believe that we would utilize WSUS & the Software Update Points in Configuration Manager to do our patching and truth be known we do.  However, non-compliant servers with specific hot-fixes are not allowed to come on the network and to avoid big delays we do not push the ConfigMgr clients to our servers “immediately.”  Thus, we depend on Windows Update connectivity for our servers and also proxies as we can’t get to the internet without them.

In today’s post, I’m going to share a settings configuration we use in our unattend.xml to ensure that our automated scripts effectively reach the internet and secondly the script itself.  The script is shared as-is and credit goes to Ben Shy & Michael Schmidt on my team for the actual building of the script though I’m consuming it in my design of dynamic provisioning.

Setting Proxy Settings to work during Dynamic Provisioning Servers

In some situations, you will have the following settings in your unattend.xml file though upon completion you will not have any proxy settings set.  This is very troublesome and I couldn’t locate much of any data or information regarding how to correct this on the internet and had to use internal resources to troubleshoot (Thanks Eric!).  For the longest time, I had the following settings in my unattend XML:

This is all that is discussed in the WAIK unattended help file and thus should just work, right?  Wrong!  Unfortunately, there was a little unknown (to me) key called ProxySettingsPerUser that needed to get set though this wasn’t outlined anywhere in the documentation.  I can’t, unfortunately, provide you a lot of insight as to what is so important other than what is covered in TechNet.  It basically changes the Windows behavior where proxy settings are per machine rather than per user.  However, I had no idea this was needed nor that I could include it in my unattend.xml file.  However, with a little digging, some help, and testing we found that it was absolutely possible to include this in the WAIK’s unattend.xml and it would be honored.  Thus, my working XML is the following that sets my proxy settings so that scripts running on the server but not as a user would honor the proxy settings:

After testing, the script ran successfully and completely thus ensuring that our virtual machines were compliant quickly after they hit the network.

Scripting Critical Updates for Windows Server from Windows Update

In this section, we will focus on the script itself that does the work of connecting to Windows update, providing the appropriate client-based information, and installing the updates.  The first thing to note that is critical is the error message received when you do not have a viable connection to the internet.

Using Windows Update vs. Microsoft Update:  That is the question…

A great number of folks don’t quite know the reason that Microsoft has two viable update engines, Windows (WU) and Microsoft update (MU).  The former focuses only on the Windows platform and will only serve the core Windows operating system binaries.  The latter, though, is an opt-in service, that offers the ability for servers to get updated and patched that are not only running Windows but additional Microsoft software such as Office, etc.  In order for the Microsoft update engine to work, the Windows update agent client is replaced with the Microsoft update agent in Windows Vista & above.  For older operating systems, a different ActiveX control is installed in Internet Explorer.

For our environment, we wanted to ensure that we downloaded only critical updates from Windows Update at the time of server creation.  In order to do this, we utilized the following sample script:

Using Microsoft Update instead of Windows Update

In the above script, the updateSearcher.ServerSelection option can be modified to instead use Microsoft update.  This depends on your scenario and your goal.  For our environment, we didn’t need to focus on Microsoft update rather just Windows update.  For example, if i set server selection to 1 then I would get a ton of Microsoft updates including items like Windows Live Essentials (ouch!) and other bogus updates.  I corrected this behavior changing the value for ServerSelection to 2 which uses Windows Update and only gets Critical Updates.

Summary

In this post, I shared how I modified our unattend.xml file that is used with our provisioning of new virtual machines to ensure that the servers not only were ready to serve clients but also they were correctly patched.  This is very important for the security of the network and your sanity.  In order to do this, I used a unknown setting in my unattend.xml file that is called PolicyProxySettingsPerUser and set this value to 0 that forced the proxy settings per machine.

Lastly, I shared a sample script managed and maintained by my team for patching clients using MU and I altered it to use WU instead. 

Enjoy!

-Chris

Configuring Servers for deployment as a Configuration Manager Distribution Point (DP)

In order to go from nothing to a finished, completed installation of a System Center Configuration Manager (ConfigMgr) client services role such as a Distribution Point (DP) requires that an administrator configures the future DP server automatically.  This includes a few key details that, often are missed, I will outline in today’s post.

Granting Central\Primary Site Server Administrator Privileges

Before you can even get started, the first step is to ensure that your server’s Primary Site Server has administrative privileges otherwise it will fail to install.  This is extremely simple to do using Windows built-in command net localgroup.

The command, net localgroup administrators /add {Site Server Name}, is easy to call as part of your SCVMM Guest OS profile GUIRunOnce action used when you auto logon.  This command should work on Windows 2003 & Windows 2008 servers.

Configuring Server with IIS for a Configuration Manager Deployment

The next step is to ensure that you get the pre-requiristies laid down properly so that when the server comes online clients will have everything necessary to get serviced.  In this case, a DP requires IIS and as such you will need to configure your server to have IIS installed and prepared.

To review, the following requirements are needed to be a ConfigMgr DP:

In order to be successful in automation, it is required that the server administrator automated this so let’s look at how to do this successfully.

Install IIS

This functionality is rather easy using the built-in Server Manager command-line interface (ServerManagerCmd) in Windows Server 2008.  To successfully install all features of IIS (we will not focus today on doing a scaled back install), use the following command:

Command Line:  ServerManagerCmd –install Web-Server –allSubFeatures

Install BITS (Background Intelligent Transfer Services)

As mentioned earlier, BITS is a feature that requires IIS so you should install IIS first, BITS second.  Using, again, the same command ServerManagerCMD interface allows you to install both roles and features.  To install BITS, use the following command:

Command Line:  ServerManagerCmd –install BITS

Install WebDAV

The downside here is that you will need to obtain WebDAV externally as it isn’t shipped in the Windows Server 2008 CD and instead was pushed as a download on the Microsoft Download Center.  The WebDAV installer is your standard MSI which means to “automate” you have to use msiexec.exe that is built-in to Windows.

The command to install WebDAV silently and without interaction is the following:

Command Line:  msiexec /qna /i webDav_x86.msi

Configuring IIS for DPs

A final step that

In the following command/batch file (no error handling at all!) we use the AppCmd command-line tool to configure IIS7’s Default Web Site WebDAV configuration.  You can simply copy this code and place into a batch file and then call it via a command-line.  The command-line used in automation is cmd /c ConfigureDP.bat.

   1: c:\windows\system32\inetsrv\AppCmd set config "Default Web Site/" /section:system.webServer/webdav/authoring /enabled:true /commit:apphost

   2:  

   3: :\windows\system32\inetsrv\AppCmd set config "Default Web Site/" /section:system.webServer/webdav/authoringRules /+[users='*',path='*',access='Read'] /commit:apphost", 0, true

   4:  

   5: :\windows\system32\inetsrv\AppCmd set config "Default Web Site/" /section:system.webServer/webdav/authoring /properties.allowAnonymousPropfind:true /commit:apphost

   6:  

   7: :\windows\system32\inetsrv\AppCmd set config "Default Web Site/" /section:system.webServer/webdav/authoring /properties.allowCustomProperties:false /commit:apphost

   8:  

   9: :\windows\system32\inetsrv\AppCmd set config "Default Web Site/" /section:system.webServer/webdav/authoring /properties.allowInfinitePropfindDepth:true /commit:apphost

  10:  

  11: :\windows\system32\inetsrv\AppCmd set config "Default Web Site/" /section:system.webServer/webdav/authoring /fileSystem.allowHiddenFiles:true /commit:apphost

  12:  

  13: :\windows\system32\inetsrv\AppCmd set config "Default Web Site/" /section:system.webServer/webdav/authoring /fileSystem.allowHiddenFiles:true /commit:apphost

  14:  

I would recommend that you add some error handling.

Summary

In this post, I quickly shared with you how to automate taking a sysprep’d machine to now be a Web Server with the required features for a System Center Configuration Manager DP.  These steps could easily get defined via a single batch file which is the point but for now this is broken out to single steps.

Enjoy!

-Chris

Building a “Sample” SCOM Management Pack for Dynamically Building Virtual Machines based on Performance of physical or virtual machines

In today’s post, I will focus on how I worked with our System Center Operations Manager (SCOM) team to build the actual infrastructure and setup alerting for our infrastructure to ensure that if our Configuration Manager Distribution Points (DPs) are getting a high number of connections, we will spin up additional virtual machines to help with the load.

For the most part, there are some requirements that should be in place for this post to be useful:

    Virtual Machine Manager 2008 should be present in your environment
    SCVMM Library has the appropriate Hardware & Guest OS Profiles built along with a template

If you have the above requirements, then you will be able to follow this blog to expand your infrastructure with SCOM 2007 to do active monitoring of your client services and “dynamically” respond to their performance by building additional infrastructure to remove the load.

Example Infrastructure Design for SCOM 2007 (& R2)

First off, I have to give a shout out to my resident SCOM experts – Duncan Ngarachu, JJ Lindner, and Ivan Ivanov – who guided me through getting a better understanding of SCOM.  I knew what I wanted it to do but I wasn’t sure how to articulate how to get SCOM to do it.  Big shout out…

With that said, let’s explain a small-scale design for our SCOM infrastructure for a bit of understanding.  The primary thing that most folks realize is that ConfigMgr infrastructures depend heavily on Active Directory (AD) and this makes the requirement that SCOM has this ability to cross forests where applicable.  For our environment, we have our System Center Virtual Machine Manager (SCVMM) infrastructure & SCOM in a privately hosted domain that has a one-way trust to our service infrastructure.  Because of this, we have to rely on two management servers – one Root Management Server (RMS) and another that is a Gateway server hosted in our services infrastructure domain.

In this scenario, you use Machine Authentication between the SCOM Gateway and SCOM RMS server so you can manage services that exist in the domains where your users getting manage exist.  For the Management Infrastructure, where SCOM & SCVMM servers live, you manage all of your other services.  I will post a blog a bit later that outlines exactly how to setup PKI correctly to support SCOM Gateway as I have found most documentation to not be completely accurate.

NOTE:  You must have the SCVMM and SCOM servers in the same forest or have two-way trusts.  You will also need to ensure that you have a service account in each domain with privileges to take actions in the Services Forest.  You will also need to get SCOM & SCVMM integrated per the following post from Cheng’s Blog.

Downloading & Importing the Management Packs

After you get your SCOM infrastructure planned out and you have the correct infrastructure in place, you will need to move to the next step of downloading and importing your management packs.  For our purposes, I am going to share the management packs that we have installed so that you can take the sample management pack and alter as you see fit.  For now, we have the following management packs as a requirement to get dynamic provisioning working:

You will notice that we don’t include things like SCVMM, etc. and this is on purpose.  We could have built the management pack to be dependant on the SCVMM class library though every time we would like to modify we would have to “revision” and re-import.  This seemed costly so we included bare miminum though the ones listed above aren’t completely bare either.  However, you need them in order to successfully import.  (This happens when the author – Ivan Ivanov – has these as the defaults for his authoring console and I don’t have the knowledge yet to figure out how to strip them out).

Installing SCOM agents on your Service Infrastructure Servers

The last step is to get agents distributed to your infrastructure pieces you would like to monitor.  SCOM has the ability to extend AD to allow auto deployment but for the purposes of this post I’m going to consider this out of scope.  There is also the ability to discover windows servers and install the agents manually and I will step through this process today.

At minimum, you should install agents on your Virtual Machine Manager server as well as any of your ConfigMgr infrastructure.  This is pre-req to ensuring that you have what you need to get these up and running.

To Install agents manually on your server, do the following:

1. Open the SCOM Administrators Console
2. Click Administration
3. Right-click on Agent Managed, select Discovery Wizard
4. Select Windows Computers from the device to manage wizard
5. Click Next
6. Click either Automatic discovery to use Active Directory or advanced to select a specific type of client or server
7. Click the Domain, and select it
8. For the Administrator account, keep the default to Use selected Management Server Action Account
9. Click Discover
10. Select the object and then the agent will install

For more information on installing & discovering servers & clients, see the following KB.

Sample Custom Management Pack for System Center Configuration Manager DPs

After installing your agents, you will notice that all of the pre-defined monitoring groups setup by the management packs for Configuration Manager show no objects (e.g. servers) in your infrastructure.  This is often the case as the discovery logic is broken, I have found, for many of the groups because servers are missing the appropriate registry and\or file settings to get discovered.  You can manually fix this by using SCOM to determine what the discover logic is or you can do your own detection.  For our purposes, we created our own discovery logic that is based on a registry key.

On each of of our DP infrastructure pieces, we have added the following registry key for detection:

Path:  HKLM\System\Software\Microsoft

Key:  MPSD (or Org Name)

String Name:  RedmondDP (stands for DPs in Redmond but could be anything)

String Value:  RedmondDP

For our management pack, we have one called DynamicDP that does a discovery of any servers that have this registry key and if detected will then assume it is a DP.  The same logic can be used for a DynamicMP or DynamicSUP with a different management pack which is what we do.

We have some logic setup whereby we create monitors and alerts based on Web Service Concurrent Connections (e.g. the total number of connections open currently on the Web server which is what a DP is).  In this section, we create an alert that says the following:

As I said, this is a sample but to give you a blueprint I’m going to share out our sample Management Pack XML which you should be able to open in the Authoring Console and modify to your liking.

Configuring Recovery Tasks for SCOM’s Root Management Server

The “typical’ design of a management pack is to take “recovery” actions against the target server whose state is critical.  For example, if your Web server is sitting at 100% CPU you would take a recovery action against that server such as recycle the application pool.  The target in this case is the server who through the alert.

However, in the case of scaling your infrastructure, the target completely changes as you are interested in using the “state” of a machine and asking it to do nothing.  For example, I want my Web server who is sitting at 100% CPU to do nothing other than go unhealthy so I can take action.  This is a bit trickier overall.

In this case, you would use SCOM’s Channel, Subscriptions, and Notifications to setup the alert to take action through a command action.  This also could be email, text message, or IM you when the problem occurs.  The extreme action is to spin up additional servers to help support the load that is currently in place.

How do I do this?  Glad you asked…

Create Channel

The following step will create our appropriate recovery action to initiate the recovery steps when our target servers get in a state which we don’t desire such as Critical.  To do this, do the following:

NOTE:  To perform these actions, you will need the appropriate SCOM privileges setup

1. Open the SCOM Administrators Console
2. Click the Administration Tab
3. Click Channels under Notifications
4. Under New, select Command
5. In the Command Notification wizard, input the Channel Name & Description
6. Click Next
7. For the Notification Channel, input the command to execute (e.g. dp_recovery.cmd), select Alert Source (this isn’t used in our scenario),  and the startup folder (the command-line parameters are optional)
8. Click Finish

This will successfully create the channel for us that initiates the command-line action.  It uses a local path because if you remember that we know the SCOM server will take all the actions so we can safely assume that the directory is highly available.

Add Subscriber

The purpose of the Subscriber is to bind the “who” to the what which is the channel.  So we need to add a active subscriber to our infrastructure. In our case, the subscriber is the SCVMM server for our infrastructure.

To setup a subscriber, do the following:

1. Open the SCOM Administrators Console
2. Click the Administrators tab
3. Click Subscriber under Notification
4. Right click, select New to open the New Subscriber Wizard
5. Input the Subscriber name
6. Select to always send notifications (this is a nice thing to have where you could scope it to a schedule if you needed too)
7. Click the Add on the Notification Subscriber Wizard which starts an additional wizard
8. In the Describe the Subscriber Address, input the address name such as SCVMM 
9. Let’s then provide the Channel & Delivery Address to use (delivery address isn’t really used)
10. Click the Command Channel drop down and select DP Recovery (or whatever you named it)
11. For the Delivery address, input address
12. Click Next
13. Select Always Send Notifications for scheduling
14. Click Finish

The completed setup for this section should result in something similar to the following:

Create Subscription

The next, and last, step is to create a subscription.  In this section I will walk you through creating your subscription so that you have completed the process of creating the channel, subscriber, and subscription.  The following are the steps to create the subscription:

1. Open the SCOM Administrator’s Console
2. Click the Administration Tab
3. Click the Subscriptions under Notifications
4. Right-click and select New to open the Notification Subscription Wizard
5. Enter a subscription name such a Redmond DP and a Description
6. In the criteria selection, select the following:
7. Click Next
8. To search, you would select the following based on the name of your MP:
9. In the subscriber search, select the available subscribers created above:
10. Highlight and click Add
11. Click Add on the Channels wizard, and then click Search
12. Click Finish

That’s it.  You now have your SCOM wired up to monitor your DP, and based on criteria you created, take an action that would include the opportunity to build you a new DP.

NOTE:  Your SCOM server’s service account is used for the creation of your virtual machine.  In order for this to work, you will need to add your SCOM service account to the administrator’s role in SCVMM.  You can also add additional accounts for notification if you prefer to specify a different account is used.

Summary

What we focused on today and how we tied together the two pieces of our infrastructure – SCVMM & SCOM and focused heavily on how to correctly set these up so that you can “scale” on demand.  The key piece of this puzzle is the management pack which we have built along with several other posts I have made that make up the building blocks of this entire principle - “scale on demand” – not after it is too late.  For us, we focused on how to build a “typical” SCOM infrastructure, including installing agents, and how to effectively get the management packs imported so you can effectively use your custom management pack since we are dependent on them.

I will start soon pulling this all together so that you can see what “dynamic provisioning” using System Center is all about!

Enjoy!

-Chris