Dave's blog

The Case of the Unexplained Windows Server 2012 Replica Kerberos Error : 0x8009030C 0x00002EFE

2013-02-07T02:50:00Z

Windows Server 2012 Hyper-V Replica is a new feature that replicates all changes on a virtual machine to a counterpart virtual machine hosted by a different server. Replica overview and deployment considerations can be found here. This feature is well documented so I'll skip setup and installation and strictly focus on troubleshooting. I received a call from one of my customers who was trying to evaluate this feature for a disaster recovery scenario in test. Unfortunately, any time he attempted to enable this feature an exception was raised. I'd like to share some of the troubleshooting steps we worked though to resolve this case. You may not have this exact issue but the troubleshooting steps should still be applicable.

The customer environment was fairly straightforward, each Hyper-V host was a domain member and for testing purposes the physical machines themselves were actually located in the same datacenter with gigabit connectivity. Since this feature was new to me, I started out by first setting it up from scratch in a test environment using commodity hardware which consisted of a DC and two member servers. Armed with this new knowledge and walking through deployment guide above I felt more confident continuing in the investigation.

Domain - REPLICADOMAIN

DC - REPLICADC

Hyper-V Hosts - REPLICAHOST1 (Primary), REPLICAHOST2 (Replica)

In my customer environment, any attempt to enable replication for a VM returned the following error, 100% repro. Replica uses mutual authentication based on Kerberos or mutual authentication based on certificates for the server (not for the user).

First place to start is reviewing Windows event logs for items of interest. I reviewed the following logs on each machine:

System, Application, Security and Microsoft-Windows-Hyper-V-VMMS. In the latter, I found:

02/04/2013 01:12:06 PM Error ITSUSRALAB039.df 32000 Microsoft-Windows-Hyper-V-VMMS N/A NT AUTHORITY\SYSTEM Hyper-V failed to enable replication for virtual machine 'YOURVMNAME': The connection with the server was terminated abnormally (0x00002EFE).

02/01/2013 03:35:17 PM Error ITSUSRALAB039.df 29212 Microsoft-Windows-Hyper-V-VMMS N/A NT AUTHORITY\SYSTEM Hyper-V failed to authenticate the primary server using Kerberos authentication. Error: The logon attempt failed (0x8009030C)

You can use the Microsoft Exchange Server Error Code Look-up tool called ERR.exe to help convert the error codes to text.

err 0x00002EFE
# for hex 0x2efe / decimal 12030
ERROR_INTERNET_CONNECTION_ABORTED                              inetmsg.h
ERROR_WINHTTP_CONNECTION_ERROR                                    winhttp.h
ERROR_INTERNET_CONNECTION_ABORTED                              wininet.h
# 3 matches found for "0x00002EFE"

err 0x8009030C
# for hex 0x8009030c / decimal -2146893044
SEC_E_LOGON_DENIED winerror.h

The second one was a great tip but unfortunately I didn't put it all together..... so we did more troubleshooting.

The deployment guide provides specific guidance on what is required for Kerberos delegation and this seemed like a reasonable place to start as well as verifying that all the SPNs are registered. There are a couple ways to verify this information. One would be to use ldifde.exe to dump all the configuration settings of the computer object in Active Directory. The command using ldifde.exe would this:

[Each command would need to be executed against each Hyper-V hosts computer account]

C:\>ldifde -d "CN=REPLICAHOST1,CN=Computers,DC=REPLICADOMAIN,DC=com" -f REPLICAHOST1.log
Connecting to "REPLICADC.REPLICADOMAIN.com"
Logging in as current user using SSPI
Exporting directory to file output.log
Searching for entries...
Writing out entries..
2 entries exported

The command has completed successfully

To retrieve the SPN's for each computer object we can leverage setspn.exe, the command would be:

C:\>setspn -L REPLICADOMAIN\REPLICAHOST1

Registered ServicePrincipalNames for CN=REPLICAHOST1,CN=Computers,DC=REPLICADOMAIN,DC=com:

Hyper-V Replica Service/REPLICAHOST1
Hyper-V Replica Service/REPLICAHOST1.REPLICADOMAIN.com
Microsoft Virtual System Migration Service/REPLICAHOST1
Microsoft Virtual System Migration Service/REPLICAHOST1.REPLICADOMAIN.com
Microsoft Virtual Console Service/REPLICAHOST1
Microsoft Virtual Console Service/REPLICAHOST1.REPLICADOMAIN.com
WSMAN/REPLICAHOST1
WSMAN/REPLICAHOST1.REPLICADOMAIN.com
TERMSRV/REPLICAHOST1
TERMSRV/REPLICAHOST1.REPLICADOMAIN.com
HOST/REPLICAHOST1
HOST/REPLICAHOST1.REPLICADOMAIN.com

Next we need to verify that the computer objects have Kerberos delegated correctly:

Verify each Hyper-V host can successfully communicate with Active Directory:

C:\>nltest /SC_VERIFY:REPLICADOMAIN
Flags: b0 HAS_IP HAS_TIMESERV
Trusted DC Name \\REPLICADC.REPLICADOMAIN.com
Trusted DC Connection Status Status = 0 0x0 NERR_Success
Trust Verification Status = 0 0x0 NERR_Success
The command completed successfully

Verify that KDC is assigning Kerberos tickets:

C:>klist

Current LogonId is 0:0x3364e

Cached Tickets: (7)

#0> Client: Administrator @ REPLICADOMAIN.COM
Server: krbtgt/REPLICADOMAIN.COM @ REPLICADOMAIN.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x60a10000 -> forwardable forwarded renewable pre_authent name_canonicalize
Start Time: 2/6/2013 15:38:20 (local)
End Time: 2/7/2013 0:36:21 (local)
Renew Time: 2/13/2013 14:36:21 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0x2 -> DELEGATION
Kdc Called: REPLICADC.REPLICADOMAIN.com

#1> Client: Administrator @ REPLICADOMAIN.COM
Server: krbtgt/REPLICADOMAIN.COM @ REPLICADOMAIN.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
Start Time: 2/6/2013 14:36:21 (local)
End Time: 2/7/2013 0:36:21 (local)
Renew Time: 2/13/2013 14:36:21 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0x1 -> PRIMARY
Kdc Called: REPLICADC.REPLICADOMAIN.com

#2> Client: Administrator @ REPLICADOMAIN.COM
Server: cifs/REPLICADC.REPLICADOMAIN.com @ REPLICADOMAIN.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize
Start Time: 2/6/2013 15:38:20 (local)
End Time: 2/7/2013 0:36:21 (local)
Renew Time: 2/13/2013 14:36:21 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: REPLICADC.REPLICADOMAIN.com

#3> Client: Administrator @ REPLICADOMAIN.COM
Server: ldap/REPLICADC.REPLICADOMAIN.com @ REPLICADOMAIN.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize
Start Time: 2/6/2013 15:38:20 (local)
End Time: 2/7/2013 0:36:21 (local)
Renew Time: 2/13/2013 14:36:21 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: REPLICADC.REPLICADOMAIN.com

#4> Client: Administrator @ REPLICADOMAIN.COM
Server: RPCSS/REPLICAHOST2 @ REPLICADOMAIN.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
Start Time: 2/6/2013 14:39:01 (local)
End Time: 2/7/2013 0:36:21 (local)
Renew Time: 2/13/2013 14:36:21 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: REPLICADC.REPLICADOMAIN.com

#5> Client: Administrator @ REPLICADOMAIN.COM
Server: LDAP/REPLICADC.REPLICADOMAIN.com/REPLICADOMAIN.com @ REPLICADOMAIN.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x40a50000 -> forwardable renewable pre_authent ok_as_delegate name_canonicalize
Start Time: 2/6/2013 14:36:53 (local)
End Time: 2/7/2013 0:36:21 (local)
Renew Time: 2/13/2013 14:36:21 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: REPLICADC.REPLICADOMAIN.com

#6> Client: Administrator @ REPLICADOMAIN.COM
Server: RPCSS/REPLICAHOST2.REPLICADOMAIN.com @ REPLICADOMAIN.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
Start Time: 2/6/2013 14:36:21 (local)
End Time: 2/7/2013 0:36:21 (local)
Renew Time: 2/13/2013 14:36:21 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: REPLICADC.REPLICADOMAIN.com

So far everything looks good, no indications of a Kerberos problem (darn it!!). At this point, we moved on to running a network trace from Primary and Replica server simultaneously. In previous versions of Windows, we would need to install Netmon or Wireshark to collect this information. A really great trick is leveraging ETW providers to collect the trace.

[Commands to start trace]

ipconfig /flushdns
klist purge
netsh trace start capture = yes

*repro issue*

[Commands to stop trace]

netsh trace stop

You can then copy over the ETL trace files to your local machine and view with Netmon. Unfortunately, the network trace revealed no clues on where to look next.

When in doubt, always turn to Process Monitor and collect a trace on both sides, I didn't find any Access_Denied or anything else of interest.

At this point, we decided to refocus on the error itself, Kerberos. I enabled Kerberos debug logging on each Hyper-V machine, reproduced the issue and then stopped trace. Here is the command:

[Start the trace]
Logman.exe start kerb -p "Security: Kerberos Authentication" 0x40043 -o .\kerb.etl -ets

Run through the Hyper-V wizard and stop traces immediately when exception is raised.

[Stop the tace]
Logman.exe stop kerb –ets

Rename Kerb.etl to include the respective server name from both servers. Unfortunately, in order to convert this trace you will need to engage CTS Support. My trace revealed a clue which really helped:

Failed to create token: 0xc000015b. Let's convert that:

err.exe 0xc000015b

STATUS_LOGON_TYPE_NOT_GRANTED

To me this has more to do with user rights security rather than Kerberos security. In comparing user right assignments between two environments, I noticed a difference which would lead to a break in the case. From the Local Group Policy Editor, here are the differences:

[Working - Windows Default]

[Broken - Customer environment]

Now we are getting somewhere! So if this is a security issue, shouldn't an event have been written to Security log which I reviewed in the beginning? Sigh... Yes. I simple missed it because I only looked at the Security Log on the Primary machine not the Replica. Going back to the Security Log I found the following event on the Replica Server:

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          2/6/2013 12:02:01 PM
Event ID:      4625
Task Category: Logon
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      REPLICAHOST2.REPLICADOMAIN.com
Description:
An account failed to log on.

Subject:
Security ID:  NULL SID
Account Name:  -
Account Domain:  -
Logon ID:  0x0

Logon Type: 3

Account For Which Logon Failed:
Security ID:  NULL SID
Account Name:  REPLICAHOST1$
Account Domain:  REPLICADOMAIN.COM

Failure Information:
Failure Reason:  The user has not been granted the requested logon type at this machine.
Status:   0xC000015B
Sub Status:  0x0

Process Information:
Caller Process ID: 0x0
Caller Process Name: -

Network Information:
Workstation Name: -
Source Network Address: -
Source Port: -

Detailed Authentication Information:
Logon Process: Kerberos
Authentication Package: Kerberos
Transited Services: -
Package Name (NTLM only): -
Key Length: 0

This particular customer has a requirement to harden Windows security where possible so instead of simply adding back in all the Windows defaults we granted "Authenticated Users" group the following user right "Access this computer from the network." to both machines which resolved issue!

Summary:

This was a great reminder for me to always take the necessary time to review all of the event logs before engaging in advanced troubleshooting. I hope some of the troubleshooting steps we used may be helpful for your future investigations.

Windows Server 2012 Continuous Availability File Server feature

2012-10-22T00:57:00Z

General availability of Windows Server 2012 and Windows 8 is almost here! As a result, I have been slowly absorbing all of the new features so I can better support my customers. Continuous Availability File Server is my favorite feature, so I wanted to take a few minutes to make the case to highligh its importance and potential value. When I think back and consider some of the more difficult cases I have worked on over the years many them are intermittent in nature. One such scenario often was related to highly available file shares on Windows Failover Clusters. The idea is if your business has a network file server the business has defined as mission critical, deploying a solution to optimize uptime and reliability certainly makes sense. Deployment of a stand alone file server does not meet those requirements as a simple hardware failure causes those resources to become unavailable. Back in Windows 2000 (yes, believe it or not I still get calls on it) the product team made Highly Available File Shares to address this issue. The basic idea is cluster creates a Virtual Network Name and IP address and uses a shared disk presented to both nodes so resources can be hosted on either node providing fault tolerance. If Node 1 owns the file share resource group and reboots for patching, the resource fails over to Node 2 within seconds providing highly available solution.

Notice my statement "highly available", I didn't say fault tolerant. You are still getting 99% uptime, but the reality is...your client had an outage. When the group moves the file share, depending on the application, customer may experience an outage. The cluster service, in order to provide high availability, routinely checks for the health of the file share to ensure online status. How does it do that? It routinely performs a directory listing of the file share itself to ensure it's alive and well. There are many reasons why this process may fail, not limited to memory pressure, server service load, file or system filter drivers such as A/V, list goes on and on. Typical errors may look like the following:

Error Messages: ClusSvc Event ID:1055
Cluster File Share resource ‘demo’ has failed a status check. The error code is 64.

00000888.00001108::2009/08/04-02:16:09.021 INFO File Share <DEMO>: Retrying FindFirstFile on error 64 for share \\HAFileShare\demo\*.* !

ERR 64 = The specified network name is no longer available.

The question in my mind is “Does Windows Server 2012 offer a solution to this problem?” The answer is obvious; otherwise I wouldn’t have written this blog ;)

One of the primary design goals the product team had for Windows Server 2012 was to enable workloads such as SQL Server or Hyper-V to use SMB File Server back end storage as primary storage for datacenter scenarios. So what does SQL Server and Hyper-V have to do with the scenario above? Well, part of the framework required for their objective included a major overhaul of SMB to version 3 and to include “transparent failover”. Windows Server 2012 offers two file server types, the one we all know and love now called “Classic” file server and "scale-out" file server to support workloads above. The “classic” file server role provides the ability to failover a file share resource with zero downtime and is the scenario we have been discussing. Scale-out file server is a different discussion for another blog.

I know what you’re thinking…this is awesome, all I have to do is migrate by backend file server to Windows Server 2012 and I’m golden. After all, you’re more than half way complete with Windows XP refresh project to Windows 7. Well… No. This technology, at its core, depends on SMB3 which is included only in Windows Server 2012 and Windows 8. Therefore, you must be running those SKU’s to take advantage of this feature.

For me, seeing is believing, so let’s move from the academic discussion into real world application and testing. My lab consists of five machines:

Windows Domain controller (iSCSI target for lab scenario)
Two Node Windows Server 2012 Cluster with shared storage using iSCSI and “Classic File Share” configured with continuous availability.
One XP Client
One Windows 8 Client

Test Scenario:
Create continuous available file share and initiate large file transfer from client. During copying process, move resource group containing file share to other node and note any disruptions. We know on XP client, any disruption in availability of file share will immediately return error to user. How will Windows 8 fare?

[Add file share role to each cluster node]

[Select file server for general use, scale-out is for Hyper-V or SQL only]

[Create client access point, the virtual name your clients will use to connect]

[Select a disk resource to host file share]

[Complete wizard]

[HAFileShare is my CAP and it’s online owned by Node 1]

[Create file share for continuous availability]

[Make a selection, notice the other settings item pending]

[Define share name]

[Default option foe continuous availability is selected]

[Demo Share is online]

[Initiate large file transfer from client to file server on XP, move cluster group containing file share during copy process]

[Network trace snippet]
1832 1:37:42 PM 10/21/2012 17.3213007 System HAFILESHARE SMB SMB:C; Nt Create Andx, FileName = \largefile.iso {SMB:639, SMBOverTCP:19, TCP:18, IPv4:17}
245 1:37:30 PM 10/21/2012 5.6650507 System HAFILESHARE SMB SMB:R; Nt Create Andx - NT Status: System - Error, Code = (52) STATUS_OBJECT_NAME_NOT_FOUND {SMB:30, SMBOverTCP:19, TCP:18, IPv4:17}
287 1:37:30 PM 10/21/2012 5.6650507 System HAFILESHARE SMB SMB:R; Nt Create Andx - NT Status: System - Error, Code = (172) STATUS_PIPE_NOT_AVAILABLE {SMB:39, SMBOverTCP:19, TCP:18, IPv4:17}

[Initiate large file transfer from client to file server on Windows 8, move cluster group containing file share during copy process. Notice the temporary dip in transfer rate but no exception, copy completes]

[Event logs on the Windows 8 machine]

Log Name:      Microsoft-Windows-SMBClient/Operational
Source:        Microsoft-Windows-SMBClient
Event ID:      30624
Level:         Information
User:          SYSTEM
Description:
Connection to share \hafileshare\demo was re-established.

Log Name:      Microsoft-Windows-SMBClient/Operational
Source:        Microsoft-Windows-SMBClient
Event ID:      30623
Level:         Warning
Description:
Connection to share \hafileshare\demo was lost. Status 0xC000020C

Summary:
Continuous available file share is a great new feature and I hope you will evaluate Windows Server 2012 and Windows 8 with the common goal to provide your business partner with highly available and fault tolerant services.

The Case of the ADAM/AD LDS error A referral was returned from the server

2012-02-24T19:23:00Z

My customer uses ADAM/AD LDS for directory aware applications instead up extending schema for Active Directory Domain Services. In short, they prefer to isolate line of business or 3rd party applications in their own sandbox and leave Active Directory schema alone when possible. The definition for ADAM\AD LDS from technet is "Active Directory Application Mode (ADAM) is a part of Microsoft’s fully integrated directory services available with Windows Server 2003, and is built specifically to address directory-enabled application scenarios. ADAM runs as a non-operating-system service, and, as such, it does not require deployment on a domain controller. Running as a non-operating-system service means that multiple instances of ADAM can run concurrently on a single server, and each instance can be configured independently."

In the past for their business, each application required a single application partition. As a result, their standard procedure included provisioning a new ADAM instance with a single application partition. The problem started when the requirements slightly changed. The new line of business application required two application partitions not one. They used their existing procedures for setting up one application partition for two application partitions scenario. Lab certification was failing due to intermittant errors in lab with error "0000202B: RefErr: DSID-03100697 A referral was returned from the server." when attempting to write to second application partition. They could not recreate the error when writing to first partition in a consistent manner.

For clarity let's walk through lab setup. Let's assume for the moment we have a domain called ACME.COM and our ADAM servers are called ADAM1.ACME.COM and ADAM2.ACME.COM. Let's create our ADAM instance and call it something simple and straight foward like "BlogInstance". Additionally, we can call our application partitions "BlogPartition1" and "BlogPartition2". Here are a few screen shots for initial setup to create the partition and first partition from which we can leverage the ADAM wizard.

[Use wizard to create ADAM instance]

[Specify ports]

[Create and name the application partition, first one of course]

[Import in box ldif files to create classes etc.]

OK. For brevity, I'll skip the rest of the wizards, next button throughout. So now in services on ADAM1.ACME.COM we can verify that new NT Service is installed.

As I mentioned, process is very important as it should be to the customer. Next step is to add ADAM2.ACME.COM to replica. From ADAM2.ACME.COM, let's run the ADAM wizard and create replica of existing instance.

[From ADAM2]

[Notice the application partitions available to copy, we have only created one and therefore this is only one presented]

At this point, the wizard completes successfully and we verify replication between ADAM1.ACME.COM and ADAM2.ACME.COM is working using command such as C:\windows\adam>repadmin /syncall ADAM1:389. So far so good. Next step which is new territory includes creation of second application partition for instance "BlogInstance". Let's say the second application partition is called "BlogPartition2" and is created on ADAM1.ACME.COM.

If you need to create the second partition for the application you can use dsmgmt.exe to accomplish task.

[Create second partition called BlogPartition2]

So now we have the same setup as the customer. Sure enough, if I try to create an object in the second partition from ADAM2.ACME.COM I get a referral error. Being able to reproduce the problem is huge in finding a resolution. Through research and collaboration with teammates an attribute, msDS-NC-Replica-Locations, was identified which could explain issue. This attribute was populated with both ADAM1.ACME.COM and ADAM2.ACME.COM for first partition where no problems were encountered. However, this same attribute for the second partition only listed one server where second partition was originally created, in our case ADAM1.ACME.COM. I think we are on to something here as our referral error goes away when executed from server that is populated in attribute msDS-NC-Replica-Locations.We need to understand how and why this value gets populated.

From MSDN:

2.194 Attribute msDS-NC-Replica-Locations

This attribute specifies a list of servers that are the replica set for the corresponding non-domain naming context.

cn: ms-DS-NC-Replica-Locations
ldapDisplayName: msDS-NC-Replica-Locations
attributeId: 1.2.840.113556.1.4.1661
attributeSyntax: 2.5.5.1
omSyntax: 127
omObjectClass: 1.3.12.2.1011.28.0.714
isSingleValued: FALSE
schemaIdGuid: 97de9615-b537-46bc-ac0f-10720f3909f3
systemOnly: FALSE
searchFlags: 0
linkID: 1044
systemFlags: FLAG_SCHEMA_BASE_OBJECT

[Partition 1, notice partition 1 lists both ADAM1.ACME.COM and ADAM2.ACME.COM]

[Partition 2 only contains one entry for ADAM1.ACME.COM]

How does referral error factor into this? The application making the query is reponsible for "chasing" a referral. In our case, ldifde.exe does not chase referrals, it's simply saying in order to make a change to second application partition, the change is required to be made on the server defined by msDS-NC-Replica-Locations attribute.

So how can we resolve the issue, customer preference is command line so setup can be automated.

1. C:\windows\ADAM>dsmgmt.exe
2. partition management
3. Connections
4. server connections: connect to server ADAM1:389 (Where ADAM1 is server where second partition was created and port is 389)
5. server connections: q
6. partition management: add NC Replica "DC=BlogPartition2,DC=ACME,DC=COM" ADAM2:389 (Where BlogPartition2 is second partition and you want to add ADAM2 to host NC replica)
8. partition management: q
8. dsmgmt: q

Let's verify that second application partition msDS-NC-Replica-Locations attribute includes both servers. Simply go back to ADSI-Edit and check second partition, it will now show both servers. Time to see if we can create object in second partition from ADAM2.ACME.COM as previously we received referral error.

Using LDIFDE.exe, placing ldf file in root of C:\ for simplicity.

[LDF - CreateContainer.ldf]

dn: CN=CreatOnADAM2,CN=Roles,DC=BlogPartition2,DC=ACME,DC=COM
changetype: add
objectClass: container

[ACME2.ACME.COM]

Success!

I hope this is helpful, I certainly learned quite a bit about ADAM\AD LDS and specifically purpose of Attribute msDS-NC-Replica-Locations as well as "Add NC Replica"

The Case of the Strange Hyper-V Networking Issue

2012-01-25T14:04:00Z

I received a call from a customer who said they were experiencing networking problems with virtual machines running on Hyper-V. The customer had implemented a two node Windows 2008 R2 SP1 Failover Cluster using Live Migration for Cluster Shared Volumes. Specifically, it was reported, virtual machines running on cluster could not access CIFS-based share on NetAPP appliance. Oddly enough, all of the virtual machines running on either node could access any standard Windows share; it was only the NetAPP appliance which could not be reached. Keep in mind that NetAPP appliance was not new to the environment and could be accessed from any machine, physical or virtual, as long as the client was not being hosted by cluster in question. Also, from the Hyper-V host, the NetAPP share could be reached without a problem. The error from the client virtual machine(s) when attempting to access share is below. When sometimes accessing the share this error would be generated immediately, other times the explorer shell would sit for a full minute. A network trace from the client showed traditional three hand shake between virtual machine and NetAPP followed by reset from the client. I did not have a network trace available from the NetAPP appliance to see what was occurring on server side.

The first question of course is the infamous “What has changed?” Fortunately, customer had a formal change process and could detail changes made before problem. In a nutshell:

Recommended hotfixes and updates for Windows Server 2008 R2 SP1 Failover Clusters
http://support.microsoft.com/kb/2545685
The network location profile changes from "Domain" to "Public" in Windows 7 or in Windows Server 2008 R2
http://support.microsoft.com/kb/2524478
Last month’s security updates via Windows Update
Latest HP Proliant Support Pack

At the time of the problem I had the following thoughts:

Can’t be the NetAPP appliance because it can be accessed from other Windows machine outside of Cluster
Can’t be the Hyper-V host because from each Hyper-V host, the parent partition, the Administrator could access the share
Not a hardware problem since the issue is present on both cluster nodes
Highly unlikely Windows security patches are relevant especially since they were applied only to the Hyper-V hosts not the virtual machines
Highly unlikely recommended updates to cluster as they will be included in SP2 for 2008 R2
Can’t be security because the same Administrator who accesses the share from Hyper-V host gets the permission error within the VM, therefore the dialog box is bogus

The problem with train of thought is I essentially ruled out everything…. We tried all the usual steps to collect more information about problem such as connecting/reconnecting virtual NICs within VM’s, changed from synthetic to legacy adapters within VM’s, restarted VM’s, restarted Hyper-V hosts etc. etc.

After a lot of testing through trial and error, a suggestion was made to roll back the network card driver which was updated in the HP PSP to latest version on one of the cluster nodes. Immediately after replacing the latest driver with older driver problem was solved. Evidently a bug exists in new driver which prevented only the virtual machines from accessing the NetAPP appliance, truly a strange one indeed.

The problematic driver was:

In summary, with multiple layers of virtualization and integration with 3rd party devices, troubleshooting and identifying where the problem resides can certainly be difficult. One lesson I learned which may have made identification of the problem easier would have been updating only one cluster node at a time and having client certify and sign off before patching second node. Of course, I am not sure that a basic certification would have even revealed problem unless someone attempted to access NetAPP device from within VM.

The Case of the Unexplained Server Hang

2011-08-26T04:41:00Z

Recently, I received a call from one of my customers reporting a new server deployed in their environment became unresponsive or‘hung’ almost daily. The customer explained that the server was deployed using their standard build process and wanted to collaborate on troubleshooting the issue.
I asked a few scoping questions:

What Operating System and Service Pack? Windows 2003 Enterprise SP2 X86

Was RDP possible during unresponsiveness? No

Does server return to normal operation after some amount of time? No

Does the server respond to ping? Yes

Do they have Poolmon data for review? Yes

Did they manually capture a crash dump that we could review? No. Suggested to customer Windows feature lets you generate a memory dump file by using the keyboard

Did they collect MPS Reports for review? Yes. Microsoft Product Support Reports

At this point I have some data to review and giving the customer a hard time about deploying an operating system in extended support with 32 bit architecture was unlikely to be well received.

[System Event Log - beginning with clean start of system]

System 7/14/2011
1:04 Information 6009 EventLog POOLMONEXAMPLE Microsoft (R) Windows (R) 5.02. 3790 Service Pack 2 Multiprocessor Free.

System 7/14/2011
8:29 Error 2019 Srv POOLMONEXAMPLE The server was unable to allocate from the system nonpaged pool because the pool was empty.

So, luckily enough, we have something interesting only seven hours and twenty-five minutes later and after clean boot, kernel nonpaged pool appears to be depleted? Why? We have a time frame that we should be able to correlate with Poolmon logs. You may already be familiar with Poolmon but by definition the tool is designed to “display data that the operating system collects about memory allocations from the system paged and nonpaged kernel pools and about the memory pool”. Microsoft CTS Support has a very simple script to put the collection in place if the KB by itself is not sufficient.

Ok, so on to Poolmon, let’s start with when the system was clean booted for baseline information.

[Clean boot happened at 7/14/2011 1:04]

First sample from Poolmon is shown below. The N: stands for nonpaged pool and P: stands for Paged Pool. The error above stated that the system ran out of nonpaged pool so we may want to keep a close eye on that number which is currently 26 MB. It’s important to note that 32 bit architecture is severely limited by the amount of paged and nonpaged pool accessible at startup. This is not a Windows limitation, per se, but a mathematical one. Having said that, what is the maximum amount of nonpaged pool which Windows can use on this platform and SKU? Roughly 256MB. Our friend and Windows expert Mark Russinovich has written a definitive article on this subject: Pushing the Limits of Windows: Paged and Nonpaged Pool

Next sample shows that nonpaged pool has grown at an alarming rate.

Now we are getting somewhere. We know what is happening, next step is to determine root cause by identifying offending driver. At first glance this appears too difficult to read, but for what we are after its relatively straight forward. Thankfully, poolmon sorts the drivers based on tags and the amount of pages consumed. When drivers in kernel space need memory, they ask the memory manager to assign a page of memory. There is nothing wrong for systems under load to take memory, but it is important to free that memory when load diminishes. Some of the key fields are: Memory/Available memory – no problem there, Pool N or nonpaged is maxed out which is our problem, paged pool is fine. Looking at the tags listed, you see a tag/driver called “Tidt”which has consumed 214 MB which is almost all available nonpaged pool where the next highest consuming driver only used 35 MB. We need to know what developer owns TidT tag? A few searches on the internet, unfortunately, did not yield any promising clues. A teammate shared a trick where you can parse all the files in system32 directory to locate a matching tag name. It is common practice that tag names may be in the header of the image file. This is the command I used:

C:\WINDOWS\system32\drivers>findstr TidT *.*
LhmonProxy.sys:...LhmonIoCompletion...c:\winstaps\v81\tap\win\src\lhmonproxy\ioc
ompletion.c...LhmonProxy.sys..................ï Uï.â.?í$Ç..Sï]?Vïp(3.ëE.ëE°èC!ä.
Wëunt.ïC`ÇH..SF[. SF.² SF_. ï}.à .ä....ìN. ..p..ï.ïw.ë.ëq.ïMnèQ.ä.t.ïw.à÷ïW?ë
U?t. A?d.ïu.è.â.. ..p..à÷.ä....èC$ä.t.èG.ä.u.ïC.à.|?èO.ä.u?à.}_èG.ä.tXSh.O..j.hÆ
...h.P..h(P..h`P..FU...ïE?ïM.â..PSQ .ShpP..j.hö...höP..h¿P..haP..ëE°F#...â..d.ïE
?à.t j.P .Lp..ïunâ..ï. ..p..ïMn I?è.ï. ..p..WFl...ïE°_^[ïs].?.ïG?à.t j.P .Lp.
.WFJ...ïE°_^[ïs].?...............ï Uï.Vïu?èF!ä.t.ïF`ÇH..VF.. VFìn VF.n 3.^].?
.................ï Uï.â..í$Ç..SVïp(ì^.Wï.ë]° ..p..ì~.ï7;.êE t!3.ÉïF?;.ë^.t?SP .L
p..ë^?ï6;.usï]°èU ï. ..p.._^[ïs].................ï Uï.SVWï°í$Ç..ïX(F9...ï.à÷tZïM
.ïU?èE.ëN.èM.ëV?èU.êN.êF.êV..G..Q..ëw .G.aì{.ï. ..p..ìK.è.ïA.ë.ëF.ë0ëq.ï. ..p.._
^3.[]..._^.....[]................ï Uï.ïM.ïU.ïE.ï@`QïM.RïU.QïM?RQFL ]..........
ïM.PëH? .tp..]........ï Uï.QVï. ..p..ìw.êE 96t.Sï.Lp..ï.ï.j.ë.Pëq. .96ue[èU 3.ï.
ëG?ëG.ëG.ëG. ..p..^ïs]................ï SWï. ..p..ïN.è.ìF.;.t.ï.ë.ëB.ïF?ï.Hè.ï.ë
F? ..p..ï._[.ïF.ïN.PQj. ..p..è.ï.ï° ..p..ï._[.........ï Vï5(p..Wh`í.. ..dí..3 h@
í..úhí..údí..ë=lí....tí..(...ë=pí....xí..xtCE ..Dí..haá..úHí..úDí..ë=Lí....Tí...
...ë=Pí....Xí..gToI ..Sá..h.í..úFá..úSá..ë=.á.....á..X...ë=.á....°á..TidT ...í..
h í..ú.í..ú.í..ë=?í.....í..P...ë=.í.....í..CidT ..$í..h.á..ú(í..ú$í..ë=,í....4í.
.....ë=0í....8í..gTcA ...á..ë=.á..ë=.á.._ú.á..ú.á.....á..?......á..gTcD^........
......ï W.`í..F.² .@í..F.² .aá..F.² ..í..F.² . í..F½² ..á..Fí² _..........
......ï V.`í..Fp² à.^t!3.ï.ë

This points us to the lhmonproxy.sys driver written by 3^rdparty vendor. The customer, armed with this information, was able to obtain a patch from the vendor to resolve the problem. I hope this blog is helpful providing a real world example of troubleshooting with poolmon and provides further encouragement to move towards X64 platform.

The Case of the Unexplained High CPU Condition

2011-04-07T20:45:00Z

Usually I start these posts with "One of my customers..." but in this instance this happened to me while supporting a internal line of business application. The platform is Windows Server 2008 Enterprise R2 running workloads IIS, (SCVMM) System Center Virtual Machine Manager 2008 R2, SQL 2008 SP1 on a server with 12 CPU's and 64 GB of RAM. This configuration had been in service for about two years without issue until a recent upgrade of our line of business application. Since that upgrade, the server with each day would exhibit increasingly high CPU culminating in all 12 CPUs being 100% utilized causing slow performance and client timeouts.

First steps were of course data collection. Our performance team recommends creating two standard data collector sets, the commands are below:

Setup a Performance monitor:
1. Open a command prompt and run the following command to create and start a Performance monitor log:

Logman.exe create counter Perf-Counter-Log15sec -o "c:\perflogs\Perflog15sec.blg" -f bincirc -v mmddhhmm -max 300 -c "\LogicalDisk(*)\*" "\Memory\*" "\Netlogon(*)\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\Server Work Queues(*)\*" "\System\*" "\Thread(*)\*" -si 00:00:15

Logman.exe create counter Perf-Counter-Log15min -o "c:\perflogs\Perflog15min.blg" -f bincirc -v mmddhhmm -max 300 -c "\LogicalDisk(*)\*" "\Memory\*" "\Netlogon(*)\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Processor(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\Server Work Queues(*)\*" "\System\*" "\Thread(*)\*" -si 00:15:00

Logman start Perf-Counter-Log15sec
Logman start Perf-Counter-Log15min

2. After an event, run the following to stop logging.

Logman stop Perf-Counter-Log15sec
Logman stop Perf-Counter-Log15min

These two commands will create data collectors which collect data every 15 sec and 15 min. This will help build baseline information such as how long the condition takes to present itself.

After the problem happened "perfmon" showed that, on average, all CPUs were utilized at roughly 60% where SQL accounted for 30%. However, when the problem occurred SQL CPU time doubled, causing a, high CPU condition.

So, now the question is, "What is causing this condition?" I turned to a great tool called xperf which can help diagnose a variety of issues in great detail. Xperf may require you to enable the following registry key, documented in the link below.

Reboot required supporting this:
http://technet.microsoft.com/en-us/library/cc959492.aspx

Description
“Specifies whether user-mode and kernel-mode drivers and kernel-mode system code can be paged to disk when not in use.”

Once you have your prerequisites in place, start Xperf by running the commands to isolate high CPU condition:
Trace Command(s)
xperf.exe –on Latency –stackWalk Profile

<repro, allow the High CPU condition to run for a couple of minutes.>

xperf -d HighCPU.etl

After you have collected the trace, while the high CPU condition was in state, open the file with xperf.

As we expected, similar to perfmon, CPU's are 100%. Next, we can filter based on processes.

Again, confirming data similar to perfmon, SQL is having problems. So let's use Xperf to investigate some internals to see what is going on... You can click and select a segment of the graph and right-click summary table.

At this point, the weight column shows us our resource consumption. You can see the SQL Server is the busiest process with 54%. What is it doing? By expanding the collapsed cells around SQL you can view internal functions within SQL which will help us understand what SQL is doing.

I cannot share the actual trace results, as SQL functions are private. But, I can say that Xperf will show the entire stack down to the NIC level. What I noticed was SQL Server was stuck on a series of spinlocks. By definition, a spinlock is a lock where the thread simply waits in a loop ("spins"), repeatedly checking until the lock becomes available. As the thread remains active but isn't performing a useful task, the use of such a lock is a kind of busy waiting.

Why does SQL have a bunch of threads in a loop? Let's take a look at SQL more closely using tools out of box. I launched SQL Server Management Studio and clicked on "Activity Monitor". I can see immediately that the system has 19 waiting tasks, where it should be 0.

It's clear that the first item in the list is the culprit. For each time this procedure is called, it takes anywhere from five to seven and a half seconds to complete. Further, we are calling the procedure 177 times per minute. This is simply not sustainable. Over time as SQL cannot complete this task, jobs are placed in a queue via spinlocks which can never complete, thereby eating CPU over time.

I expanded the Process tab and sorted by running tasks. Sure enough all of the tasks pending are the same stored procedure.

In my view there are two main questions:

1. Can we reduce the execution frequency of the stored procedure?

2. Can the prcocedure itself be optimized?

The procedure itself is design to collect information about running virtual machines in our farm. In our test envrionment, the number of running virtual machines is quite small. Our production environment has about 2,000 virtual machines so this is really a matter of scale and performance load. Regarding question number one, we reduced the frequency of the call to every ten seconds versus every two seconds, especially since based on our data above we know that the query itself can take between 5 and 8 seconds to complete. Therefore, we should never be calling the same query until the first one completes. On the second question, we leveraged the "Execution Plan" feature in SQL Server to provide advice on how to improve our query itself. Adding the index recommended below greatly improved the performance of query.

Suggestion solution from SQL Server Execution Plan:

Results from the performance tuning:

In summary, we reduced overall CPU utilization from 78% to 35% reducing it by more than half. Further, we leveraged some terrific tools such as Perfmon, XPerf and Activity monitor to determine root cause and implement solution. If you come across a similar situation, these tools and troubleshooting steps may be of some benefit.

Dave Guenthner [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

DPM 2010 Backup Error 0x800423F4

2011-02-03T21:09:00Z

A customer reported an issue that they could not protect virtual machines running various Microsoft workloads on a Windows Server 2008 R2 Enterprise Hyper-V Server. In Hyper-V terms, these virtual machines are defined as “Child Partitions”. More information about architecture can be found here:
Hyper-V Architecture
http://technet.microsoft.com/en-us/library/cc768520(BTS.10).aspx

At a high level, "The Volume Shadow Copy Service (VSS) is a set of COM interfaces that implements a framework to allow volume backups to be performed while applications on a system continue to write to the volumes. Using a combination of flushing changes to disk and holding writes allows the system to become in a consistent state for backup."

Overview of Processing a Backup Under VSS
http://msdn.microsoft.com/en-us/library/aa384589(v=VS.85).aspx

In my customer’s case, they are using (DPM) Microsoft Data Protection Manager 2010. The error message reported in the jobs log was a consistency check failure.

Type: Consistency check
Status: Failed
Description: The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
(0x800423F4))
More information
End time: 2/2/2011 5:25:54 PM
Start time: 2/2/2011 5:15:17 PM
Time elapsed: 00:10:37
Data transferred: 0 MB
Cluster node -
Source details: \Backup Using Child Partition Snapshot\MYBLOGVM

Protection group: BLOGGroup

Items scanned: 0
Items fixed: 0

C:\>err 0x800423F4
# for hex 0x800423f4 / decimal -2147212300 :
VSS_E_WRITERERROR_NONRETRYABLE vswriter.h
# indicates that the writer failed due to an error
# that most likely would occur if another snapshot is created
# 1 matches found for "0x800423F4"

Based on this error we can conclude that this is not necessarily a DPM issue but a VSS issue preventing DPM from working properly. The first thing I always recommend is ensuring that DPM is running with the latest rollup available. At the time this blog was authored, I applied the following update.

System Center Data Protection Manager 2010 QFE Rollup [KB 2250444]
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=f399fbfa-5c8b-4eb6-bda2-ea997745919a

After installing this update, the job still failed. The error within DPM had a link called “More Information” referencing the following information:

Automount should be enabled on all nodes
Error: (ID 30111 Details: Unknown error (0x800423f4) (0x800423F4))
Posible solution: Enable automount (cmd /c mountvol.exe /E) on the Hyper-V host (on all nodes in a cluster if the hosts are part of a Failover Cluster)

This Hyper-V server is not a part of a cluster but the error code is specific so I checked whether the automount feature of Windows was enabled, which is the default. The system had automount disabled and after some research found that this is a requirement for child partitions to be backed up via VSS Hyper-V Writer. I enabled automount and restarted consistency check jobs successfully!

Dave Guenthner [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

The case of the PSRemotingTransportException

2010-12-10T23:10:00Z

The case of the PSRemotingTransportException

I have a SCVMM 2008 R2\Hyper-V farm in which we use Powershell to automate provisioning of virtual machines. All of the servers are Windows Server 2008 R2. During our deployment process I encountered a failure and this of course sparked my interest to investigate further for root cause. The script includes a command to remotely invoke a command executed on a Hyper-V server. In our environment, we have many volumes so to ease Administration we rename the volume label to match the customer VM for identification. Therefore, we can correlate from a storage perspective which VMs reside on a particular volume from disk management for example.

At any rate, the command should rename the volume label.

Invoke-Command -ComputerName $hostname -ScriptBlock {param($f, $e)cmd /c label /MP $f $e} -ArgumentList $accesspath, $vmname

[Error]

The WSMan service could not launch a host process to process the given request. Make sure the WSMan provider host server and proxy are properly registered. For more information, see the about_Remote_Troubleshooting Help topic. + CategoryInfo: OpenError: (:) [], PSRemotingTransportException + FullyQualifiedErrorId : PSSessionStateBroken

I found this puzzling since the server had been in production for almost two years without issue. I executed the same command from my desktop without error. I then executed the command from the same server from where the exception took place, still the command succeeded. I took a look at the Windows Event Viewer and a few errors regarding with user profiles. The event log named the profile owner as my service account that our automation depends.

[Application Event Log]

Event 1530 Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards.

I logged server via RDP with my account and noticed that the profile for the service account had a status of “Backup” where it should say “Local”. This can occur when a profile cannot be loaded and a temporary profile is used instead. I deleted the user profile from User Profiles and logged off and then successfully established a remote desktop session with the service account as before this was not possible. I found that new profile in a desired state “Local”. I issued the same command to rename the volume label using the service account and it succeeded using the service account!!

Finally, I deployed some VMs using our automating and all Powershell tasks executed successfully. I wanted to share this in the event that someone else encountered this issue as its quite odd and the solution had no impact on customer’s or their running VMs. The root cause was either a corrupt user profile or that profile was locked in use by another process.

Dave Guenthner [MSFT]

This posting is provided "AS IS" with no warranties, and confers no rights.

How to change ownership of virtual machines via PowerShell with SCVMM 2008 R2

2010-10-01T14:29:00Z

I recently spoke with a customer who owned roughly fifty virtual machines in his SCVMM environment and he needed to reassign ownership of all virtual machines to a teammate. As you can imagine he was not interested in double clicking on all those virtual machines from the SCVMM console and change the owner property. Is it possible to automate this task?
Leveraging the PowerShell Cmdlets provided by the VMM team this can be achieved with the following script. Just replace the variables with the appropriate values for your environment. This task will reset ownership of all virtual machines one at a time.

* Where "%Domain\1stUser% is the current owner and "%Domain\2ndUser% is the desired owner

Get-VM -VMMServer "%YourVMMServername%" | where {$_.Owner -eq "%Domain\1stUser%"} | Set-VM -Owner "%Domain\2ndUser%"

System Center Virtual Machine Manager 2008 R2 Cmdlet Reference

Dave Guenthner [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

How to create a bootable ISO using tools from the WAIK

2010-08-17T20:18:00Z

I encountered an issue where I needed to create a bootable ISO from a set of files. In my SCVMM R2/Hyper-V environment, it's much easier in some instances to attach an ISO containing content versus trying to copy over the network etc. Additionally, I could store the ISO in my SCVMM library for future needs. In this example, I needed a ISO containing Windows Server 2003 media and did not have an Internet connection to download media from MSDN but I had the flat containing installation files on hand. Microsoft provides the (WAIK) Windows® Automated Installation Kit (AIK) for Windows® 7 which helps you to install, customize, and deploy Microsoft Windows. I already had the WAIK installed on my machine and leveraged a couple of executables to create the ISO. I used the following steps:

1. Created a folder called C:\tools and placed oscdimg.exe and etfsboot.com in directory.

2. Created a folder called C:\flat and copied my installation files to directory.

3. From an elevated command prompt, executed

C:\tools>oscdimg.exe -etfsboot.com -n -h c:\flat c:\foo.iso where -b specifies the location of the El Torito boot sector file, -n Enables long file names,-h Includes hidden files and directories and foo.iso is the name of the ISO to be created.

You can obtain detailed information about command line options for oscdimg.exe from Windows Automated Installation Kit (WAIK) Users's Guide for Windows 7 in section "Oscdimg Command-Line Options"

Dave Guenthner [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

Using SCVMM Cmdlets to change VM startup and shutdown actions

2010-08-13T19:44:00Z

I was recently contacted by a customer running VMM 2008 R2 managing a farm of virtual machines. This customer had a business requirement where all virtual machines must have an action when physical server stops as "Turn off virtual machine". Originally, the environment was scoped for lab testing. As time passed, workloads running on these virtual machines began to be used more as production than test. Therefore, requirements changed and a new action when physical server stops as "SaveState" was needed. The automatic start action did not require a change as it was already set to "Automatically turn on the virtual machine if it was running when physical server stopped." The customer was concerned about all the mouse clicks and administrative time required from the Virtual machine Manager Administrator Console required to make this change to 300 virtual machines. Surely there must be an easier way to accomplish this task? Further, the customer virtual machines resided in a HostGroup specific for his line of business defined in SCVMM.

I downloaded the System Center Virtual Machine Manager 2008 R2 Cmdlet Reference and wrote the follwing script which will change the action when physical server stops to "SaveVM" for all virtual machines in customer host group.

* Notice line 7 where a Stop action value is evaluated. I added this so script is smart enough to change property of VMs which are not in compliance only

Param($VMMServer)
Add-PSSnapin Microsoft.SystemCenter.VirtualMachineManager -ErrorAction SilentlyContinue
Get-VMMServer -ComputerName $VMMServer
$HostedVMs = @(Get-VM) | where {$_.HostGroupPath -like "All Hosts\TEST\*"}
foreach ($i in $HostedVMs)
{
if ($i.StopAction.value__ -ne 0)
{
Set-VM $i -StopAction SaveVM
}
}

<#
.SYNOPSIS
You will need to pass in the VMM Server name at run time and modify line 4 with appropriate Hostgroup. This script will set properties to all VMs in that Hostgroup defined
.DESCRIPTION
Set action when physical server stops to Saved State for VMs running in specific host group
.Example
Script requires one parameter containing the VMMServer
#>

You will begin to see jobs running in VMM Administrator Console similar to below:

I would also like to point out that you can change line 6 and leverage Set-VM to change any number of other properties of your VM.

This script sets the "Action when physical server stops" to "Shut down guest OS"

<#
.SYNOPSIS
You will need to pass in the VMM Server name at run time and modify line 4 with appropriate Hostgroup. This script will set properties to all VMs running in that hostgroup
.DESCRIPTION
Set action when physical server stops to Shutdown for VMs running in specific host group
.Example
Script requires one parameter containing the VMMServer
#>

Dave Guenthner [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

How to identify Hyper-V VM by worker process if VM is hung on Starting/Stopping/Snapshotting

2010-08-12T16:04:00Z

I recently encountered a situation where one of my Hyper-V virtual machines in production environment hung at Starting... 0%. This particular server had about 30 other VMs running normally used by customers. I wanted multiple methods and options to identify which VM worker process was hung and terminate it without disturbing other systems. In the few instances that I have encountered this issue, terminating the worker process with VM allowed the VM to start normally. I knew that each VM running would have a Virtual Machine Worker Process associated with it but I didn't know how to identify VM by worker process. The first glance at Task Manager was daunting and unclear:

I found three easy methods for identification:

1. From within Task Manager you can click View -> Select Columns and check Image Path Name and Command Line

Now Task Manager provided an additional piece of information I was missing, "Command Line" revealing the GUID used by Hyper-V. Since I knew the VM name which was experiencing the issue, I browsed the directory where the VM resided on file system. Under the folder containing the VM a sub-directory existed name "Virtual Machines". With that folder a XML resided, I simply documented part of that string and searched matching name in task manager. Once I identified the process I could terminate it from task manager.

2. Since I am a Mark Russinovich fan, I naturally turned to Process Explorer for help. I simply went to live.sysinternals.com and download Process Explorer and launched with elevated rights. (This is just an example, which is the reason the GUID between screen shots do not match, but they certainly would if real) Once worker process has been identified in the same way as above, you can use the "Kill process" button.

3. Finally, I downloaded process monitor from live.sysinternals.com. This time I used the name of the VM called "FINDMEVM"as a filter to help identify the GUID for XML file.

Use same process as options 1 and 2 to terminate process.

Dave Guenthner [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

SCVMM 2008 R2 Creating Template with Keyboard Layout

2010-08-06T14:38:00Z

I recently saw a forum post asking for a procedure to deploy a sysprepped VM and have the keyboard layout something other than English. SCVMM builds an unattend.xml and by default sets English as default language. You can override that value by placing section of the unattend.xml in the library with the desired settings and associate with template. In my case I have chosen to deploy Windows 7 with German keyboard layout. The values in my custom unattend were:

I performed a clean install of Windows 7 X64 Enterprise into a workgroup and sysprepped VM without an unattend.xml, shutdown and exported in Hyper-V. I then copied the VHD to library share along with custom unattend.xml named "x64 08 de-DE.xml", ran a job in VMM to refresh library share. At this point you can create a new template with customization and add a reference to the custom unattend.xml from library share.

As you can see the deployment results in default input language as German.

Dave Guenthner [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

SCVMM R2 Deployments with customization under the hood

2010-07-21T17:17:00Z

SCVMM R2 Deployments with customization under the hood

SCVMM streamlines and simplifies operating systems deployments by way of templates with customization. This technology creates a sysprep.inf or unattend.xml on the fly based on operating system associated with the template. This is very powerful as the System Administrator does not need to be a deployment expert with the WAIK or other tools to easily deploy Windows in an automated fashion.

In a basic scenario, I attach a Windows DVD to a VM and perform a clean build in a workgroup. In a more advanced scenario I may leverage (MDT) Microsoft Deployment Toolkit to provide quality control leveraging task sequences for applications and customizations. You can now Sysprep, shutdown VM, export in Hyper-V and copy VHD to SCVMM library share. You must manually refresh the library share from SCVMM Administrator Console or wait for standard refresh interval. Based on the VHD you just stored, created a template with customization and input appropriate fields such as product key or organization name.

[Creating the template from VHD]

Now we are ready for a deployment. SCVMM will copy VHD over the network via BITS network service to Hyper-V host; attach virtual floppy disk containing a dynamically generated answer file. The setup engine of Windows will look for well know locations for answer files. In this case, since its Windows XP, the sysprep.inf will be found on the virtual floppy and subsequently automating (OOBE) Out of Box Experience.

Collecting a SCVMM trace while deployment job is running will reveal steps taken to complete task. For brevity, I will show a few of the main subtasks highlighting the overall process. In this case the trace shows a sysprep.inf as template was based on Windows XP Operating System.

[Trace Exert]

Cmdlet New-VM:

Placement: Final DeploymentPreference for the set: is: BITS

Description Deploy file (using BITS over HTTPS): 0 of 1 files (76.50 MB/3.25 GB) time remaining

Inside GenerateSysPrepFile:

[Sysprep.inf being created]

GenerateSysPrepFile: Adding sysprep entry, Section = UserData, Key = FullName, Value = User

GenerateSysPrepFile: Adding sysprep entry, Section = UserData, Key = OrgName, Value = Microsoft

GenerateSysPrepFile: Adding sysprep entry, Section = UserData, Key = ComputerName, Value = DAVG12345624 in targetFile = C:\ProgramData\Microsoft\Virtual Machine Manager\0febe968-2417-41ac-94dc-2c55e0202582\sysprep.inf

[Copying sysprep.inf to virtual floppy]

Initializing CreateSysprepVfdSubtask to create vfd file C:\ProgramData\Microsoft\Virtual Machine Manager\0febe968-2417-41ac-94dc-2c55e0202582\DAVG12345624.vfd containing customized sysprep.inf

[Deploying virtual floppy to Hyper-V host and attaching to VM]

Initializing DeployVfdFilesSubtask to deploy vfd file C:\ProgramData\Microsoft\Virtual Machine Manager\0febe968-2417-41ac-94dc-2c55e0202582\DAVG12345624.vfd from VMMSERVER.DAVELAB.com to c:\images\DAVG12345624\DAVG12345624.vfd on HYPERVHOST.DAVELAB.com

Initializing attachSysPrepVfdToVmSubtask to attach floppy image c:\images\DAVG12345624\DAVG12345624.vfd to VM 762968F7-E598-49E7-93F8-F9E7303BB1E7

As you can see, VMM tracing is a great way to not only troubleshoot problems but also gain more insight on how the product works overall.

Dave Guenthner [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

DPM 2010 Remote Agent Install (windows firewall settings)

2010-07-20T17:10:00Z

I was recently tasked with setting up a DPM 2010 environment to troubleshoot a customer issue. When attempting to deploy the DPM agent from the DPM Administrator console, I received an error 349. “An error occurred when the agent operation attempted to transfer agent installation files to <server name>. Review the error details, take the appropriate action, and then retry the agent operation.”
In my case the DPM Server and protected agents were all Windows Server 2008 R2. The out of box firewall settings do not allow DPM Server and agent to communicate. When running the "Protection Agent Installation Wizard" I noticed the "Install Agent" option is for computers where required firewall settings have already been configured. Since I did not know what those setting should be, I opted to execute agent installer manually from a share on the DPM Server to resolve issue.
[Option A]
\\%DPMServer\c$\ProgramFiles\Microsoft DPM\DPM\ProtectionAgents\RA\3.0.7696.0\amd64\DPMAgentInstaller_x64.exe %DPMServerName%
If you do not pass in the DPMServer name you may need to run the second command on the protected server to update/create firewall rules.

[Option B]
C:\Program Files\Microsoft Data Protection Manager\DPM\bin>SetDpmServer.exe -DPMServerName %DPMServerName%

[Output from command line]
Configuring dpm server settings and firewall settings for dpm server
Configuration completed successfully!!!

After the commands were executed I was successfully able to manage agents from DPM Administrator Console. Reviewing Windows Firewall with Advanced Security shows several new inbound rules for dpmra.exe.

[DPM 2010 Error Code Catalog]
http://technet.microsoft.com/en-us/library/ff399290.aspx

Note:

Each DPM QFE comes with a new version of the agent and is installed on the DPM server under C:\Program Files\Microsoft DPM\DPM\agents\RA\<version> folder
For instance KB 2250444 agent is under the 3.0.7706.0 folder and KB 2465832 is under 3.0.7707.0 folder. You can use the agent from the QFE version or the RTM version defined above, then update the agent to the latest under the DPM agents console – it will say update available, just click the link and it will automatically install the latest agent.

Dave Guenthner [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.