If you read nothing else in this blog post take a look at this http://msexchangeteam.com/archive/2006/06/15/427966.aspx .

Recently I was faced with a situation where a DC was receiving Event ID 623 events in the Directory Services eventlog. I had to quite a bit of searching for information on these events and came to the conclusion that AD guys don’t seem to have as many issues with the AD database as much as the Exchange guys. The reason I came to this conclusion, is that the majority of the information that relates to the JET database (or at least to the Version Store) that services both AD and Exchange is written in an Exchange context. Basically, the event relates to Version Store Exhaustion.

So before we go any further I thought I’d better refresh on what the Version Store is and found a couple of explanations. The official explanation is here http://technet.microsoft.com/en-us/library/cc772829.aspx , but I prefer simpler wording (being a simple guy) and think this article helps my understanding better “What is Version Store?” - http://msexchangeteam.com/archive/2006/04/19/425722.aspx ; yes I know it’s by Exchange guys, but that’s how this research generally panned out and it is still relevant. This is an extract and there is also a pretty good explanation of the event ID 623 in the rest of the article.

The Version Store keeps an in-memory list of modifications made to the database. This list has several uses:

1.       Rollback - If a transaction needs to rollback it looks in the Version Store to get the list of operations it performed. By performing the inverse of all the operations the transaction can be rolled-back.

2.       Write-conflict detection - If two different sessions try to modify the same record the Version Store will notice and reject the second modification.

3.       Repeatable reads - When a session begins a transaction it always sees the same view of the database, even if other sessions modify the records it is looking at. When a session reads a record the Version Store is consulted to determine what version of the record the session should see.

4.       Deferred before-image logging - A complicated optimization that lets us log less data than "other" database engines.

In simple terms, the Version Store is where transactions are held in memory until they can be written to disk.  If something is preventing us from completing transaction or writing to disk we will consume this cache and the store will stop responding to request until there is room in the cache again.

By default the maximum amount of memory that can be allocated to the Version Store is the lesser of one fourth the amount of physical memory or 100Mb. This can be increased, but it is advised that this is a temporary solution to ease a specific issue. HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\”EDB max ver pages (increment over the minimum)” and it is a REG_DWORD. This value represents 16kbyte pages, e.g. a value of 1000 would be 1000 x 16kbytes.

Anyway, back to the reason for this article. I had my suspicions what was causing the 623 events (combination of an automated provisioning solution and\or Garbage Collection of ridiculous amounts of tombstones (another story) and underspec hardware; it was old tin). So, I decided that firstly, we needed to get some performance data, but the performance counters for monitoring the Version Store are not readily available and I needed to do some searching. Once again the majority of the information I found came from Exchange resources, but I found this http://technet.microsoft.com/en-us/library/cc961947.aspx which relates to Windows 2000, but is still valid. The results of the findings I will have to leave for another article.

To be able to view the allocated (16kb) pages of the version store you will need to carry out the following process:-

1.       Copy the performance DLL (Esentprf.dll) located in SystemRoot\ System32 to any directory (for example, C:\Perf).

2.       Run Regedit.exe, and make sure that the following registry subkeys exist:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESENT
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESENT\Performance
If these subkeys do not exist, you need to create them.
For more information about creating registry subkeys, see Windows 2000 Server Help.

3.       Make sure that, under the Performance subkey, the registry values that have the following settings exist:

a.       Open : data type REG_SZ : OpenPerformanceData

b.      Collect : data type REG_SZ : CollectPerformanceData

c.       Close : data type REG_SZ : ClosePerformanceData

d.      Library : data type REG_SZ : c:\perf\esentprf.dll

e.      Show Advanced Counters : data type REG_DWORD : 1 (This used to be called “Squeaky Lobster” and is pretty well known in Exchange circles; interesting blog on why it was called this from Brett Shirley here http://msexchangeteam.com/archive/2006/06/15/427966.aspx . It still works if you use “Squeaky Lobster” instead of “Show Advanced Counters” J).

4.       Open a command prompt and change directory to SystemRoot \Winnt\System32 or to another folder that contains the files Esentperf.ini and Esentprf.hxx generated when Esentprf.dll was compiled.

5.       (Optional) To verify that previous counter information is not present in the registry, at the command prompt, type unlodctr.exe ESENT .

6.       To load the counter information into the registry, at the command prompt run Lodctr.exe Esentprf.ini.

To view the counters for the Database object, restart Performance Monitor. The counters are under Database and the one you need to monitor the Version store size is “Version Buckets Allocated”. This will show the number of 16kbyte pages allocated in the Version Store.

Recently, a colleague of mine had a customer that had a Hyper-V server whose operating system had failed. They did not have any backups of the Operating system and had not taken any backups of the Hyper-V guests using Export\Import (I won't bang on about Best Practices here; I think you probably guess what I would say if I did though :-)). They had access to the disks on which the Hyper-V guest VHD's and configuration xml files were located and intended re-installing the the OS and Hyper-V Role. They were asking whether there is a method of recovering the Hyper-V guests to the new server. Fortunately another one of my colleagues suggested the method detailed on this great blog here http://blogs.msdn.com/b/robertvi/archive/2008/12/19/howto-manually-add-a-vm-configuration-to-hyper-v.aspx which I am led to believe worked.

Funnily enough, I had to use parts of this same method to recover a couple of Hyper-V guests on my PC when I had done some disk volume management and moved the VHD's and config files around a bit, which caused permissions changes obviously; honestly guv'nor I had forgotten I had these guests running from the said volume (I know Best Practice and all that ;-) , but it is just my laptop and not a production service providing environment).

My colleague then commented that it would be cool if such a process were to be scripted. Tada ...... "his wish was my command". To be honest I had put this on my list of things to do, to get me to extend my scripting skills from VBScript to Powershell and having to actually go through the process myself made me agree with my colleague a script would be so much easier and useful to anybody else having to do it. There may actually be another script out there that does this, but at least I actually got to do some Powershell scripting :-) . It may not be the most correctly written script, but it is my first crack of the whip (as the saying goes). There is currently no error trapping and\or exception handling, but I have provided some command line markers that should make it easier to determine at which points it fails (if it does). I intend adding error handling later, but if I didn't share the script now it would always be sat on my laptop as a work in progress without being used (and tested).

What I must say is, that after recovering the Hyper-V Guest using this script, the Hyper-V Guest is in an unsupported state and it is highly recommended that before starting the guest OS, you should use Hyper-V Manager to Export the Guest, remove it from Hyper-V Manager and then Import again into Hyper-V Manager. This will set all the permissions on all files as they should be.

This script definitely works with Windows Server 2008 R2 Hyper-V and I suspect it also works on Windows Server 2008 (I don't have a n environment to check this though). The script utilises icacls and mklink (both provided as command line applications with the OS), because I was unable to use inbuilt Powershell functionality without some deep coding knowledge (not quite there yet) requiring that I write some additional libraries that someone has actually already made available here http://pscx.codeplex.com/ . Basically, the Powershell cmdlet Set-ACL is not able to change owner when setting permissions on certain types of files and folders, this requires some special jiggery-pockery of which I am not yet averse. You may ask why I didn't decide to use the PSCX myself; well I thought it best to produce a script that is standalone and does not require anything that is not already available through Powershell.

Additionally, the script will only work correctly if the volume paths (i.e. drive letters) are the same as from before the issue that caused the Hyper-V guests to be disconnected from the Hyper-V Host Management Services.

The only other thing you will need to do when using this script is to open Powershell with elevated privileges and to run "set-executionpolicy remotesigned".

It may be worth checking the formatting of the code if you copy and paste it from this page as some of the lines have wrapped to the next line on this page. Alternatively, just download the txt file at the bottom of this post.

Enjoy. Oh and please provide feedback if it works or not for you.

####################################################################
# Recover-VM.ps1
#
# Syntax:  Recover-VM.ps1 "<drive:\path\VM_Config_File_as_GUID.xml>"
#          (Use quotes in the argument)
#
# Example: Recover-VM.ps1 "v:\VirtualMachine1\Virtual Machines\7660AA46-BA41-4171-8820-CDD7C71050A0.xml"
#
# Purpose: Creates Symbolic Links and sets the required permissions
#          to make a Hyper-V Virtual Machine available in Hyper-V
#          Management Console. This script is intended for use in
#          recovery scenario where a Virtual Guests files have been
#          manually moved to another Virtual Server, but the volume
#          drive mappings are the same, only.
#          The methodology that this script employs is as discussed
#          in the below blog.
#          It should be noted that once this script has been used
#          the Virtual Guest is in an unsupported state and should
#          be exported and re-imported in Hyper-V Manager to
#          correctly set all the Virtual Machines permissions.
#
# Params:  The drive, folder and filename of the Virtual Machine
#          xml configuration file.
#
# Req:     Windows 2008 or above, Powershell V2, mklink, icacls.
#          run "set-executionpolicy remotesigned" in Powershell
#
# Ref:     http://blogs.msdn.com/b/robertvi/archive/2008/12/19/howto-manually-add-a-vm-configuration-to-hyper-v.aspx
#          http://blogs.technet.com/b/carlh
#
# Author:  CarlH
#
# Myself or my employer do not warantee this script in any way and use
# of it is entirely at the users own risk. The script is intended to
# Recover Virtual Machines only in a recovery scenario, and such VM's
# should be exported and imported onto the Hyper-V server to ensure
# a supported state.
#
# Version: 1.0 - First cut
#
####################################################################

Param($VMConfigFile)

Write-Host ""
Write-Host "---------------Recover-VM Script Start"
$SystemDrive = (Get-item env:systemdrive).Value
$VMFolder = (Get-Item $VMConfigFile).directoryname
$VMFolder = (Get-Item $VMFolder).parent
$VMFolderName = $VMFolder.fullname
Write-Host ""

#Read from the VM Configuration file. VM GUID for Service account. VM Name.
Write-Host "---------------Reading VM Configuration File"
[xml]$VMXML = Get-Content $VMConfigFile
$VMSvcAccount = $VMXML.configuration.properties.global_id."#text"
$VMName = $VMXML.configuration.properties.name."#text"
Write-Host "---------------Completed reading VM Configuration File"
Write-Host ""

#This function takes a file name including the full path as a ThisItem string parameter and gets the current ACL.
#It adds the VM Service account to the current ACL and then Sets the new ACL
function SetACL
{
    Param($ThisItem)
    Write-Host ""
    Write-Host "---------------Setting Permissions for $ThisItem"
    Write-Host ""
    $Acl = Get-Acl $ThisItem
    $AccessRule = New-Object  system.security.accesscontrol.filesystemaccessrule("NT VIRTUAL MACHINE\$VMSvcAccount","FullControl","Allow")
    $Acl.SetAccessRule($AccessRule)
    Set-Acl $ThisItem $Acl  
    Write-Host ""
    Write-Host "---------------Finished Permissions for $ThisItem"
}

#This Function uses the cmd line utility mklink to create a symbolic link of the physical file passed to it in the HardFile parameter as a string
#The LinkLocation string parameter defines what folder the link will be created in under systemdrive\programdata\Microsoft\Windows\Hyper-V\

function MkSymLink
{
    Param($HardFile, $LinkLocation)
    Write-Host ""
    Write-Host "---------------Creating Symbolic Link for $HardFile"
    Write-Host ""
    $LinkFile = (get-item $HardFile).name
    #Write-Host "%systemdrive%\programdata\Microsoft\Windows\Hyper-V\$LinkLocation\$LinkFile" $HardFile
    cmd /c mklink "%systemdrive%\programdata\Microsoft\Windows\Hyper-V\$LinkLocation\$LinkFile" $HardFile
    Write-Host ""
    Write-Host "---------------Finished Symbolic Link for $HardFile"
}

#Get the VM config xml Fully Qualified Name and pass it to the MKSymLink function with the location for the for storing the Symbolic Link
$VMFileString = (get-item $VMConfigFile).name
MkSymLink -HardFile $VMConfigFile -LinkLocation "Virtual Machines"

#Call the SetACL function and pass it the FQN of the Symbolic Link to which we need to set the ACL
$ConfigFileSymLink = "$SystemDrive\programdata\Microsoft\Windows\Hyper-V\Virtual Machines\$VMFileString"
SetACL -ThisItem $ConfigFileSymLink

#Get the FQN of the boot VHD from the VM Config xml file passed into the script. Then pass this to the SetACL function
$VMVHDBootFile = $VMXML.configuration.'_83f8638b-8dca-4152-9eda-2ca8b33039b4_'.controller0.drive0.pathname."#text"
SetACL -ThisItem $VMVHDBootFile

#Build a string that defines the VM Service Account and use it with the cmd line utility icacls to set the permissions for all folders and files of the VM
Write-Host ""
Write-Host "---------------Setting Permissions for Folder $VMFolderName"
Write-Host ""
$VMSvcAccountFQN = 'NT VIRTUAL MACHINE\'+$VMSvcAccount
cmd /c icacls ""$VMFolderName"" /T /grant ""$VMSvcAccountFQN""':(F)'
Write-Host ""
Write-Host "---------------Finished Permissions for Folder $VMFolderName"

#Discover if there are any Snapshots. If so, for each one, create a Symbolic link for each by passing the FQN of the snapshot and snapsot link location
#Also, call SetACL to set permissions for the VM Services account on the Symbolic Link
foreach ($Thing in (Get-ChildItem $VMFolderName))
{
    If ($Thing.name -eq "Snapshots")
    {
    $SnapshotsFolder = $Thing.Fullname
        foreach ($Thing2 in (Get-ChildItem $SnapshotsFolder))
        {       
            If ($Thing2.extension -eq ".xml")
            {
            $VMSnapshotFileString = $Thing2.name
            MkSymLink -HardFile $Thing2.fullname -LinkLocation "Snapshots"
            $SnapshotFileSymLink = "$SystemDrive\programdata\Microsoft\Windows\Hyper-V\Snapshots\$VMSnapshotFileString"
            SetACL -ThisItem $SnapshotFileSymLink
            }
        }
    }
}
Write-Host ""
Write-Host "----------------End Script Recover-VM.ps1"

How many times have you had to acquire a kernel memory dump, but you or your customer (quite rightly) refuses to have the target system attached to the internet (which usually needed to download the symbol files). Well, I have had the dubious pleasure 3 times in the past 3 months. So, to remind me of the process I decided to write it down for future reference. If you know this already, sorry to waste your time, but for everyone else it’s one for your cerebral index.

Firstly you need to get the correct symbols files for the kernel memory dump, and just downloading the ones from WHDC or MSDN for the OS version and Service Pack version is not quite good enough, because the symbols change for each version of the kernel files that are released. There may be kernel files that have changed with Updates since the Service Pack was released.

1.    Install the Windows Debugging Tools http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx on the computer from which you require a kernel memory dump.

2.   Download LiveKd from http://www.microsoft.com/technet/sysinternals/SystemInformation/LiveKd.mspx and on the computer that you require a Kernel Memory Dump from, uncompress it to the location that the Windows Debug Tools have been installed to (the default is "C:\Program Files\Debugging Tools for Windows (X86)\").

3.    Copy ntoskrnl.exe, ntkrnlpa.exe, kernel32.dll and ntdll.dll files (not sure if all of these are required every time, but I do it just in case) from the System32 folder of the computer to be debugged (without internet access) and copy them to a folder (e.g. c:\debugsymbols\system32) on a computer with internet access. (If the internet facing computer has the same versions of these files to the non internet facing then there is no need for this part, but the Symbol cache will be enormous and all we want is the symbol files for the kernel).

4.    Install the Debugging tools on the internet facing system (or copy the installed folder from the server you installed them on previously; the default is “C:\Program Files\Debugging Tools for Windows”).

5.    From a command prompt on the Internet facing server, run this command:

C:\<path to debugging tools>\Symchk.exe   /if <path to copied file>\*.*   /s srv*<path to folder to store symbols locally>*http://msdl.microsoft.com/download/symbols

e.g. C:\Program Files\Debugging Tools for Windows (x86)>Symchk.exe /if c:\debugsymbols\system32\*.* /s srv*c:\debugsymbols*http://msdl.microsoft.com/download/symbols

6.   Copy the cached symbols (i.e. the c:\debugsymbols folder and its new contents, minus the \system32 folder with the 4 exe\dll’s in it) from the internet facing server to the original server.

7.   On the computer that you require a Kernel Memory Dump from, Open WinDbg, click on the  "File" menu, and choose "Symbol Search Path". Enter the following path SRV*c:\debugsymbols* and click Ok. Close Windbg.exe.

8.    With Administrator privileges, open a Command prompt and navigate to the WinDbg installation folder (e.g. C:\Program Files\Debugging Tools for Windows (X86)\) if this is where you expanded LiveKd to and run livekd.exe.

9.    Type “.dump  -f c:\memory.dmp” (without the quotes) to generate the Complete Memory Dump on the C: drive (you will need to make sure there is enough space on this drive).

10.  Type q to quit LiveKD.

11.  You should find memory.dmp in the root of c:\

Your work is now complete and you can compress and deliver the kernel dump for analysis.

I recently had to assist someone who had virtualised a single Windows 2003 Domain Controller from physical hardware to a virtualised Virtual Guest file using a third party (Non Microsoft) utility (not that I am saying this issue would be any different using a Microsoft utility). This is not an advised\recommended practice for Domain Controllers, but in this instance they had no choice and the DC concerned was in fact a single DC (once again; Bad Practice, I know, but we work with what we have sometimes) in a test environment. Ignoring the list of bad practices, the resulting DC was producing Event ID 13559 errors in the NTFRS log (see description below)

The File Replication Service has detected that the replica root path has changed from "x:\windows\sysvol\domain" to "x:\windows\sysvol\domain". If this is an intentional move then a file with the name NTFRS_CMD_FILE_MOVE_ROOT needs to be created under the new root path. This was detected for the following replica set: "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"

Changing the replica root path is a two step process which is triggered by the creation of the NTFRS_CMD_FILE_MOVE_ROOT file. [1] At the first poll which will occur in 5 minutes this computer will be deleted from the replica set. [2] At the poll following the deletion this computer will be re-added to the replica set with the new root path. This re-addition will trigger a full tree sync for the replica set. At the end of the sync all the files will be at the new location. The files may or may not be deleted from the old location depending on whether they are needed or not.

Indeed the description is also discussed in this KB article http://support.microsoft.com/kb/887440 and it would appear that the disk signatures for the volumes that host the SYSVOL folder structure are changed during the conversion of the system from physical to virtual. Following the procedures detailed in the error event and in the article did in fact remedy the situation and the NTFRS event logs showed this fact by logging 13560, 13553 and 13516 (which effectively proves the DC is functioning correctly from and FRS\SYSVOL perspective) over a period of time.

This issue has also been seen during the Restore of a Domain Controller from backup or during a failed attempt to move SYSVOL to another volume as per this article http://support.microsoft.com/kb/842162

Here's a weird one, you are migrating users from multiple domains to a single domain and your migration process\procedure had determined that the primary domain of a user was not what it actually is, so the settings (Profile Path, Home Drive\Directory, and Script Path) for the new user are for the wrong account. For various reasons you don't want or cannot merge the user objects attributes. This was one of the queries posed to me recently by a customer.

The script below gives a remedy to this by taking as a command line parameter the name and domain of one domain user and copying profile settings (the Profile Path, Home Drive & Directory, and Script Path) to relevant attributes of another command line parameter specified user name (in another domain). I even included the ability to prefix the ScriptPath Attribute (in case you use a different folder tree in the new domain).

Before using the script read the header to give you a clue as to the syntax. I have attached a copy of the code in a text file at the bottom of this post.

<#
####################################################################
Copy-UserProfile.ps1

Syntax:  Copy-UserProfile.ps1 -srcDomain <SourceDomain> -destDomain <DestinationDomain> -scriptPathPrefix <ScriptPrefix> -srcsAMAccountName <UserNetbiosName> -

destsAMAccountName <UserNetbiosName>

Example: Copy-UserProfile.ps1 -srcDomain domain1.local -destDomain woodgrovebank.com -scriptPathPrefix domain1\ -srcsAMAccountName carlh -destsAMAccountName carlh2

Purpose: This Sets the ProfilePath, ScriptPath, HomeDrive and HomeDirectory of the user carlh2 in the
          Woodgrovebank.com domain to the same settings as those for carlh in domain.local. Additionally,
          the scriptPath attribute content is prefixed with the word domin1\

Params:  As shown in syntax above or by typing the script name at the command prompt

Req:     Windows 2003 SP2 or above, Powershell V2.
  run "set-executionpolicy remotesigned" in Powershell

  http://blogs.technet.com/b/carlh

Author:  Carl Harrison

    This script is provided "AS IS" with no warranties, confers no rights and
    is not supported by the authors or Microsoft Corporation.
    Use of this script sample is subject to the terms specified at
    http://www.microsoft.com/info/copyright.mspx.

Version: 1.0 - First cut

####################################################################
#>

Param (
 [Parameter()][string]$srcDomain='',
 [Parameter()][String]$destDomain='',
        [Parameter()][String]$scriptPathPrefix='',
        [Parameter()][String]$srcsAMAccountName='',
        [Parameter()][String]$destsAMAccountName='')

Function GetSetUserHelp () {
$helptext=@"
NAME: Copy-UserPofile.ps1
Used to copy User Profile, Logon Script and Home details of user
from one domain to the next.

PARAMETERS:
-srcDomain          Source Domain (Required)
-destDomain         Destination Domain (Required)
-scriptPathPrefix   Prefix to add to Script Path attribute (include any back slashes or forward slashes as required)
-srcsAMAccountName  Netbios name of the user account in the source domain (Required)
-destsAMAccountName Netbios name of the user account in the destination domain (Required)

SYNTAX:
Copy-UserProfile.ps1 -srcDomain domain1.local -destDomain woodgrovebank.com -scriptPathPrefix domain1\ -srcsAMAccountName carlh -destsAMAccountName carlh2

This Sets the ProfilePath, ScriptPath, HomeDrive and HomeDirectory of the user carlh2 in the
Woodgrovebank.com domain to the same settings as those for carlh in domain.local. Additionally,
the scriptPath attribute content is prefixed with the word domin1\

"@
$helptext
exit
}

Function Get-LDAPUser ($UserName, $SourceDomain) {
    $domain1 = new-object DirectoryServices.DirectoryEntry ("LDAP://$SourceDomain")
    $searcher = new-object DirectoryServices.DirectorySearcher($domain1)
    $searcher.filter = "(&(objectClass=user)(sAMAccountName= $UserName))"
    $searcher.findone().getDirectoryEntry()
    $domain1 =""
}

Function Set-LDAPUser ($UserName2, $DestinationDomain) {
    $domain2 = new-object DirectoryServices.DirectoryEntry ("LDAP://$DestinationDomain")
    $searcher = new-object DirectoryServices.DirectorySearcher($domain2)
    $searcher.filter = "(&(objectClass=user)(sAMAccountName= $UserName2))"
    $destUser = $searcher.findone().getDirectoryEntry()
    $destUser.scriptPath = "$Global:ScriptPathPrefix" + $Global:srcUser.scriptPath
    $destUser.profilePath = $Global:srcUser.profilePath
    $destUser.homeDrive = $Global:srcUser.homeDrive
    $destUser.homeDirectory = $Global:srcUser.homeDirectory
    $destUser.setinfo()
    $domain2 = ""
}

if(!($srcDomain)) {"Source Domain Required";GetSetUserHelp}
if(!($destDomain)) {"Destination Domain Required";GetSetUserHelp}
if(!($srcsAMAccountName)) {"Netbios Name or Source Account Required";GetSetUserHelp}
if(!($destsAMAccountName)) {"Netbios Name or Destination Account Required";GetSetUserHelp}

$Global:ScriptPathPrefix = $ScriptPathPrefix
$Global:srcUser = get-ldapuser $srcsAMAccountName $srcDomain
Write-Host $Global:srcUser.displayName "in domain $srcDomain settings are:"
$Global:srcUser.scriptPath
$Global:srcUser.profilePath
$Global:srcUser.homeDrive
$Global:srcUser.homeDirectory
set-ldapuser $destsAMAccountName $destDomain
$Global:destUser = get-ldapuser $destsAMAccountName $destDomain
Write-Host ""
Write-Host $Global:destUser.displayName "in domain $destDomain settings are now:"
$Global:destUser.scriptPath
$Global:destUser.profilePath
$Global:destUser.homeDrive
$Global:destUser.homeDirectory

$Global:destUser = ""
$Global:srcUser = ""
$ScriptPathPrefix = ""