Step-by-Step for Mirrored Storage Spaces Resiliency using PowerShell

Step-by-Step for Mirrored Storage Spaces Resiliency using PowerShell

  • Comments 6
  • Likes
 
0 - Overview
 
This blog post is a walkthrough of the resiliency offered by Storage Spaces when using mirrored spaces.
For this exercise we have a single virtual machine with multiple data disks used to create the spaces.
The idea is to simulate a number of different failure and recovery scenarios, reviewing the behavior.
It includes the procedure to replace a disk and repair a degraded Space.

Notes and disclaimers:

  • Obviously, putting all virtual disks on a single physical disk won't provide real resiliency.
  • The specific Storage Spaces configuration shown in this blog post is not supported. Microsoft Support will only answer questions and assist in troubleshooting configurations where Storage Spaces uses a physical machine (not a VM) and uses one of the certified JBOD hardware solutions (see http://www.windowsservercatalog.com/results.aspx?bCatID=1573&cpID=0&avc=10&OR=1)
  • Because of the two items above, the configuration described here should only be used for demos, testing or learning environments.
 
1 - Two-way mirror with 3 disks
 
Part 1 of this post focuses on testing the resiliency of a two-way mirrored virtual disk (storage space) on a storage pool with 3 physical disks.
For each section you have the steps to perform (useful if you want to cut/paste), the actual output of the cmdlets on my test system and finally some notes explaining what happened.
 
1.1 - Attach 3 disks to a VM
 
Steps to perform on the host:
 
1..3 | % { New-VHD –Path D:\demo\disk$_.VHDX –Dynamic –SizeBytes 20GB } | Out-Null
1..3 | % { Add-VMHardDiskDrive –VMName VM2 –Path D:\demo\disk$_.VHDX -ControllerType SCSI}
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> 1..3 | % { New-VHD –Path D:\demo\disk$_.VHDX –Dynamic –SizeBytes 20GB } | Out-Null
PS C:\> 1..3 | % { Add-VMHardDiskDrive –VMName VM2 –Path D:\demo\disk$_.VHDX -ControllerType SCSI}
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                0                             D:\demo\disk1.VHDX
VM2    SCSI           0                1                             D:\demo\disk2.VHDX
VM2    SCSI           0                2                             D:\demo\disk3.VHDX
 
Notes:
 
Nothing much. Just your regular VM with 1 IDE boot disk and 3 SCSI data disks.
 
1.2 - Create pool and mirrored space
 
Steps to perform on the guest:
 
$s = Get-StorageSubSystem -FriendlyName *Spaces*
$d = Get-PhysicalDisk -CanPool $true
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks $d
$p = Get-StoragePool Pool1
Set-ResiliencySetting -StoragePool $p -Name Mirror -NumberofColumnsDefault 1 -NumberOfDataCopiesDefault 2
$p | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Mirror –UseMaximumSize
 
$L ="X”
$N = (Get-VirtualDisk -FriendlyName Space1 | Get-Disk).Number
Set-Disk -Number $N -IsReadOnly 0
Set-Disk -Number $N -IsOffline 0
Initialize-Disk -Number $N -PartitionStyle MBR
New-Partition -DiskNumber $N -DriveLetter $L -UseMaximumSize 
Start-Sleep –Seconds 10
Initialize-Volume -DriveLetter $L -FileSystem NTFS -Confirm:$false
 
Output:
 
PS C:\> $s = Get-StorageSubSystem -FriendlyName *Spaces*
PS C:\> $d = Get-PhysicalDisk -CanPool $true
PS C:\> New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks $d
 
FriendlyName            OperationalStatus       HealthStatus            IsPrimordial            IsReadOnly
------------            -----------------       ------------            ------------            ----------
Pool1                   OK                      Healthy                 False                   False
 
PS C:\> $p = Get-StoragePool Pool1
PS C:\> Set-ResiliencySetting -StoragePool $p -Name Mirror -NumberofColumnsDefault 1 -NumberOfDataCopiesDefault 2
PS C:\> $p | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Mirror –UseMaximumSize
 
FriendlyName        ResiliencySettingNa OperationalStatus   HealthStatus        IsManualAttach                     Size
                    me
------------        ------------------- -----------------   ------------        --------------                     ----
Space1              Mirror              OK                  Healthy             False                             27 GB
 
PS C:\> $L ="X”
PS C:\> $N = (Get-VirtualDisk -FriendlyName Space1 | Get-Disk).Number
PS C:\> Set-Disk -Number $N -IsReadOnly 0
PS C:\> Set-Disk -Number $N -IsOffline 0
PS C:\> Initialize-Disk -Number $N -PartitionStyle MBR
PS C:\> New-Partition -DiskNumber $N -DriveLetter $L -UseMaximumSize
 
   Disk Number: 4
 
PartitionNumber  DriveLetter Offset                                        Size Type
---------------  ----------- ------                                        ---- ----
1                X           1048576                                      27 GB Logical
 
PS C:\> Start-Sleep –Seconds 10
PS C:\> Initialize-Volume -DriveLetter $L -FileSystem NTFS -Confirm:$false
 
DriveLetter       FileSystemLabel  FileSystem       DriveType        HealthStatus        SizeRemaining             Size
-----------       ---------------  ----------       ---------        ------------        -------------             ----
X                                  NTFS             Fixed            Healthy                   26.9 GB            27 GB
 
Notes:
 
Created a Pool and Space. Resulting disk is initialized and mounted as volume X:.
 
1.3 - Verify health of the pool and space
 
Steps to perform on the guest:
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
Output:
 
PS C:\> Get-Volume X | FT -AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            26.9 GB 27 GB
 
PS C:\> Get-Volume X | Get-Partition | FT -AutoSize
 
   Disk Number: 4
 
PartitionNumber DriveLetter Offset   Size Type
--------------- ----------- ------   ---- ----
1               X           1048576 27 GB IFS
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | FT -AutoSize
 
Number Friendly Name                  OperationalStatus Total Size Partition Style
------ -------------                  ----------------- ---------- ---------------
4      Microsoft Storage Space Device Online                 27 GB MBR
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                OK                Healthy      False          27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        OK                Healthy      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage           Size
------------  ------- ----------------- ------------ -----           ----
PhysicalDisk2 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk1 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk3 False   OK                Healthy      Auto-Select 19.25 GB
 
Notes:
 
All is healthy: Volume, Space (Virtual Disk), Pool and Physical Disks.
 
1.4 - Remove 1 disk
 
Steps to perform on the host:
 
Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                1                             D:\demo\disk2.VHDX
VM2    SCSI           0                2                             D:\demo\disk3.VHDX
 
Notes:
 
Removed one of the 3 SCSI disks for the VM.
That is the equivalent, in a physical machine, to pulling a drive out while the system is running (assuming you have a drive that supports hot removal).
 
1.5 - Verify effect on the pool and space
 
Steps to perform on the guest:
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
Get-EventLog -LogName System -Source Disk | ? EntryType -ne "Information" | FL
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Level -lt 4 | FL
 
Output:
 
PS C:\> Get-Volume X | FT -AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            26.9 GB 27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                Incomplete        Warning      False          27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        Degraded          Warning      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName   CanPool OperationalStatus  HealthStatus Usage           Size
------------   ------- -----------------  ------------ -----           ----
PhysicalDisk2  False   OK                 Healthy      Auto-Select 19.25 GB
PhysicalDisk-1 False   Lost Communication Warning      Auto-Select 19.25 GB
PhysicalDisk3  False   OK                 Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-EventLog -LogName System -Source Disk | ? EntryType -ne "Information" | FL
 
Index              : 14802
EntryType          : Warning
InstanceId         : 2147745949
Message            : Disk 1 has been surprise removed.
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {\DR11, 1}
Source             : disk
TimeGenerated      : 3/29/2014 5:04:16 PM
TimeWritten        : 3/29/2014 5:04:16 PM
UserName           :
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Level -lt 4 | FL
 
TimeCreated  : 3/29/2014 5:04:16 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 4:54:00 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 103
Message      : The capacity consumption of the storage pool {1874344D-B794-11E3-80B7-00155D010205} has exceeded the
               threshold limit set on the pool. Return Code: STATUS_SUCCESS
 
Notes:
 
As expected, the mirror is now incomplete and pool is degraded, but the volume is still healthy.
Applications won’t see the event, but alarm bells are ringing in the event logs.
 
1.6 - Add back the removed disk
 
Steps to perform on the host:
 
Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                0                             D:\demo\disk1.VHDX
VM2    SCSI           0                1                             D:\demo\disk2.VHDX
VM2    SCSI           0                2                             D:\demo\disk3.VHDX

Notes:
 
Added the VHDX file back.
That is the equivalent, in a physical machine, plugging the drive back in (assuming you have a drive that supports hot plugging).
 
1.7 - Verify that the pool and space recover
 
Steps to perform on the guest:
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Message -like "*repair*" | FL
 
Output:
 
PS C:\> Get-Volume X | FT -AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            26.9 GB 27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                OK                Healthy      False          27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        OK                Healthy      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage           Size
------------  ------- ----------------- ------------ -----           ----
PhysicalDisk2 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk1 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk3 False   OK                Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Message -like "*repair*" | FL
 
TimeCreated  : 3/29/2014 5:11:39 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874345F-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 4:54:00 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874345F-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
Notes:
 
Storage Spaces, seeing the drive coming back, attempts a repair and succeeds. Not sure why the log shows two repairs.
After that, all is well again: virtual disks, storage pools and physical disks all report as healthy.
Again, this is completely transparent to the applications.
 
1.8 - Remove 2 disks
 
Steps to perform on the host:
 
Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                2                             D:\demo\disk3.VHDX
 
Notes:
 
Removing 2 of the 3 disks now. There is now only one surviving disk. This will be trouble...
 
1.9 - Verify effect on the pool and space
 
Steps to perform on the guest:
 
Get-Volume X | FT –AutoSize
Get-VirtualDisk Space1 | FT -AutoSize
Get-VirtualDisk Space1  | Get-StoragePool | FT -AutoSize
Get-VirtualDisk Space1  | Get-PhysicalDisk | FT -AutoSize
Get-EventLog -LogName System -Source Disk -After "3/29/2014 5:10 PM” | ? EntryType -ne "Information" | FL
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 5:10 PM") } | FL
 
Output:
 
PS C:\> Get-Volume X | FT –AutoSize
Get-Volume : No MSFT_Volume objects found with property 'DriveLetter' equal to 'X'.  Verify the value of the property and retry.
At line:1 char:1
+ Get-Volume X | FT –AutoSize
+ ~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (X:Char) [Get-Volume], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound_DriveLetter,Get-Volume
 
PS C:\> Get-VirtualDisk Space1 | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                Detached          Unhealthy    False          27 GB
 
PS C:\> Get-VirtualDisk Space1  | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        Read-only         Unhealthy    False        False
 
PS C:\> Get-VirtualDisk Space1  | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName   CanPool OperationalStatus  HealthStatus Usage           Size
------------   ------- -----------------  ------------ -----           ----
PhysicalDisk-1 False   Lost Communication Warning      Auto-Select 19.25 GB
PhysicalDisk-1 False   Lost Communication Warning      Auto-Select 19.25 GB
PhysicalDisk3  False   OK                 Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-EventLog -LogName System -Source Disk -After "3/29/2014 5:10 PM” | ? EntryType -ne "Information" | FL
 
Index              : 14807
EntryType          : Warning
InstanceId         : 2147745949
Message            : Disk 4 has been surprise removed.
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {\DR14, 4}
Source             : disk
TimeGenerated      : 3/29/2014 5:17:32 PM
TimeWritten        : 3/29/2014 5:17:32 PM
UserName           :
 
Index              : 14806
EntryType          : Warning
InstanceId         : 2147745949
Message            : Disk 2 has been surprise removed.
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {\...\DR12, 2}
Source             : disk
TimeGenerated      : 3/29/2014 5:17:32 PM
TimeWritten        : 3/29/2014 5:17:32 PM
UserName           :
 
Index              : 14805
EntryType          : Warning
InstanceId         : 2147745949
Message            : Disk 1 has been surprise removed.
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {\DR15, 1}
Source             : disk
TimeGenerated      : 3/29/2014 5:17:31 PM
TimeWritten        : 3/29/2014 5:17:31 PM
UserName           :
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 5:10 PM") } | FL
 
 
TimeCreated  : 3/29/2014 5:17:32 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 5:17:32 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 203
Message      : Physical drive {38133EAF-D68C-7229-E30B-64D68782E509} failed an IO operation. Return Code: This device
               does not exist.. Additional related events may be found in the System event log for Disk 2.
 
 
               This drive may need to be replaced. To view its reliability counters, run this command in PowerShell:
 
               Get-PhysicalDisk | ?{ $_.ObjectId -Match "{38133EAF-D68C-7229-E30B-64D68782E509}" } |
               Get-StorageReliabilityCounter
 
               This drive may be located using the following information:
 
               Drive Manufacturer: Msft
               Drive Model Number: Virtual Disk
               Drive Serial Number:
 
               More information can be obtained using this PowerShell command:
               Get-PhysicalDisk | ?{ $_.ObjectId -Match "{38133EAF-D68C-7229-E30B-64D68782E509}" }
 
               If this drive is in an enclosure, it may be located using the following information:
 
               Enclosure Manufacturer:
               Enclosure Model Number:
               Enclosure Serial Number:
               Enclosure Slot: -1
 
               It may also be located by running this command in PowerShell:
               Get-PhysicalDisk | ?{ $_.ObjectId -Match "{38133EAF-D68C-7229-E30B-64D68782E509}" } |
               Enable-PhysicalDiskIndication
 
TimeCreated  : 3/29/2014 5:17:32 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 303
Message      : Drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or are
               missing. As a result, no copy of data is available. Return Code: This device does not exist.
 
TimeCreated  : 3/29/2014 5:17:32 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 303
Message      : Drives hosting data for storage space {18743460-B794-11E3-80B7-00155D010205} have failed or are
               missing. As a result, no copy of data is available. Return Code: This device does not exist.
 
TimeCreated  : 3/29/2014 5:17:31 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 5:11:38 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
Notes:
 
As expected, things get broken. The virtual disk is detached and the volume simply disappears completely.
The storage pool is showing as unhealthy and read-only. Two physical disks are correctly reported missing.
If there is an application using this specific volume, there will be failures and down time associated with this event.
 
1.10 - Add back the removed disks
 
Steps to perform on the host:
 
Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX
Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1 –Path D:\demo\disk2.VHDX
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX
PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1 –Path D:\demo\disk2.VHDX
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                0                             D:\demo\disk1.VHDX
VM2    SCSI           0                1                             D:\demo\disk2.VHDX
VM2    SCSI           0                2                             D:\demo\disk3.VHDX
 
Notes:
 
Putting the two drives back in. This should fix things.
 
1.11 - Verify that the pool and space recover
 
Steps to perform on the guest:
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Message -like "*repair*") -and ($_.TimeCreated -gt "3/29/2014 5:20 PM") } | FL
 
Output:
 
PS C:\> Get-Volume X | FT -AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            26.9 GB 27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                OK                Healthy      False          27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        OK                Healthy      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage           Size
------------  ------- ----------------- ------------ -----           ----
PhysicalDisk4 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk1 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk3 False   OK                Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Message -like "*repair*") -and ($_.TimeCreated -gt "3/29/2014 5:20 PM") } | FL
 
 
TimeCreated  : 3/29/2014 6:12:50 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874345F-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 6:12:47 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874345F-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
Notes:
 
Without any need for repair commands, the physical disks are detected and go back to a healthy state.
The storage pool goes back to healthy and is no longer read-only. The virtual disk recovers and the volume shows up again.
You will have some work to do restart and recover your applications, which suffered the equivalent of a hard crash.
 
1.12 - Remove 1 disk, add replacement disk
 
Steps to perform on the host:
 
Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
New-VHD –Path D:\demo\disk1r.VHDX –Dynamic –SizeBytes 20GB
Add-VMHardDiskDrive –VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1r.VHDX
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
PS C:\> New-VHD –Path D:\demo\disk1r.VHDX –Dynamic –SizeBytes 20GB | Out-Null
PS C:\> Add-VMHardDiskDrive –VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1r.VHDX
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                0                             D:\demo\disk1r.VHDX
VM2    SCSI           0                1                             D:\demo\disk2.VHDX
VM2    SCSI           0                2                             D:\demo\disk3.VHDX
 
Notes:
 
Now we’re simulating that you lost a disk and it’s really broken and never coming back.
You add a brand new disk that will be the replacement to the old disk.
 
1.13 - Verify effect on the pool and space
 
Steps to perform on the guest:
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
Get-PhysicalDisk –CanPool $true | FT -AutoSize
Get-EventLog -LogName System -Source Disk -After "3/29/2014 6:20 PM” | ? EntryType -ne "Information" | FL
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 6:20 PM") } | FL
 
Output:
 
PS C:\> Get-Volume X | FT –AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            26.9 GB 27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                Incomplete        Warning      False          27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        Degraded          Warning      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName   CanPool OperationalStatus  HealthStatus Usage           Size
------------   ------- -----------------  ------------ -----           ----
PhysicalDisk4  False   OK                 Healthy      Auto-Select 19.25 GB
PhysicalDisk-1 False   Lost Communication Warning      Auto-Select 19.25 GB
PhysicalDisk3  False   OK                 Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-PhysicalDisk –CanPool $true | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage        Size
------------  ------- ----------------- ------------ -----        ----
PhysicalDisk1 True    OK                Healthy      Auto-Select 20 GB
 
PS C:\> Get-EventLog -LogName System -Source Disk -After "3/29/2014 6:20 PM” | ? EntryType -ne "Information" | FL
 
 
Index              : 14813
EntryType          : Warning
InstanceId         : 2147745949
Message            : Disk 1 has been surprise removed.
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {\DR16, 1}
Source             : disk
TimeGenerated      : 3/29/2014 6:24:13 PM
TimeWritten        : 3/29/2014 6:24:13 PM
UserName           :
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 6:20 PM") } | FL
 
 
TimeCreated  : 3/29/2014 6:24:13 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874345F-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
Notes:
 
As we saw previously, we survive the failure of a single disk just fine. Volume is healthy, but the virtual disk and pool are degraded.
We note that there is a brand new disk available that can be pooled, but Storage Spaces will not automatically attempt to use it.
 
1.14 - Retire old disk, add new disk, repair and verify
 
Steps to perform on the guest:
 
Get-PhysicalDisk | ? OperationalStatus –eq “Lost Communication” | Set-PhysicalDisk –Usage Retired
Add-PhysicalDisk –StoragePoolFriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk –CanPool $True)
Repair-VirtualDisk Space1
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT –AutoSize
Get-StoragePool Pool1 | Get-PhysicalDisk | FT –AutoSize
 
Remove-PhysicalDisk -StoragePoolFriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk | ? OperationalStatus -eq "Lost Communication" )
Get-StoragePool Pool1 | Get-PhysicalDisk | FT –AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
Output:
 
PS C:\> Get-PhysicalDisk | ? OperationalStatus –eq “Lost Communication” | Set-PhysicalDisk –Usage Retired
PS C:\> Add-PhysicalDisk –StoragePoolFriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk –CanPool $True)
PS C:\> Repair-VirtualDisk Space1
PS C:\> Get-Volume X | FT -AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            26.9 GB 27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                OK                Healthy      False          27 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        Degraded          Warning      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT –AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage           Size
------------  ------- ----------------- ------------ -----           ----
PhysicalDisk4 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk1 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk3 False   OK                Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName   CanPool OperationalStatus  HealthStatus Usage           Size
------------   ------- -----------------  ------------ -----           ----
PhysicalDisk4  False   OK                 Healthy      Auto-Select 19.25 GB
PhysicalDisk-1 False   Lost Communication Warning      Retired     19.25 GB
PhysicalDisk1  False   OK                 Healthy      Auto-Select 19.25 GB
PhysicalDisk3  False   OK                 Healthy      Auto-Select 19.25 GB
 
PS C:\> Remove-PhysicalDisk -StoragePoolFriendlyName Pool1 -PhysicalDisks (Get-PhysicalDisk | ? OperationalStatus -eq "Lost Communication" )
 
Confirm
Are you sure you want to perform this action?
Removing a physical disk will cause problems with the fault tolerance capabilities of the following storage pool: "Pool1".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):
 
PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage           Size
------------  ------- ----------------- ------------ -----           ----
PhysicalDisk4 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk1 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk3 False   OK                Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        OK                Healthy      False        False
 
Notes:
 
After we add the new disk to the pool and command a repair, things go back to normal.
The old disk can either be retired (remains showing in the pool) or it can be completely removed from the pool.
 
1.15 - Add the old disk back
 
Steps to perform on the host:
 
Add-VMHardDiskDrive –VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 3 –Path D:\demo\disk1.VHDX
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Add-VMHardDiskDrive –VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 3 –Path D:\demo\disk1.VHDX
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                0                             D:\demo\disk1r.VHDX
VM2    SCSI           0                1                             D:\demo\disk2.VHDX
VM2    SCSI           0                2                             D:\demo\disk3.VHDX
VM2    SCSI           0                3                             D:\demo\disk1.VHDX
 
Notes:
 
Simulating an odd but plausible situation. You have cleaned up the pool and repaired everything after a disk went bad, but someone put that old disk back in.
Since you’ve already removed it from the pool, what is the result?
 
1.16 - Verify effect on the pool, reset disk
 
Steps to perform on the guest:
 
Get-PhysicalDisk | FT –AutoSize
Get-StoragePool Pool1 | Get-PhysicalDisk
Get-PhysicalDisk | ? OperationalStatus -eq "Unrecognized Metadata" | Reset-PhysicalDisk
Get-PhysicalDisk | FT -AutoSize
 
Get-EventLog -LogName System -Source Disk -After "3/29/2014 6:20 PM” | ? EntryType -ne "Information" | FL
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 6:20 PM") } | FL
 
Output:
 
PS C:\> Get-PhysicalDisk | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus     HealthStatus Usage           Size
------------  ------- -----------------     ------------ -----           ----
PhysicalDisk0 False   OK                    Healthy      Auto-Select    32 GB
PhysicalDisk4 False   OK                    Healthy      Auto-Select 19.25 GB
PhysicalDisk5 False   Unrecognized Metadata Unhealthy    Unknown        20 GB
PhysicalDisk1 False   OK                    Healthy      Auto-Select 19.25 GB
PhysicalDisk3 False   OK                    Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-StoragePool Pool1 | Get-PhysicalDisk
 
FriendlyName        CanPool             OperationalStatus   HealthStatus        Usage                              Size
------------        -------             -----------------   ------------        -----                              ----
PhysicalDisk4       False               OK                  Healthy             Auto-Select                    19.25 GB
PhysicalDisk1       False               OK                  Healthy             Auto-Select                    19.25 GB
PhysicalDisk3       False               OK                  Healthy             Auto-Select                    19.25 GB
 
PS C:\> Get-PhysicalDisk | ? OperationalStatus -eq "Unrecognized Metadata" | Reset-PhysicalDisk
PS C:\> Get-PhysicalDisk | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage           Size
------------  ------- ----------------- ------------ -----           ----
PhysicalDisk0 False   OK                Healthy      Auto-Select    32 GB
PhysicalDisk4 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk5 True    OK                Healthy      Auto-Select    20 GB
PhysicalDisk1 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk3 False   OK                Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 7:00PM") } | FL
 
TimeCreated  : 3/29/2014 7:10:53 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 201
Message      : Physical drive {5A464324-FBA5-A176-5DE8-DE8FE63E3951} has invalid meta-data. Resetting the health
               status, using command line or GUI, might bring the physical drive to the primordial pool. Return Code:
               STATUS_SUCCESS
 
Notes:
 
So you can see that the old disk comes back as unhealthy, with a status of “unrecognized metadata”.
It has already been removed from the pool and contains metadata that is inconsistent with the current state of the pools and virtual disks in the system.
There is an interesting event log in the Storage Spaces event highlighting the situation. The recommendation is to reset the health status.
We go ahead and reset it as suggested, making it healthy again and available to add back to the pool if necessary (that last portion is not shown).
 
2 - Three-way mirror with 5 disks
 
Part 2 of this post focuses on testing the resiliency of a 3-way mirrored virtual disk on a storage pool with 5 physical disks. It follows the same format as Part 1.
We are going to start fresh with a new set of disks. At this point, if you’re following along on your system, you should:
  • Inside the VM, remove all virtual disks and pools from Part 1.
  • On the host, remove the data VHDX files from the VM, then delete them.
  • Inside the VM, clear the System log and the Storage Spaces.
 
2.1 - Attach 5 disks to a VM
 
Steps to perform on the host:
 
1..5 | % { New-VHD –Path D:\demo\disk$_.VHDX –Dynamic –SizeBytes 20GB }
1..5 | % { Add-VMHardDiskDrive –VMName VM2 –Path D:\demo\disk$_.VHDX -ControllerType SCSI}
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> 1..5 | % { New-VHD –Path D:\demo\disk$_.VHDX –Dynamic –SizeBytes 20GB } | Out-Null
PS C:\> 1..5 | % { Add-VMHardDiskDrive –VMName VM2 –Path D:\demo\disk$_.VHDX -ControllerType SCSI}
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                0                             D:\demo\disk1.VHDX
VM2    SCSI           0                1                             D:\demo\disk2.VHDX
VM2    SCSI           0                2                             D:\demo\disk3.VHDX
VM2    SCSI           0                3                             D:\demo\disk4.VHDX
VM2    SCSI           0                4                             D:\demo\disk5.VHDX
 
Notes:
 
Nothing much. Just your regular VM with 1 IDE boot disk and 5 SCSI data disks.
 
2.2 - Create pool and 3-way mirrored space
 
Steps to perform on the guest:
 
$s = Get-StorageSubSystem -FriendlyName *Spaces*
$d = Get-PhysicalDisk -CanPool $true
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks $d
$p = Get-StoragePool Pool1
Set-ResiliencySetting -StoragePool $p -Name Mirror -NumberofColumnsDefault 1 -NumberOfDataCopiesDefault 3
$p | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Mirror –UseMaximumSize
 
$L ="X”
$N = (Get-VirtualDisk -FriendlyName Space1 | Get-Disk).Number
Set-Disk -Number $N -IsReadOnly 0
Set-Disk -Number $N -IsOffline 0
Initialize-Disk -Number $N -PartitionStyle MBR
New-Partition -DiskNumber $N -DriveLetter $L -UseMaximumSize 
Start-Sleep –Seconds 10
Initialize-Volume -DriveLetter $L -FileSystem NTFS -Confirm:$false
 
Output:
 
PS C:\> $s = Get-StorageSubSystem -FriendlyName *Spaces*
PS C:\> $d = Get-PhysicalDisk -CanPool $true
PS C:\> New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks $d
 
FriendlyName            OperationalStatus       HealthStatus            IsPrimordial            IsReadOnly
------------            -----------------       ------------            ------------            ----------
Pool1                   OK                      Healthy                 False                   False
 
PS C:\> $p = Get-StoragePool Pool1
PS C:\> Set-ResiliencySetting -StoragePool $p -Name Mirror -NumberofColumnsDefault 1 -NumberOfDataCopiesDefault 3
PS C:\> $p | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Mirror –UseMaximumSize
 
FriendlyName        ResiliencySettingNa OperationalStatus   HealthStatus        IsManualAttach                     Size
                    me
------------        ------------------- -----------------   ------------        --------------                     ----
Space1              Mirror              OK                  Healthy             False                             30 GB
 
PS C:\> $L ="X”
PS C:\> $N = (Get-VirtualDisk -FriendlyName Space1 | Get-Disk).Number
PS C:\> Set-Disk -Number $N -IsReadOnly 0
PS C:\> Set-Disk -Number $N -IsOffline 0
PS C:\> Initialize-Disk -Number $N -PartitionStyle MBR
PS C:\> New-Partition -DiskNumber $N -DriveLetter $L -UseMaximumSize
 
   Disk Number: 6
 
PartitionNumber  DriveLetter Offset                                        Size Type
---------------  ----------- ------                                        ---- ----
1                X           1048576                                      30 GB Logical
 
PS C:\> Start-Sleep –Seconds 10
PS C:\> Initialize-Volume -DriveLetter $L -FileSystem NTFS -Confirm:$false
 
DriveLetter       FileSystemLabel  FileSystem       DriveType        HealthStatus        SizeRemaining             Size
-----------       ---------------  ----------       ---------        ------------        -------------             ----
X                                  NTFS             Fixed            Healthy                   29.9 GB            30 GB
 
Notes:
 
Created a storage pool and storage space. Resulting disk is initialized and mounted as volume X:.
 
2.3 - Verify health of the pool and space
 
Steps to perform on the guest:
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
Output:
 
PS C:\> Get-Volume X | FT -AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            29.9 GB 30 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                OK                Healthy      False          30 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        OK                Healthy      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage           Size
------------  ------- ----------------- ------------ -----           ----
PhysicalDisk3 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk1 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk4 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk2 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk5 False   OK                Healthy      Auto-Select 19.25 GB
 
Notes:
 
All is healthy: volume, storage space (virtual disk), storage pool and physical disks.
 
2.4 - Remove 2 disks
 
Steps to perform on the host:
 
Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                2                             D:\demo\disk3.VHDX
VM2    SCSI           0                3                             D:\demo\disk4.VHDX
VM2    SCSI           0                4                             D:\demo\disk5.VHDX
 
Notes:
 
Removed 2 of the 5 SCSI disks for the VM.
Since this is a 3-way mirror, we’re going to hit it right away with a double disk failure. But it should be able to survive that.
 
2.5 - Verify effect on the pool and space
 
Steps to perform on the guest:
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
Get-EventLog -LogName System -Source Disk | ? EntryType -ne "Information" | FL
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Level -lt 4 | FL
 
Output:
 
PS C:\> Get-Volume X | FT -AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            29.9 GB 30 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                Incomplete        Warning      False          30 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        Degraded          Warning      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName   CanPool OperationalStatus  HealthStatus Usage           Size
------------   ------- -----------------  ------------ -----           ----
PhysicalDisk3  False   OK                 Healthy      Auto-Select 19.25 GB
PhysicalDisk-1 False   Lost Communication Warning      Auto-Select 19.25 GB
PhysicalDisk4  False   OK                 Healthy      Auto-Select 19.25 GB
PhysicalDisk-1 False   Lost Communication Warning      Auto-Select 19.25 GB
PhysicalDisk5  False   OK                 Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-EventLog -LogName System -Source Disk | ? EntryType -ne "Information" | FL
 
Index              : 14841
EntryType          : Warning
InstanceId         : 2147745949
Message            : Disk 2 has been surprise removed.
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {\DR22, 2}
Source             : disk
TimeGenerated      : 3/29/2014 8:38:11 PM
TimeWritten        : 3/29/2014 8:38:11 PM
UserName           :
 
Index              : 14840
EntryType          : Warning
InstanceId         : 2147745949
Message            : Disk 1 has been surprise removed.
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {\DR21, 1}
Source             : disk
TimeGenerated      : 3/29/2014 8:38:10 PM
TimeWritten        : 3/29/2014 8:38:10 PM
UserName           :
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Level -lt 4 | FL
 
 
TimeCreated  : 3/29/2014 8:38:11 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 8:38:10 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 8:31:56 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 103
Message      : The capacity consumption of the storage pool {18743521-B794-11E3-80B7-00155D010205} has exceeded the
               threshold limit set on the pool. Return Code: STATUS_SUCCESS
 
Notes:
 
As expected, the 3-way mirrored space was able to survive the double disk failure. The storage space is now incomplete and the storage pool is degraded, but the volume is still healthy.
Applications won’t see the event, but again there are many alarm bells ringing in the event logs. You better act fast because the next failure will be fatal…
 
2.6 - Add back the removed disks
 
Steps to perform on the host:
 
Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX
Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1 –Path D:\demo\disk2.VHDX
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 –Path D:\demo\disk1.VHDX
PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1 –Path D:\demo\disk2.VHDX
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                0                             D:\demo\disk1.VHDX
VM2    SCSI           0                1                             D:\demo\disk2.VHDX
VM2    SCSI           0                2                             D:\demo\disk3.VHDX
VM2    SCSI           0                3                             D:\demo\disk4.VHDX
VM2    SCSI           0                4                             D:\demo\disk5.VHDX
 
Notes:
 
Added the two VHDX files back to restore the VM to the original state.
 
2.7 - Verify that the pool and space recover
 
Steps to perform on the guest:
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Message -like "*repair*" | FL
 
Output:
 
PS C:\> Get-Volume X | FT -AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            29.9 GB 30 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                OK                Healthy      False          30 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        OK                Healthy      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage           Size
------------  ------- ----------------- ------------ -----           ----
PhysicalDisk3 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk1 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk4 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk2 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk5 False   OK                Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? Message -like "*repair*" | FL
 
TimeCreated  : 3/29/2014 8:45:51 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 8:45:47 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 8:31:56 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
Notes:
 
Storage Spaces, seeing the drives coming back, attempts a repair and succeeds.
After that, all is well again: virtual disks, storage pools and physical disks all report as healthy.
This shows that we can survive two disk failures in a way that is completely transparent to the applications.
 
2.8 - Remove 3 disks
 
Steps to perform on the host:
 
Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1
Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 2
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0
PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 1
PS C:\> Remove-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 2
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                3                             D:\demo\disk4.VHDX
VM2    SCSI           0                4                             D:\demo\disk5.VHDX
 
Notes:
 
Removing 3 of the 5 disks now. There are now only two surviving disks. That should be too much to bear.
 
2.9 - Verify effect on the pool and space
 
Steps to perform on the guest:
 
Get-Volume X | FT –AutoSize
Get-VirtualDisk Space1 | FT -AutoSize
Get-VirtualDisk Space1  | Get-StoragePool | FT -AutoSize
Get-VirtualDisk Space1  | Get-PhysicalDisk | FT -AutoSize
Get-EventLog -LogName System -Source Disk -After "3/29/2014 8:40 PM” | ? EntryType -ne "Information" | FT TimeGenerated, Source, Index, EventID, EntryType, Message -AutoSize
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 8:40 PM") } | FL
 
Output:
 
PS C:\> Get-Volume X | FT –AutoSize
Get-Volume : No MSFT_Volume objects found with property 'DriveLetter' equal to 'X'.  Verify the value of the property and retry.
At line:1 char:1
+ Get-Volume X | FT –AutoSize
+ ~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (X:Char) [Get-Volume], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound_DriveLetter,Get-Volume
 
PS C:\> Get-VirtualDisk Space1 | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                Detached          Unhealthy    False          30 GB
 
PS C:\> Get-VirtualDisk Space1  | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        Read-only         Unhealthy    False        False
 
PS C:\> Get-VirtualDisk Space1  | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName   CanPool OperationalStatus  HealthStatus Usage           Size
------------   ------- -----------------  ------------ -----           ----
PhysicalDisk-1 False   Lost Communication Warning      Auto-Select 19.25 GB
PhysicalDisk-1 False   Lost Communication Warning      Auto-Select 19.25 GB
PhysicalDisk4  False   OK                 Healthy      Auto-Select 19.25 GB
PhysicalDisk-1 False   Lost Communication Warning      Auto-Select 19.25 GB
PhysicalDisk5  False   OK                 Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-EventLog -LogName System -Source Disk -After "3/29/2014 8:40 PM” | ? EntryType -ne "Information" | FT TimeGenerated, Source, Index, EventID, EntryType, Message -AutoSize
 
TimeGenerated        Source Index EventID EntryType Message
-------------        ------ ----- ------- --------- -------
3/29/2014 8:50:21 PM disk   14847     157   Warning Disk 6 has been surprise removed.
3/29/2014 8:50:21 PM disk   14846     157   Warning Disk 3 has been surprise removed.
3/29/2014 8:50:20 PM disk   14845     157   Warning Disk 2 has been surprise removed.
3/29/2014 8:50:19 PM disk   14844     157   Warning Disk 1 has been surprise removed.
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Level -lt 4) -and ($_.TimeCreated -gt "3/29/2014 8:40 PM") } | FL
 
TimeCreated  : 3/29/2014 8:50:21 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 8:50:21 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 203
Message      : Physical drive {06724EEA-84C4-283C-1EF9-F26D195E0D7A} failed an IO operation. Return Code: This device
               does not exist.. Additional related events may be found in the System event log for Disk 3.
 
 
               This drive may need to be replaced. To view its reliability counters, run this command in PowerShell:
 
               Get-PhysicalDisk | ?{ $_.ObjectId -Match "{06724EEA-84C4-283C-1EF9-F26D195E0D7A}" } |
               Get-StorageReliabilityCounter
 
               This drive may be located using the following information:
 
               Drive Manufacturer: Msft
               Drive Model Number: Virtual Disk
               Drive Serial Number:
 
               More information can be obtained using this PowerShell command:
               Get-PhysicalDisk | ?{ $_.ObjectId -Match "{06724EEA-84C4-283C-1EF9-F26D195E0D7A}" }
 
               If this drive is in an enclosure, it may be located using the following information:
 
               Enclosure Manufacturer:
               Enclosure Model Number:
               Enclosure Serial Number:
               Enclosure Slot: -1
 
               It may also be located by running this command in PowerShell:
               Get-PhysicalDisk | ?{ $_.ObjectId -Match "{06724EEA-84C4-283C-1EF9-F26D195E0D7A}" } |
               Enable-PhysicalDiskIndication
 
TimeCreated  : 3/29/2014 8:50:21 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 303
Message      : Drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or are
               missing. As a result, no copy of data is available. Return Code: This device does not exist.
 
TimeCreated  : 3/29/2014 8:50:20 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 8:50:19 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 8:45:49 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 8:45:46 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 304
Message      : One or more drives hosting data for storage space {1874353C-B794-11E3-80B7-00155D010205} have failed or
               are missing. As a result, at least one copy of data is not available. However, at least one copy of
               data is still available. Return Code: STATUS_SUCCESS
 
Notes:
 
As expected, the volume is gone and any applications running here would have stopped.
The virtual disk is detached, the volume is dismounted, the pool is unhealthy and 3 physical disks are missing.
Again we get more disk failures and Storage Spaces failures as I would expect. Not sure why that happens.
 
2.10 - Add back the 3 removed disks
 
Steps to perform on the host:
 
Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 5 –Path D:\demo\disk1.VHDX
Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 6 –Path D:\demo\disk2.VHDX
Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 7 –Path D:\demo\disk3.VHDX
Get-VMHardDiskDrive -VMName VM2
 
Output:
 
PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 5 –Path D:\demo\disk1.VHDX
PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 6 –Path D:\demo\disk2.VHDX
PS C:\> Add-VMHardDiskDrive -VMName VM2 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 7 –Path D:\demo\disk3.VHDX
PS C:\> Get-VMHardDiskDrive -VMName VM2
 
VMName ControllerType ControllerNumber ControllerLocation DiskNumber Path
------ -------------- ---------------- ------------------ ---------- ----
VM2    IDE            0                0                             c:\demo\v2.vhdx
VM2    SCSI           0                3                             D:\demo\disk4.VHDX
VM2    SCSI           0                4                             D:\demo\disk5.VHDX
VM2    SCSI           0                5                             D:\demo\disk1.VHDX
VM2    SCSI           0                6                             D:\demo\disk2.VHDX
VM2    SCSI           0                7                             D:\demo\disk3.VHDX
 
Notes:
 
Simulating putting the 3 drives back in. This should fix things.
Just for fun, adding them in different controller locations, the equivalent of putting it back in a drive bay different than where it was originally.
Storage Spaces relies on the metadata stored in the disk itself, so this should not make any difference.
 
2.11 - Verify that the pool and space recover
 
Steps to perform on the guest:
 
Get-Volume X | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Message -like "*repair*") -and ($_.TimeCreated -gt "3/29/2014 9:00 PM") } | FL
 
Output:
 
PS C:\> Get-Volume X | FT -AutoSize
 
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining  Size
----------- --------------- ---------- --------- ------------ -------------  ----
X                           NTFS       Fixed     Healthy            29.9 GB 30 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | FT -AutoSize
 
FriendlyName ResiliencySettingName OperationalStatus HealthStatus IsManualAttach  Size
------------ --------------------- ----------------- ------------ --------------  ----
Space1       Mirror                OK                Healthy      False          30 GB
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-StoragePool | FT -AutoSize
 
FriendlyName OperationalStatus HealthStatus IsPrimordial IsReadOnly
------------ ----------------- ------------ ------------ ----------
Pool1        OK                Healthy      False        False
 
PS C:\> Get-Volume X | Get-Partition | Get-Disk | Get-VirtualDisk | Get-PhysicalDisk | FT -AutoSize
 
FriendlyName  CanPool OperationalStatus HealthStatus Usage           Size
------------  ------- ----------------- ------------ -----           ----
PhysicalDisk6 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk1 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk4 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk3 False   OK                Healthy      Auto-Select 19.25 GB
PhysicalDisk5 False   OK                Healthy      Auto-Select 19.25 GB
 
PS C:\> Get-WinEvent -LogName Microsoft-Windows-Stor* | ? { ($_.Message -like "*repair*") -and ($_.TimeCreated -gt "3/29/2014 9:00 PM") } | FL
 
TimeCreated  : 3/29/2014 9:13:14 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 9:13:08 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
TimeCreated  : 3/29/2014 9:13:05 PM
ProviderName : Microsoft-Windows-StorageSpaces-Driver
Id           : 308
Message      : A repair attempt for storage space {1874353C-B794-11E3-80B7-00155D010205} was initiated by the driver.
               Return Code: STATUS_SUCCESS
 
Notes:
 
Again, without any need for repair commands, the physical disks are detected and everything goes back to a healthy state.
Keep in mind that your application probably will need to be restarted and you should follow the required recovery steps, since there was down time.
 
3 - Conclusion
 
This is it. I hope this tour of mirrored storage spaces resiliency was helpful.
The idea was to save you the trouble of finding this out for yourself, but I’m sure you’ll still be interested in experimenting with it further.
The goal was also to prepare you for an eventual disk failure. You will likely be a bit more stressed at that time, so knowing what to expect and where to look might be helpful.
I have lots of additional ideas to try like trying more columns, adding storage tiers to the mix, using parity spaces or actually using an actual bad disk to go beyond just surprise removals.
But I guess that will have to wait for another opportunity. This one is already too long as it is… :-)

For additional content related to Storage Spaces, check these:

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment