WDS and DFSR: Love at First Sync

WDS and DFSR: Love at First Sync

  • Comments 1
  • Likes

Ned here again. Windows Server 2008 introduced a new way to roll out your computers called Windows Deployment Services (WDS). WDS replaces the old Remote Installation Services introduced a decade ago with Windows 2000. By leveraging WDS, you can create image-based deployments, script settings, multicast, and all that other stuff that gets the OS installation folks excited.

If you dig into the WDS documentation you will find that DFSR and DFSN are recommended for scaling out your deployment servers. You can centralize administration and improve performance by creating multiple distribution points that replicate OS images. All while using components you already paid for when you bought your servers.

A few months ago, someone asked if DFSR’s Remote Differential Compression (RDC) feature had any benefits when being used with WDS, where OS images are periodically updated. Would a robocopy script perform better? Are there any interoperability issues? Is the only advantage that files are automatically copied?

I’ve been asked about this a few more times since then, so today I’ll talk about the results. Spoiler alert: WDS and DFSR work really well together. :)

The Test Environment

With the invaluable assistance of Scott “the silver fox” McArthur from our sibling Web site AskCore I setup the following repro environment:

  • Two Windows Server 2008 R2 WDS servers
  • Both servers running DFSR with hotfix 975763 to update DFSRS.EXE.
  • Server “A” is the WDS administrative server where all management work is done
  • Server “B” is a WDS branch server that simply accepts whatever is sent to him and is used to deploy computers remotely
  • A variety of Win7/R2 family WIM files in an install image group, plus their single instance store RWM file. More on this in a bit.

image

image

  • Using WDS methodology to:
  1. Export an image
  2. Mount it
  3. Make changes to it ( in my case, about ~1MB of data added to the image)
  4. Dismount it
  5. Replace the previous image

WDS uses a single instance store (SIS) as part of its storage mechanism. In basic terms, this means that data common among images is unified in one RWM file. And then the differences are stored in the OS image WIM file, which operates like a stub when you start installing the OS. This saves disk space, which is critical when you’re talking about dozens of images. This also means that both the Install*.WIM and Res.RWM files are altered when modifying your image.

Note: in my case I did not have to increase the staging quota as my files still easily fit in the 4GB default staging space. You may need to increase it using our old, reliable DFSR performance tuning article:

http://blogs.technet.com/b/askds/archive/2010/03/31/tuning-replication-performance-in-dfsr-especially-on-win2008-r2.aspx

The Results

WDS always replaces the existing Install*.WIM file with another copy. Additionally, the binary contents of the replaced file are effectively completely different than the original – it’s just how WIM creation works. So RDC does not provide any gains when replicating a WIM file. The file is copied in its entirety, but more on this below. However, the WIM files in WDS are comparatively small, so DFSR moves them very fast.

WDS does not replace RWM files; it modifies them in place. This means that DFSR uses RDC and we see substantial savings over the wire in bandwidth. In our test, we added a small amount of data to the mounted image, which modifies the Res.RWM file. DFSR dies not replicate the whole 3.6GB of data; instead, DFSR replicates the delta of the change plus some overhead. The total delta of replicated data with the RWM file was around 1MB of data. That is an RDC savings of roughly 99.98%.

The cost here when using RDC, is that a file this large spends most of its time in local disk I/O and CPU time on both WDS servers.

  • Upstream on “A” - copying the file to staging, walking all of its data to create signatures – took several minutes
  • Downstream on B – copying the file to staging (unless it already exists), walking all of its data to create signatures, taking the bits that were sent over the wire and reconstructing the file with its changes, installing the file

So only a few seconds and 1MB of data was network time – the rest is “thinking time”. If you have a low-latency, high-bandwidth, gigabit network between all servers, then using DFSR with RDC to save bandwidth is not advantageous over just copying the files with say robocopy or xcopy – it might appear slightly slower. But if the network WAN connections are not fast or high bandwidth, then DFSR with RDC saves a ton of bandwidth and time. Also, you can turn off RDC on just these super-fast network scenarios and at least get DFSR’s “keep my files in sync so I don’t have to think about it” advantage.

That’s the short answer. Not everyone likes to read my DFSR debug diatribes, but if you do, carry on to the next section.

Seeing it all under the covers

20091007 18:06:13.330 2680 USNC  2450 UsnConsumer::UpdateIdRecord LDB Updating ID Record: ß file is modified, not replaced
+      fid                             0x30000000000B0
+      usn                             0x6b480
+      uidVisible                      1
+      filtered                        0
+      journalWrapped                  0
+      slowRecoverCheck                0
+      pendingTombstone                0
+      internalUpdate                  0
+      dirtyShutdownMismatch           0
+      meetInstallUpdate               0
+      meetReanimated                  0
+      recUpdateTime                   20091007 21:29:29.115 GMT
+      present                         1
+      nameConflict                    0
+      attributes                      0x2020
+      ghostedHeader                   0
+      data                            0
+      gvsn                            {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241
+      uid                             {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138
+      parent                          {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+      fence                           Default (3)
+      clockDecrementedInDirtyShutdown 0
+      clock                           20091007 22:06:13.298 GMT (0x1ca479a624956e1)
+      createTime                      20091007 18:46:43.088 GMT
+      csId                            {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+      hash                            00000000-00000000-00000000-00000000
+      similarity                      00000000-00000000-00000000-00000000
+      name                            Res.RWM
ß the RWM file

20091007 18:06:13.330 2680 USNC  2453 UsnConsumer::UpdateIdRecord ID record updated from USN_RECORD: ß USN update showing modification
+      USN_RECORD:
+      RecordLength:        80
+      MajorVersion:        2
+      MinorVersion:        0
+      FileRefNumber:       0x30000000000B0
+      ParentFileRefNumber: 0x10000000000C7
+      USN:                 0x6b480
+      TimeStamp:           20091007 18:06:13.298 Eastern Standard Time
+      Reason:              Close Data Extend Data Overwrite
ß modifying here, not replacing
+      SourceInfo:          0x0
+      SecurityId:          0x0
+      FileAttributes:      0x2020
+      FileNameLength:      14
+      FileNameOffset:      60
+      FileName:            Res.RWM
ß the RWM file

20091007 18:06:13.689 2808 STAG  4257 Staging::GetStageReaderOrWriter ß file gets staged
+      fid                             0x30000000000B0
+      usn                             0x6b480
+      uidVisible                      1
+      filtered                        0
+      journalWrapped                  0
+      slowRecoverCheck                0
+      pendingTombstone                0
+      internalUpdate                  0
+      dirtyShutdownMismatch           0
+      meetInstallUpdate               0
+      meetReanimated                  0
+      recUpdateTime                   20091007 22:06:13.330 GMT
+      present                         1
+      nameConflict                    0
+      attributes                      0x2020
+      ghostedHeader                   0
+      data                            0
+      gvsn                            {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241
+      uid                             {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138
+      parent                          {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+      fence                           Default (3)
+      clockDecrementedInDirtyShutdown 0
+      clock                           20091007 22:06:13.298 GMT (0x1ca479a624956e1)
+      createTime                      20091007 18:46:43.088 GMT
+      csId                            {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+      hash                            00000000-00000000-00000000-00000000
+      similarity                      00000000-00000000-00000000-00000000
+      name                            Res.RWM
ß the RWM file
+      StageReader:0000000000000000 StageWriter:000000000032A1B0 Policy:1

20091007 18:06:14.064 2808 RDCX   467 StreamToIndex RDC generate begin: (0..4), uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 fileName:Res.RWM csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} ß RDC signature calculation starts
20091007 18:09:49.903 2808 RDCX   509 StreamToIndex RDC generate end: (0..4), uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 fileName:Res.RWM csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} ß RDC signatures calculation done (took ~ 3 minutes 35 seconds)

20091007 18:09:50.341 2808 SRTR  2592 InitializeFileTransferAsyncState::~InitializeFileTransferAsyncState this:000000000032FD00
20091007 18:14:37.972 2228 OUTC  3273 OutConnectionContentSetContext::SERVER_RdcGetSignatures Sent requested RDC signatures. context:00000000003BE770 rdcState:00000000003194B0 reader:0000000000000000 level:1 offset:32826220 length:422 sizeRead:422 uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C}
ß signature data gets used to creates packages of data and is sent – this takes ~ 4 minutes 15 seconds comparing to signatures downstream and sending the data

20091007 18:14:44.113 2228 RDCX  2927 Rdc::SyncServerState::~SyncServerState RDC Need Reader Statistics: uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C}
+      TOTAL                       
+      Compression Ratio            31
+      RDC Need Size                1381790
+      Bytes sent to downstream     958704
ß Only about 1MB (!!!) of data from the 3756455865 byte file actually had to be sent to B though. Very cool RDC savings!
+      Uncompressed XPRESS blocks   0
+      Compressed XPRESS blocks     4
+      Copied XPRESS Blocks         1
+      Bytes read using async I/Os  914469

Server B:

20091007 18:14:43.826  324 XRNA   276 XpressRdcNeedAssembler::ProcessRdcNeeds All needs processed. this:0000000017384710
20091007 18:14:43.826  324 XRNA   291 XpressRdcNeedAssembler::FinalizeRpcDownload Shutting down RPC pipe thread this:0000000017384710
20091007 18:14:43.826 1436 XRNA   919 XpressRdcNeedAssembler::GetSourceRdcNeedDataThread Exiting RDC source need download thread. this:0000000017384710
ß finished getting the files bits

20091007 18:14:43.873  324 STAG   798 StageWriter::CompleteDownloadStage Completed download or stage file 241-{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138-{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241-Downloaded.frx
20091007 18:18:08.843  324 ASYN   508 AsyncUnbufferedFileWriter::Close Async WRITE Statistics:
ß all bits put together, ready to drop into real replicated folder

20091007 18:18:08.843  324 MEET  3013 Meet::InstallRename Moving contents from Installing to final destination. Attributes:0x2020 updateName:Res.RWM uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v138 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v241 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csName:RemoteInstall ß done. Took ~12 minutes of “real time”, but you saved 3.6 GB on the wire(!!!)

=======================================

INSTALL.WIM behavior:

Server A:

20091007 18:06:12.830 2680 USNC  2881 UsnConsumer::TombstoneOrDelete LDB Updating ID Record: ß install.wim is deleted and database record updated
+      fid                             0x500000000006A
+      usn                             0x6b388
+      uidVisible                      1
+      filtered                        0
+      journalWrapped                  0
+      slowRecoverCheck                0
+      pendingTombstone                0
+      internalUpdate                  0
+      dirtyShutdownMismatch           0
+      meetInstallUpdate               0
+      meetReanimated                  0
+      recUpdateTime                   20091007 21:29:31.255 GMT
+      present                         0
+      nameConflict                    0
+      attributes                      0x2020
+      ghostedHeader                   0
+      data                            0
+      gvsn                            {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v240
+      uid                             {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+      parent                          {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+      fence                           Default (3)
+      clockDecrementedInDirtyShutdown 0
+      clock                           20091007 22:06:12.642 GMT (0x1ca479a61e532f7)
+      createTime                      20091007 18:47:25.714 GMT
+      csId                            {35F6E8DD-422B-4566-B7AD-9D13338E498C}

+      hash                            5DB57C5A-C50E04D8-DEAAFED0-04B337A2
+      similarity                      3E311339-251E183B-20271C15-2C183010
+      name                            install.wim
ß the WIM

20091007 18:06:13.455 2680 USNC  3490 UsnConsumer::UidTunnelReanimate LDB Updating ID Record: ß same named/pathed file was written in (aka the delete above was really an overwrite)
+      fid                             0x600000000006A
+      usn                             0x6b5d8
+      uidVisible                      1
+      filtered                        0
+      journalWrapped                  0
+      slowRecoverCheck                0
+      pendingTombstone                0
+      internalUpdate                  0
+      dirtyShutdownMismatch           0
+      meetInstallUpdate               0
+      meetReanimated                  0
+      recUpdateTime                   20091007 22:06:12.830 GMT
+      present                         1
+      nameConflict                    0
+      attributes                      0x2020
+      ghostedHeader                   0
+      data                            0
+      gvsn                            {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v242
+      uid                             {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+      parent                          {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+      fence                           Default (3)
+      clockDecrementedInDirtyShutdown 0
+      clock                           20091007 22:06:13.408 GMT (0x1ca479a625a0788)
+      createTime                      20091007 18:47:25.714 GMT
+      csId                            {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+      hash                            00000000-00000000-00000000-00000000
+      similarity                      00000000-00000000-00000000-00000000
+      name                            install.wim
ß the WIM

20091007 18:06:13.783 2680 USNC  2450 UsnConsumer::UpdateIdRecord LDB Updating ID Record: ß WDS writes some more data to the file
+      fid                             0x600000000006A
+      usn                             0x6b6e0
+      uidVisible                      1
+      filtered                        0
+      journalWrapped                  0
+      slowRecoverCheck                0
+      pendingTombstone                0
+      internalUpdate                  0
+      dirtyShutdownMismatch           0
+      meetInstallUpdate               0
+      meetReanimated                  0
+      recUpdateTime                   20091007 22:06:13.752 GMT
+      present                         1
+      nameConflict                    0
+      attributes                      0x2020
+      ghostedHeader                   0
+      data                            0
+      gvsn                            {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v244
ß note how we’re at 244, so it’s been modified a bit more by WDS
+      uid                             {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+      parent                          {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+      fence                           Default (3)
+      clockDecrementedInDirtyShutdown 0
+      clock                           20091007 22:06:13.752 GMT (0x1ca479a628e7bdf)
+      createTime                      20091007 18:47:25.714 GMT
+      csId                            {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+      hash                            00000000-00000000-00000000-00000000
+      similarity                      00000000-00000000-00000000-00000000
+      name                            install.wim     

20091007 18:06:13.783 2680 USNC  2453 UsnConsumer::UpdateIdRecord ID record updated from USN_RECORD:
+      USN_RECORD:
+      RecordLength:        88
+      MajorVersion:        2
+      MinorVersion:        0
+      FileRefNumber:       0x600000000006A
+      ParentFileRefNumber: 0x10000000000C7
+      USN:                 0x6b6e0
+      TimeStamp:           20091007 18:06:13.752 Eastern Standard Time
+      Reason:              Close Data Extend Data Overwrite
ß data added
+      SourceInfo:          0x0
+      SecurityId:          0x0
+      FileAttributes:      0x2020
+      FileNameLength:      22
+      FileNameOffset:      60
+      FileName:            install.wim

20091007 18:06:16.502  556 RDCX  2927 Rdc::SyncServerState::~SyncServerState RDC Need Reader Statistics: uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v245 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C}
+      TOTAL
+      Compression Ratio            0
+      RDC Need Size                2660306
+      Bytes sent to downstream     2682260
ß we have to send these many bytes to server B. There is no RDC saving here, the file was scrambled by WIM format.
+      Uncompressed XPRESS blocks   0
+      Compressed XPRESS blocks     0
+      Copied XPRESS Blocks         2
+      Bytes read using async I/Os  2656520

20091007 18:06:16.517  556 ASYN  1289 AsyncUnbufferedFileReader::Close Async READ Statistics:
+      Total I/O Bytes             2656520 (3 buffers)
+      Number of Async Buffers     2
+      Size of Async Buffers       1048576
+      Number of Async I/Os        3
+      Number of Sync I/Os         0
+      Number of Pending Calls     1
+      Number of Queue Depth Empty 0
+      Number of Buffers Used      2

On Server B:

20091007 18:06:18.139 1852 INCO  6593 InConnection::LogTransferActivity Received RAWGET uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v246 fileName:install.wim connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} stagedSize:2657686 ß RDC could not be used – no matching signatures - so we do a raw copy

20091007 18:06:18.545 1852 INCO  4825 InConnection::UpdateProcessed Received Update. updatesLeft:0 processed:1 failures:0 sessionId:6 open:0 updateType:0 processStatus:0 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csId:{35F6E8DD-422B-4566-B7AD-9D13338E498C} csName:RemoteInstall update: ß final version of file replicated
+      present                         1
+      nameConflict                    0
+      attributes                      0x2020
+      ghostedHeader                   0
+      data                            0
+      gvsn                            {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v245
+      uid                             {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137
+      parent                          {7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v128
+      fence                           Default (3)
+      clockDecrementedInDirtyShutdown 0
+      clock                           20091007 22:06:13.767 GMT (0x1ca479a6290de3f)
+      createTime                      20091007 18:47:25.714 GMT
+      csId                            {35F6E8DD-422B-4566-B7AD-9D13338E498C}
+      hash                            2BAC2FE6-3735E3A6-F89F8423-37C51923
+      similarity                      1D2A1A21-262A310C-0C060426-06291D0B
+      name                            install.wim

20091007 18:06:18.498 1852 MEET  1804 Meet::InstallStep Done installing file updateName:install.wim uid:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v137 gvsn:{7E4661CA-F75E-48D6-9F23-828FE47C23C6}-v246 connId:{D3F93FEA-BF07-4CEF-8E4E-E82564BE2E7B} csName:RemoteInstall ß B has the new file. Took about 6 seconds on a fast network so this would be much slower on most WAN's. And since the file was never that big anyways, probably acceptable behavior.

So there it is. If you’re using WDS and DFSR and your servers are not connected purely by gigabit end-to-end, modifying your images and letting DFSR take care of the modification can save big time and bandwidth. And that is something you can put towards your bottom line with more productive users who can use more of that WAN for other things.

- Ned “the matchmaker” Pyle

  • Hello Ned,

    Great article!  In regards to DFSN and DFSR for WDS:  The article technet.microsoft.com/.../cc771324(WS.10).aspx suggests that you install WDS first which requires you to specify a path for WDS (typcially C:\RemoteInstall or D:\RemoteInstall).  Once WDS is installed I don't see a way to change the path (in the GUI).

    Then the article mentions that you can create a new Image Group that points to the DFS Namespace but it doesn't provide the steps to do so.  If WDS uses D:\RemoteInstall the new Image Group is created at D:\RemoteInstall\Images\NewImageGroup.

    We already have WDS configured on 3 servers and each server uses D:\RemoteInstall.  We setup a hub and spoke replication group to only replicate D:\RemoteInstall\Images and D:\RemoteInstall\WdsClientUnattend.

    With this setup the WDS MMC Console > Install Images is displaying the hidden directory "DfsrPrivate".  A DFS Namespace for Install Images would definitely remove DfsrPrivate from the view.

    Is it possible to change WDS's path from D:\RemoteInstall over to DFSN without breaking anything?  Thank you!