Windows 8 Platform Storage – Part 2

Windows 8 Platform Storage – Part 2

  • Comments 7
  • Likes

A prior blog entry began our whirlwind introductory tour of some of the more significant platform storage enhancements being introduced in Windows Server 8 – this follow-up entry concludes our introduction.

In Windows 8 Platform Storage – Part 1, we listed your storage related requirements. We also described our resultant set of assumptions guiding Windows 8 platform storage development. Finally, we had a brief introduction to some Windows 8 platform storage capabilities including:

  • Improved NTFS metadata integrity on commodity storage
  • Maximized NTFS availability through online file system scan and repair and
  • Storage Spaces – platform based storage virtualization for business critical application deployment on cost-effective commodity storage hardware

The following additional Windows 8 platform storage enhancements support objectives listed in the prior blog:

  • Data Deduplication – Windows Server 8 implements state of the art (sub-file) data chunking and compression to deliver optimal data deduplication, and thereby maximize storage capacity utilization for a variety of workloads. The implementation is transparent to the primary workload on the server – this is achieved by minimizing I/O impact through scheduled and selective optimization, and minimizing server impact through low resource (CPU/memory) usage. Further, the implementation strongly optimizes for high data integrity – this is achieved by performing data integrity validation on all data as well as metadata, and by ensuring redundancy for all metadata as well as “popular” data chunks. Windows 8 servers that enable deduplication as well as BranchCache apply the same algorithms both to on-disk and on-the-wire savings, thereby benefiting from faster download times as well as reduced bandwidth utilization over wide area networks. The below table lists a sampling of expected savings from deduplication for common workloads:

 

Workload

Content

Typical capacity savings

User Documents

Documents, Photos, Music, Videos

30-50%

Deployment Shares

Software binaries, Cab files, Symbol files

70-80%

Virtualization Libraries

Virtual Hard Disk files

80-95%

General File Share

All of the above

50-60%

  • Windows Storage Management API – a comprehensive and extensible WMI v2 based storage management API enables easier “single point” administration and management of heterogeneous storage infrastructure (including Storage Spaces). In-box PowerShell command-lets enable simplicity of end-to-end provisioning and ongoing administration, as well as inherent support for remote execution and scripting. Application developers including those delivering sophisticated storage management solutions (e.g. System Center or equivalent) can now rely on a single Windows API. Storage hardware vendors can enable easy administration of their gear either through the industry standard Storage Management Initiative Specification (SMI-S) or via a Windows host-based Storage Management Provider interface.

A comprehensive single-pane-of-glass storage management GUI utilizes this API to deliver task based, multi-machine heterogeneous storage management.

  • Offloaded Data Transfer – joint innovation with major external storage array vendors has resulted in Windows Server 8 providing an incredibly efficient mechanism to perform cross-application and cross-machine data transfers.

In situations where data is simply moved from a source location to a target and wherein no computations are performed on the dataset (e.g. file copying or virtual machine provisioning from a “golden image”), Windows Server 8 works with advanced external storage arrays to directly transfer this dataset without requiring the data to be transferred to/from the Windows servers. Obviously, this minimizes impact on host network, host CPU, or host memory and enables increased efficiencies through better external storage array and storage network utilization. In the future, we expect additional enhancements to external storage arrays that will also enable offloaded data transfers cross-storage-arrays and also across geographically distributed datacenters.

On Windows 8 servers which store data on advanced storage arrays supporting this capability, administrators can (transparently) enjoy the benefits of very fast and efficient data transfers through the simple act of “dragging and dropping” files using Windows Explorer. Further, applications have access to simple yet powerful APIs so they too can directly invoke this functionality.

  • Awareness and utilization of thinly provisioned disks – beginning with Windows Server 8, NTFS identifies thinly provisioned disks and enables transparent capacity reclamation (e.g. when files are deleted) for better capacity utilization. Further, sophisticated applications can also identify thinly provisioned disks through both WMI or via PowerShell, and can invoke commands to enable capacity reclamation when appropriate. Applications are also notified via standard events whenever capacity exhaustion thresholds are crossed on such thinly-provisioned storage, thereby enabling them to take appropriate action.
  • Storage Optimizer – Periodically, without any user intervention, a storage optimizer task completes maintenance activities including intelligent compaction of file system allocation to enable capacity reclamation on thinly provisioned disks (including on thinly provisioned spaces).

Platform storage enhancements in Windows Server 8 enable a cost-effective, scalable, and continuously available dynamic IT environment supporting business agility and flexibility. There is much more left to share about these and additional platform storage capabilities in Windows Server 8 – so stay tuned!

Rajeev Nagar

Group Program Manager – Windows Storage & File Systems

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • How is the deduplication process implemented?  Is it a post process, run on a schedule?  Or is it reliant on the underlying storage array to handle the deduplication, or can it handle it with local disks?  And the offload data transfer, I am assuming you are referring to arrays that support this feature, similar to VMware's storage API's?

  • Will these features available in Windows 8 client ?

  • With Storage Optimizer mentioned above, does that mean there is no longer need to defrag disks?

  • Great post, thanks! Is the Data Deduplication feature somewhat analogous to the algorithm SandForce uses to reduce write amplification with their SSD controller?

  • Q: With Storage Optimizer mentioned above, does that mean there is no longer need to defrag disks?

    A: The Defragment tool (now called Defragment and Optimize) has been evolved to better optimize your storage in Thinly Provisioned environments in addition to its traditional defragmentation role.  If your storage platform (such as Storage Spaces or intelligent storage array) supports it, Storage Optimizer will now consolidate lightly used ‘slabs’ of storage and release those freed storage ‘slabs’ back to your storage pool for use by other Spaces or LUNs.

  • Q: Will these features available in Windows 8 client ?

    A: With the exception of Data Deduplication, these features will be available in Windows 8 Client.

  • Deduplication is implemented as a post process which runs on a schedule and deduplicates files based on a policy defined by file age, location and type. Deduplication can be scheduled in background mode (continuously run using low resources) or in throughput mode (run during dedicated hours using more resources). It can also be triggered on-demand from Powershell.  It is not dependent on the storage array to handle the deduplication.

    For Offloaded Data Transfer, your intelligent storage array must support this feature in order to take advantage of it.