Deduplication is the business of compressing data without loss and this is now built into Windows Server 2012 as a role service.  The official marketing from us states that you will save somewhere between 20-70% of the space on your file servers if you implement this. If that’s sounds interesting my screencast shows how to configure and monitor it..

The clever thing about deduplication is that it’s built into NTFS, so you can apply it to any non system volume without the need for specialist storage.  There are some caveats:

• You can’t use it on the system volume
• It works on closed files so it won’t do any good against virtual hard disks or SQL Server database files for example. However you could and in some cases should turn it on inside a virtual machine and get the benefits of deduplication that way.
• The deduplication process does not happen when you write a file to disk, it runs in the background and is typically scheduled.  The default setting is to only deduplicate files older than 5 days the odds are that a file that hasn’t changed for 5 days won’t change that much again and is worth deduplicating.
• It will slow down your initial reads a bit (approx. 3%), but once a file is cached performance will improve a bit (approx. 8%)
• You’ll want to have a backup solution that is compatible with this new feature e.g. System Center Data Protection Manager 2012 SP1 as well as VEEAM and Symantec Backup Exec (with more vendors updating their software all the time)

To try this yourself all you’ll need is an Evaluation Copy of Windows Server 2012. Having got the idea you may also want to see how well it will work on your data. To do that install it turn on the deduplication e.g. in Powershell..

Add-WindowsFeature –Name “FS-Data-Deduplication”

and then copy windows\system32\ddpeval.exe and run this against a file share, volume etc.  Note that this might put some load on your network but otherwise shouldn’t be too invasive as it will run in the background (possibly for hours on a big volume) before telling you what you would save if you enabled this feature.

Finally thanks to my good friend Simon;  he has done most of the legwork in setting up deduplication for our IT camps and I have shamelessly used that for the screencast.