I/O Performance impact of running Start-DedupJob with –Priority High

I/O Performance impact of running Start-DedupJob with –Priority High

  • Comments 6
  • Likes

My name is Steven Andress and I am a Support Escalation Engineer with Microsoft’s Platforms Support Team.  This is a short blog post to alert you to a condition you might encounter when running a deduplication job using the Start-DedupJob PowerShell cmdlet. 

Start-DedupJob
http://technet.microsoft.com/en-us/library/hh848442.aspx

Start-DedupJob [-Type] <Type> [[-Volume] <String[]> ] [-AsJob] [-CimSession <CimSession[]> ] [-Full] [-InputOutputThrottleLevel <InputOutputThrottleLevel> ] [-Memory <UInt32> ] [-Preempt] [-Priority <Priority> ] [-ReadOnly] [-StopWhenSystemBusy] [-ThrottleLimit <Int32> ] [-Timestamp <DateTime> ] [-Wait] [ <CommonParameters>]

The Start-DedupJobcmdlet starts a new data deduplication job for one or more volumes. The Priority setting sets the CPU and I/O priority for the optimization job you run by using this cmdlet.  The only way to run a Deduplication job with High Priority is to use the cmdlet.  When Priority is set to High, I/O for other processes using the volume may be slowed down or even blocked.   If this is a Cluster Shared Volume (CSV), I/O to the volume from other nodes can be similarly impacted.   

Workaround:
Do not use "-Priority High" when starting dedup jobs if this is server is in production hours.  If you wish to use this switch, please ensure that it is done after hours so that productivity is not affected.

Regards,
Steven Andress
Senior Support Escalation Engineer
Microsoft Corporation

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Not the only one. ddpcli.exe enqueue /opt /vol * /priority high

  • I have been running Windows Deduplication on three servers and I do not understand why I cannot get it to use the resources available to it.

    For example, the most powerful server I have it on is a dell R520 with 12GB of ram and 2 TB of SSD raid 0. I am running the job in powershell with inputoutputthrottlelevel set to none, memory set to 80 priority set to high.

    when I watch the process through taskmanager, I see an average CPU use of between 4 and 5% and memory use of 3 GB.

    Meanwhile the task manager performance monitor shows disk read access around 70 MB second and write access about 7 MB a second.

    For reference, a straight file copy from the HDD raid array to the SSDs is over 500 MB/second.

    So the CPU is not busy, and the disk drive is not busy. The dedupe job is using less than half the memory I want it to use. Nothing else is running on the machine.

    and although I run it with -priority high, task manager shows the priority as normal. I manually set the priority to realtime (if've also run it with high and normal).

    I have the feeling that Micosoft's programmers are setting the 'palys well with others' mode on this software, even when I want it take over the machine and use all of its resources and get the job done as fast as possible.

    The situation of our target audiance of users is that they run a backup job that writes the backup file to the volume that has deduplication enabled. We set the delay days to 0, and run the dedupe which will typically process 500 GB to several terabytes a day.

    We then run our replication program (Replacador) to replicate the deduplicatoin volume to another deduplication volume off site.

    Obviously we want the deduplication process to run as fast as possible. We do see much highers speeds than Microsoft [publishes, but it is frustrating to think that the deduplicatoin optimization process is internally throttled to slow itself down somehow.

    We have also run deduplicatoin on a dell c6100 with 12 core AMD 2435 2.6 ghz processors and 48GB of RAM. It still runs with less than 4 GB of memory in use. (Windows does use the other 45 GB for disk caching.)

    We have been using Windows deduplication internally for 9 months, and it is a great product. We liked it so much we wrote the missing replication function for it. But we think it can run faster, unless there is just something we aren't considering.

    Are you guys using read ahead with 4MB chunks to get the best performance?

    If you could send us the source code with an NDA, we could try and tweak it for higher performance for you and give it back to you.

    Thanks

  • I wrote a blog post on how to make windows deduplication run faster:

    http://windowsdeduplication.com/how-to-make-windows-deduplication-go-faster/

  • According to Microsoft web page, -priority high does not set the task priority to high. Instead, it sets the priority when tasks have the -stopwhensystembusy parameter set so that this task will be at the head of the line when they start up again. I have run Windows Dedupe jobs many, many times from the command line and the task priority is never set to high by this parameter.