Thoughts from the EPS Windows Server Performance Team
From time to time, customers will call in to report "performance problems" that they are having when copying large files from one location to another. By "performance problems", they mean that the file isn't copying as fast as they expect. The most common scenario is copying large SQL databases from server to server, but this could just as easily occur with other file types. More often than not, the customer has tried different methods of copying the file including Windows Explorer, Copy, XCopy & Robocopy - with the same results. So ... what's going on here?
Assuming that you aren't experiencing network issues (and for the purposes of this article, we'll assume a healthy network), the problem lies in the way in which the copy is performed - specifically Buffered v Unbuffered Input/Output (I/O). So let's quickly define these terms. Buffered I/O describes the process by which the file system will buffer reads and writes to and from the disk in the file system cache. Buffered I/O is intended to speed up future reads and writes to the same file but it has an associated overhead cost. It is effective for speeding up access to files that may change periodically or get accessed frequently. There are two buffered I/O functions commonly used in Windows Applications such as Explorer, Copy, Robocopy or XCopy:
So looking at the definition of buffered I/O above, we can see where the perceived performance problems lie - in the file system cache overhead. Unbuffered I/O (or a raw file copy) is preferred when attempting to copy a large file from one location to another when we do not intend to access the source file after the copy is complete. This will avoid the file system cache overhead and prevent the file system cache from being effectively flushed by the large file data. Many applications accomplish this by calling CreateFile() to create an empty destination file, then using the ReadFile() and WriteFile() functions to transfer the data.
For copying files around the network that are very large, my copy utility of choice is ESEUTIL which is one of the database utilities provided with Exchange. To get ESEUTIL working on a non-Exchange server, you just need to copy the ESEUTIL.EXE and ESE.DLL from your Exchange server to a folder on your client machine. It's that easy. There are x86 & x64 versions of ESEUTIL, so make sure you use the right version for your operating system. The syntax for ESEUTIL is very simple: eseutil /y <srcfile> /d <destfile>. Of course, since we're using command line syntax - we can use ESEUTIL in batch files or scripts. ESEUTIL is dependent on the Visual C++ Runtime Library which is available as a redistributable package.
Addendum: The XCOPY /J switch was added in Win7/2008R2.
http://technet.microsoft.com/en-us/library/cc771254(WS.10).aspx
/j
Copies files without buffering. Recommended for very large files. This parameter was added introduced in Windows Server® 2008 R2.
- Aaron Maxwell
This saved my proverbial butt, I was able to reduce the time it took to copy a 96GB SAP DB from 1 1/2 hours to less than 30 minutes. Thank you for posting this!
I get this error when I try to add a printer on another computer running XP to my Windows 7 RC machine. never an explanation or reason available why it happens. The printer works fine if I plug it into the W7 machine and use it that way. Networking is not working between the computers as well. I can see the computers in the workgroup but no permissions to access them. What a nightmare this RC is!
Hiii...to all
Presently I write an application to copy the hard disk partition to another partition..So I wanna get it fast compare to the speed of CopyFile() function in Windows.
I have tried so many methods,,,all leads to failure....
Hii
I wanna copy an NTFs partition to another partition of same type and same size.And I tried with windows function
Copyfile() and it worked but slow speed is a problem.Then I did with readfile() and WriteFile() instead of Copyfile() again speed is a problem.
How can I get a better speed...??
I did the same operation in kernel mode and getting slow performance using zwCreatefile() ,zwReadfile() & zwWriteFile()...
How can I get a better speed .....?
I want to copy a hard disk partition into another partition. My source and destination partitions are of NTFs and same size. For that purpose first I did by copying all sectors and it is working, But I wanna copy only used sectors…
Then I find the used clusters by reading the FSCTL_VOLUME_BITMAP. But this one also a slow one ..I want to get better speed. And I tried to get the used clusters by using the FSCTL_GET_RETEIVAL_POINTER also. But it's a slow one.....
At last I tried the windows API CopyFile() also…But everything gives a slow performance…
I know fundamentally Kernel mode(ring 0) is slower than User mode in speed ,(even if ring 0 can access Hardware directly).....
Apart them these I tried also Asynchronous operation by setting OVERLAPPED flag in CreateFile.... getting small improvement....
And I've taken snapshot (Volume shadow copy)of the Volume and copied the files using Hobo copy method...but everything gives the same speed.....
Any idea to help...
I have used the Software Acronis Disk director suite .I exclaimed after finding it's speed......!!!!!!
Any idea to help me...to get a good speed.......???
Any links to the white papers related to this section...???
-----------------------
Thanking you
------------------------
does it copy folders as well if we do a *.*
For reference, Windows 7 xcopy now includes a /J option for unbuffered I/O. I don't see any documentation of this online....
I'm trying to copy everything on one drive to another using eseutil.
I get this error:
FAILURE: CreateFile: The filename, directory name, or volume label
here is the script from top i have modified:
@echo off
set DESTDIR=g:\
set SRCDIR=d:\
for %%f in (%SRCDIR%\*.*) do
ESEUTIL /Y %%f /D %DESTDIR%\%%~nxf
but it's not working. Any help please
i have trie to copy oracle db dumpfile 85 GB of size.
ESEUTIL is very fast
- first test with xcopy should take about 8,5 hours
- with ESEUTIL it should take 3,5 hours
the copy was interrupted at 43% with
FAILURE: GetOverlappedResult (read): Not enough server storage is available
now i try this Oracle Filecopy ocopy.exe
- Performance is not so bad 8,5 GB / 15 min
- Networkutilization is at 80 %
hope this will help someone
I don't understand that people here use ESEUTIL for copying all files, but it seems to be included only with Exchange and primarily designed for Exchange defrag and such. Is it available for download elsewhere (a microsoft site? ) . We thought we'll use for sql server file copy.
Thank you for a very valuable post. I was hitting resource limits copying a 120 GB .bkf file on Server 2003 Std 32-bit from staging to removable disk. Using eseutil takes care of the problem.
Where do you get ESEutil? ( I would prefer to not go through the 2 week delay of asking the exchange server team guys to make a copy for me.)
Is there a way to log the results of the file copy somewhere?
There doesn't appear to be a switch and ">results.log" gets created but stays empty.
Cheers
Im have weird Problem to copy from an w2003x64 to an Storage Server 2008x64 , I copied the exchange 2007 dll and everthing looks fine ... BUT I cant you network connection I get alway an "the specified network name is no longer available "
to copy something from THAT Directory OVER the NET is no Problem .... weird someone might have an solution ?
By the way the 2007 Exchangex64 eseutil does not start form an W2003x32 server ... "as expected" but where is a 32bit exchange 2007 eseutil ???
You've done a nice work with this interesting article. Thanks a lot for this work
Robocopy.exe also adds the /J option in Windows 8/Windows Server 2012 to allow unbuffered IO copies.