Slow Large File Copy Issues

re: Slow Large File Copy Issues

nitink — Tue, 08 May 2007 17:09:04 GMT

What ROLE would IRPStackSize play in such scenarios.. I know that usually tweaking this reg value would work-around similar problems

Copying Large Files

Carpe Diem: Flaphead.com @ Home — Tue, 08 May 2007 20:10:52 GMT

This is an interesting post from the performance team blog ... bottom line, if your copying a really

re: Slow Large File Copy Issues

Merckury — Tue, 08 May 2007 22:18:30 GMT

you can also use esefile.exe from the same place on the Exchange Server.

re: Slow Large File Copy Issues

Luke — Tue, 08 May 2007 23:24:04 GMT

Has anybody done any testing to see how much quicker eseutil would be compared to robocopy?

re: Slow Large File Copy Issues

Billy Dhillon — Wed, 09 May 2007 18:45:28 GMT

Great Article Aaron!

I think this is really good to point out because when you are doing large file copies using anything with Buffered I/O, you can actually exhaust your resoruces.

Buffered I/O will consume Paged Pool (1 KB of paged pool is required for each megabyte MB of file size that is opened for buffered I/O)

So the Bigger your File Copy, The more Paged Pool you will consume and eventually your Copy might fail if your resource runs out before your copy Finishes.

Thats why its good to use tools such as ESEUTIL that don't utilize CopyFile or CopyFileEx for large file copies.

re: Slow Large File Copy Issues

Rex2002 — Fri, 11 May 2007 16:06:46 GMT

I test it for BKF files... 2Gb aprox.

The Copy And Paste last 3 minutes

The eseutil takes over 6 minutes :(

re: Slow Large File Copy Issues

Easyshizz — Tue, 15 May 2007 21:54:15 GMT

I like wget the best and it's gpl'd

http://www.gnu.org/software/wget/index.html#downloading

ESEUTIL to copy files?

subject: exchange — Tue, 22 May 2007 10:17:54 GMT

Every Exchange Administrator knows what ESEUTIL does. It can be used to defragment the Exchange databases

re: Slow Large File Copy Issues

Mauro Kirsch — Tue, 29 May 2007 16:37:23 GMT

Great Article,

Remembering that the option of copy files the ESEUTIL alone is available on Exchange Server 2003, on Exchange 2000 Sever it does not have this option.

regards,

Mauro Kirsch

mkirsch@br.ibm.com

re: Slow Large File Copy Issues

Chris Portman — Mon, 02 Jul 2007 16:03:15 GMT

I was able to copy a 250 gig database file in just over an hour using ESEUTIL. The same copy took 4 hours using robocopy. What a great tool!

re: Slow Large File Copy Issues

Kevin Floyd — Wed, 11 Jul 2007 06:05:30 GMT

This does one file at a time I assume? I can't specify an entire directory, for instance an oracle directory where there are 70 or 80 database files, all roughly 2 gig per DB part.

re: Slow Large File Copy Issues

John W — Thu, 12 Jul 2007 01:17:09 GMT

Kevin:

ESEUTIL does one file at a time. Not a big problem though; just connect it to the for command.

Ex: for %f in (d:\source\directory\*.MDF) do ESEUTIL /Y "%f"

re: Slow Large File Copy Issues

Bryan — Thu, 12 Jul 2007 02:44:11 GMT

Aaron:

You really really REALLY oughta suggest this to the robocopy maintainers!

$0.02 ;)

re: Slow Large File Copy Issues

Dave Sucsy — Fri, 13 Jul 2007 06:33:57 GMT

This is CRAZY!!! I need to get work done. I'm a pro photographer and move large amounts of files around on internal, external (USB and 1394) and network drives. WinXP would move files on internal drives as fast as the drives could go. Vista (the albatross) cuts my file transfer speeds to between 1/4 and 1/10 XP speeds. WHY does Microsoft do this crazy stuff??? I have spent days researching and optimizing Vista to try to get it to where XP was as an out-of-the-box product, and Vista is still doing an excellent job of thwarting all of my attempts at productivity. Are you trying to get me to abandon Microsoft entirely and go Linux or Apple?

re: Slow Large File Copy Issues

Dave Sucsy — Fri, 13 Jul 2007 06:34:02 GMT

re: Slow Large File Copy Issues

Rob — Thu, 26 Jul 2007 00:52:31 GMT

Holy CRAP! I can't thank you enough for this. I thought for a long time that my Promise SX4000 IDE RAID5 was my performance killer. I tend to transfer large DVD ISOs.

Using copy, I got about 5MB/sec writes over gigabit. Server computer was essentially useless until the copy completed.

Using eseutil, I get about *40*MB/sec!

I have been searching for the cause of this issue for DAYS and finally stumbled on your post. I fought with the gigabit network card, and system cache tweaking, all to no avail.

Computer is still pretty bogged down during the transfer. The system cache jumps to 750MB (out of the system's 1GB) usage which is the probable culprit. But now it takes less than 1/8th the time to move the data, so the whole situation is a lot more tolerable. I eventually figured out that the problem only was with a network file copy and not a local transfer.

re: Slow Large File Copy Issues

Magnus Rasmussen — Thu, 02 Aug 2007 22:27:59 GMT

Any answer from microsoft on this issue? Same in both vista x64 and vista x86. Large files. Tried 8-20GB files. Copy is fast but consumes all memory on the sending machine.

re: Slow Large File Copy Issues

Clyde Tilley — Wed, 08 Aug 2007 23:49:52 GMT

I found this solution to SLOW file copy/delete/move:

Open Programs & Features

Select Turn Windows Features On or Off

De-select Remote Differential Compression

Select OK

It will take several minutes for this to be de-activated.

Reboot to be sure it takes.

Now these operations fly!

re: Slow Large File Copy Issues

Martin Cottingham — Thu, 13 Sep 2007 11:01:09 GMT

I am copying 40 2Gb files every night & all other attempts have failed, so far this seems to be working great. Who says that the internet is full of rubbish.

The bottom line for Windows though is that you shouldn't have to muck about like this, it should just work. I am copying between 2 Windows Server 2003 R2 machines, this is basic stuff.

Even with this command though I am only getting 20% throughput on a 1Gb network, no other devices connected.

re: Slow Large File Copy Issues

Martin Cottingham — Thu, 13 Sep 2007 11:53:37 GMT

I have now found Teracopy. Free, faster that eseutil & simple:

http://www.codesector.com/teracopy.asp

re: Slow Large File Copy Issues

Jack Farrel — Tue, 18 Sep 2007 11:34:24 GMT

I had just the same issues. Every "big data" copy was very slow. I've tried a lot of different tools and stopped at secure copy. http://www.scriptlogic.com/products/securecopy/ Ultra fast copying because of multithreaded technology, stability work, friendly gui interface, retaining shares and permissions during copying - those are only a few advantages of this solution. Hope it helps.

re: Slow Large File Copy Issues

Poul Erik — Tue, 30 Oct 2007 12:25:17 GMT

Are there any guide lines for how much free space you need on System drive too copy a 100GB file from e: to F:. When I try on a system with 35 GB free space on the system drive if fails even though the copy is not to the C drive but from E to F. Any one?

Large File Copy Issues

Jeff Hisson — Wed, 31 Oct 2007 00:10:34 GMT

Yeah well I beat my head against the wall trying to copy large files(>33GB) and directory permissions....essentially trying to migrate a basic volume to another larger drive. I tried Robocopy but the files corrupted. XXCopy succedded in copying a single large file but failed on the directory permissions. Swing volume using MSBackup stalled...anybody have any additional freeware utils?

re: Slow Large File Copy Issues

Ivan — Thu, 22 Nov 2007 19:11:03 GMT

Recently i tried to burn 506mb of different files to dvd... same thing. it took me cca. 20 minutes, and (IIRC) the same operation takes 3-5 minutes on xp. please hurry with the sp1 :D

re: Slow Large File Copy Issues

Eric Ericson — Thu, 06 Dec 2007 01:57:31 GMT

I'm really pleased I found this little nugget. I was banging my head up against the disk buffer while trying to move a 400GB SQLServer mdf file from one disk shelf to another.

I never waited to completion on the robocopy approach, but just from the back of the napkin calculations I'm seeing so far it look like ESEUtil is about 3x the speed.

re: Slow Large File Copy Issues

Tom — Tue, 11 Dec 2007 07:36:50 GMT

try this link I went from a 2.3m to 34m just with the patch on the site http://securevista.net/

re: Slow Large File Copy Issues

Chris — Fri, 21 Dec 2007 19:34:29 GMT

I have tested copying a 4 gig file using ESEUTIL, RoboCopy, command-line copy and Windows Explorer Cut and Paste. Windows Explorer Cut and Paste was fastest.

Windows 2003 EE SP2

re: Slow Large File Copy Issues

Adam — Fri, 18 Jan 2008 23:57:19 GMT

4 GB is really not that large a file. If your machine has 2GB or more of memory, a buffered I/O scenario should probably work fine. However, try copying 5 GB files at a time, while doing other operations on the computer that take up paged locked memory.

re: Slow Large File Copy Issues

Warren — Sat, 19 Jan 2008 05:18:01 GMT

I just ran into a situation with Eseutil where I am trying to copy a 100gb file from a 3 disk RAID 5 array to an external eSata hard drive. The file copy gets to about 70% and then the RAID controller goes all wonky and basically disconnects from the system until the next reboot. We then need to add another drive to the array, rebuild the drive and then re-add the supposed bad hard drive back into the array. In other words, the drives are fine; it's the controller that gives out.

I am concluding that the RAID controller that I am trying to use is buggy. I have tried another controller from the same manufacturer with the same results.

Eseutil is obviously copying the file so fast that it is triggering a bug in the controller. Eseutil has helped me identify this bug. Jury is still out as to whether this will happen when I use Robocopy or another file copy program.

The controller in question is a:

Highpoint RocketRaid 1640

Hopefully somebody finds this information useful.

re: Slow Large File Copy Issues

jim esposito — Tue, 12 Feb 2008 19:17:41 GMT

Need the ESEUTIL copy to overwite the exisitng file.

Any advice???

re: Slow Large File Copy Issues

Mark Pleasance — Fri, 20 Jun 2008 19:47:20 GMT

Hi Everyone,

Using robocopy (XP010) for a dozen MS SQL2000 DB backup files (and logs) . . . ~50GB-150GB (pending on what time of the month it is . . . ;0) in a batch file mapping drive letters to remote shared folders.

During the copy of the largest single file I encountered the error "ERROR 1130 (0x0000046A) Copying File <insert file name here> Not enough server storage is available to process this command. Waiting 30 seconds... Retrying..."

Tried the same copy with windows explorer . . . One file at a time would copy OK . . . but the "not enough server storage" error would occur if I tried copying ALL the files at once.

We fixed the issue with simple configuration / tuning changes . . . here's the list we used . . .

On the receiving (Windows 2003) server . . .

1.) latest greatest service pack / updates

2.) Local area connetion properties -> File and Printer sharing for Microsoft networks (properties) -> changed "Server Optimization" to "Maximize data throughput for file sharing" (was set to Net Apps)

3.) System Properties -> Advanced Tab -> Advanced Tab - Processor scheduling - Adjust for best performance of: Change to "Background services" (was set to "Programs")

4.) Set swap file to recomended parameters / size

5.) Reboot . . .

6.) Test, Test, Test, Test . . .

Hopefully someone finds this information usefull . . .

re: Slow Large File Copy Issues

Jorge Eston — Thu, 17 Jul 2008 16:37:23 GMT

hi everybody!

This is one of the best places to get answers.

I found all comments with usefull information.

Thanks

Jorge

re: Slow Large File Copy Issues

Prashant — Wed, 03 Sep 2008 22:30:46 GMT

I need to copy 150+ gb sql db backup files over the same network. Please suggest what will be the best tool to use. the backup is a sharepoint db SQL 2000 backup. The Destination and Source are Both 2003 servers

re: Slow Large File Copy Issues

karnash — Sun, 28 Sep 2008 01:08:51 GMT

Hi,

can anyone please tell me how can I get ESEUTIL.EXE and ESE.DLL if I do not have exchange server. I can't copy a 400G file. thanks

re: Slow Large File Copy Issues

karnash — Sun, 28 Sep 2008 01:25:25 GMT

I used teracopy and after 18.5% I get the error ERROR 1130 (0x0000046A) Copying File and free space is 1.2T

re: Slow Large File Copy Issues

Darryl — Wed, 01 Oct 2008 23:39:25 GMT

Recommend KB 304101, but before that, research c:\boot.ini settings /3GB and /PAE. If you've got those settings because your server has more than 4gigs memory, they can easily cause 1130 errors. Still researching to choose best solution: probably using /userva=2900 in boot.ini, but we'll see what works.

re: Slow Large File Copy Issues

Tom Bergman — Mon, 06 Oct 2008 18:08:24 GMT

Since I found this thread so helpful, I thought I'd add a few comments.

ESEUTIL works great for copying 10gb+ files on our network. It's much faster that robocopy, xcopy, teracopy etc.

I didn't have access to an exchange server so I downloaded an Exchange service pack, ran the exe with the -x option to unpack it, and then searched for and found eseutil.exe and ess.dll.

Here's a short script that I'm using to copy files I have with a .bku extension. Oviously this can be easily modified to fit your needs.

@echo off

set DESTDIR=\\servername\sharename

set SRCDIR=f:\backupfiles

for %%f in (%SRCDIR%\*.bku) do (

del %DESTDIR%\%%~nxf

ESEUTIL /Y %%f /D %DESTDIR%\%%~nxf

)

re: Slow Large File Copy Issues

Jonathan Hays — Fri, 10 Oct 2008 19:22:17 GMT

Such a great post - and a great way to increase performance on large files. We copy many livestate images from a terastation to a USB drive, and the copy itself takes 3 hours for 31 GB, the ESEUTIL took 45 minutes for the same file. Great research and result!

re: Slow Large File Copy Issues

Gary Jackson — Sun, 07 Dec 2008 20:14:28 GMT

Thank you for the great information. I have tested this using a 355GB file between 2 servers and have had excellent results, about 2.1 hours.

The question I have is that I cannot get the throughput to the max of the network. I am using gigabit between servers in their own dedicated vlan so there is no traffic on the line. The max I can push the network is about 45%. Is there a way to optimize this to get it to copy even faster or is this a limitation of eseutil.

Thanks again.

re: Slow Large File Copy Issues

Mark Sear — Thu, 11 Dec 2008 12:24:45 GMT

Using CreateFile, ReadFile and WriteFile does not seem to help unless the unbeffered options are set when the file is opened. After about 60GB Windows just gets slower and slower and eventually uses up all of the system resources. The problem is difficult to deal with if using third-party software as it is not practicable to change how the output is created.

re: Slow Large File Copy Issues

SM — Thu, 18 Dec 2008 23:55:58 GMT

All this information is great, but still doesn't address what I've been seeing... 2 different scenerios:

WinVista - even w/SP1, any type of file operations appear to dog it... Vista quite often doesn't respond to file operations properly... this is small/large, you name it... any further fixes post SP1 ? (I've also disabled Differential Compression, made a world of difference for a little while, but still not the performance of XP or Win2003...)... Vista just plain sucks! If it wasn't that I was running it for Media Center, I'd say chunck it... odd part is, if you reload the OS, it runs fine, until you try to load just a few little apps, and actually try to make it a "working system" ... all virus scanned, and same files come from other running system... I know os's extremely well, and haven't been able to locate this one yet...

2nd one on Win2003 - issue w/renaming a directory, moving it, etc... it has about 100,000 files in it (yes, they have to be there)... win2003 hangs for 5-10 mins on a dual proc quad core enterprise server.... only thing I've been able to locate so far to help I have 1 directory w/about 35,000 files in it... if I move everything excluding that dir, it's done in seconds... if I move that dir in it's entirety, it's quick, if I move it as one (~30,000 files or more at a time), it hangs... same if I move more than ~30,000 files in multiple dirs... some type of MS limitation or caching issue?

re: Slow Large File Copy Issues

Dantek — Sun, 21 Dec 2008 03:37:15 GMT

Great stuff with ESEUTIL. Never even knew it could be used to just copy.

Copying a 60 GB Exchange db took 30 minutes as opposed to waiting 2 hrs with Explorer just to tell me "Not enough server storage is available ..." And then waiting another 2 hours and hoping that the dreaded error mysteriously vanishes

Thanks a bunch for this info.

re: Slow Large File Copy Issues

Dantek — Sun, 21 Dec 2008 04:11:37 GMT

Spoke too soon.

The dreaded error showed up at 95%, only it took 30 minutes instead of 2 hours.

Shut down a bunch of services and trying it again.

re: Slow Large File Copy Issues

Ronnie Walker — Thu, 12 Feb 2009 12:36:35 GMT

Find that the ESUTIL copy is great for SQL Database backups however running into a weird issue when attempting to copy VMWare image backups to another location.

ESLUTIL appears to be changing the file size at the destination and as such the image backup is useless and fails when you try to use it.

It appears to be adding to the initial file size....but no errors are generated when copying takes place. You can see immediately that the file size does not match each time this is done!!!

Anyone else came across this issue and how was it resolved?

Thanks

Ronnie

re: Slow Large File Copy Issues

Corinna — Thu, 19 Feb 2009 07:02:09 GMT

This is a great article explaining what's going on in the copy process. Unfortunately, the eseutil did not work for us. Tests showed that SQL server process memory was still being paged out.

I ended up using rsync (from Cygwin).

See my post here for more information:

http://corinnalo.blogspot.com/2009/02/sql-memory-paged-out-during-large-file.html

- corinna

re: Slow Large File Copy Issues

David — Wed, 04 Mar 2009 22:13:14 GMT

Remember the good ol' days when a 486 could saturate a fast network connection doing a file transfer?

A little bit of buffering goes a long way. But the MS opinion is all or nothing. Either flushing the cache, or using slow unbuffered i/o. No concept of a middle ground.

You'd think after all these decades they could get file copying to work right. What does the performance team do if not work on fixing these sorts of problems?

re: Slow Large File Copy Issues

Sergey — Fri, 20 Mar 2009 02:39:14 GMT

Hmm.. I ntoiced on some machines I have that CopyFileEx with restartable flag is ~9 times slower than CopyFile.

re: Slow Large File Copy Issues

pdxRick — Tue, 02 Jun 2009 21:09:21 GMT

This saved my proverbial butt, I was able to reduce the time it took to copy a 96GB SAP DB from 1 1/2 hours to less than 30 minutes. Thank you for posting this!

re: Slow Large File Copy Issues

detexb — Wed, 03 Jun 2009 20:20:40 GMT

I get this error when I try to add a printer on another computer running XP to my Windows 7 RC machine. never an explanation or reason available why it happens. The printer works fine if I plug it into the W7 machine and use it that way. Networking is not working between the computers as well. I can see the computers in the workgroup but no permissions to access them. What a nightmare this RC is!

re: Slow Large File Copy Issues

Krishnampkkm — Tue, 30 Jun 2009 13:11:50 GMT

Hiii...to all

Presently I write an application to copy the hard disk partition to another partition..So I wanna get it fast compare to the speed of CopyFile() function in Windows.

I have tried so many methods,,,all leads to failure....

Hii

I wanna copy an NTFs partition to another partition of same type and same size.And I tried with windows function

Copyfile() and it worked but slow speed is a problem.Then I did with readfile() and WriteFile() instead of Copyfile() again speed is a problem.

How can I get a better speed...??

I did the same operation in kernel mode and getting slow performance using zwCreatefile() ,zwReadfile() & zwWriteFile()...

How can I get a better speed .....?

I want to copy a hard disk partition into another partition. My source and destination partitions are of NTFs and same size. For that purpose first I did by copying all sectors and it is working, But I wanna copy only used sectors…

Then I find the used clusters by reading the FSCTL_VOLUME_BITMAP. But this one also a slow one ..I want to get better speed. And I tried to get the used clusters by using the FSCTL_GET_RETEIVAL_POINTER also. But it's a slow one.....

At last I tried the windows API CopyFile() also…But everything gives a slow performance…

I know fundamentally Kernel mode(ring 0) is slower than User mode in speed ,(even if ring 0 can access Hardware directly).....

Apart them these I tried also Asynchronous operation by setting OVERLAPPED flag in CreateFile.... getting small improvement....

And I've taken snapshot (Volume shadow copy)of the Volume and copied the files using Hobo copy method...but everything gives the same speed.....

Any idea to help...

I have used the Software Acronis Disk director suite .I exclaimed after finding it's speed......!!!!!!

Any idea to help me...to get a good speed.......???

Any links to the white papers related to this section...???

-----------------------

Thanking you

------------------------

re: Slow Large File Copy Issues

rob — Mon, 27 Jul 2009 04:38:10 GMT

does it copy folders as well if we do a *.*

re: Slow Large File Copy Issues

SamL — Wed, 18 Nov 2009 00:34:24 GMT

For reference, Windows 7 xcopy now includes a /J option for unbuffered I/O. I don't see any documentation of this online....

re: Slow Large File Copy Issues

forumware — Wed, 25 Nov 2009 22:28:23 GMT

I'm trying to copy everything on one drive to another using eseutil.

I get this error:

FAILURE: CreateFile: The filename, directory name, or volume label

here is the script from top i have modified:

@echo off

set DESTDIR=g:\

set SRCDIR=d:\

for %%f in (%SRCDIR%\*.*) do

ESEUTIL /Y %%f /D %DESTDIR%\%%~nxf

but it's not working. Any help please