Inside Vista SP1 File Copy Improvements

re: Inside Vista SP1 File Copy Improvements

Nam — Fri, 26 Feb 2010 02:15:42 GMT

Does anybody know what Microsoft applications are using IOCTL_COPYCHUNK?

re: Inside Vista SP1 File Copy Improvements

Evan — Mon, 28 Dec 2009 02:55:57 GMT

I don't understand why vista SP2 will not save the file settings whhen an attribute is changed by administrative user, also it appears that an Administrator is not really an administrator, Windows Vista for me and my business has been the worst ever Operating System, countless headaches, non fuctional,un-user frendly, and totally useless.We and 18 other businesses in Australia are getting together to ensure Microsoft pays for the damage that has been done.A class action with mountains of evidence this will get media,you may be able to get away with this deception in other countries,but in Australia Businesses together tageting the course, rectification and results of what microsoft has done will have a massive adverse effect on the Microsoft smug attitude.

re: Inside Vista SP1 File Copy Improvements

copy — Sun, 08 Nov 2009 00:38:52 GMT

This is PATHETIC. File copying is one of the most basic operations of an OS and MS can't get it right. Have they heard of KISS principle? KISS + let user have a choice.

I'm trying to copy 13 pictures from a vista home premium PC to a vista home premium laptop over a wired network. It freezes during "calculating". Canceling causes the entire laptop become nonresponsive. Can't even shutdown gracefully. All the fancy sh*t sounds great but it's a typical programming pitfall. "It's not as simple as it sounds" only because you're doing it wrong!

Seriously, does the MS team doing all these fancy research on copying algorithm realize how very pathetic this is?

The milisecond another OS has decent UI and device support, I'm dumping MS. Been waiting for this day for 20 years now, unfortunately.

re: Inside Vista SP1 File Copy Improvements

Rob — Thu, 10 Sep 2009 09:43:09 GMT

Just to let you know Mark this post helped me. Thanks.

re: Inside Vista SP1 File Copy Improvements

MichaelG — Tue, 08 Sep 2009 13:36:09 GMT

First of all thanks to Mark. It's interesting and usefull.

Other,

I see a lot of undocumented DeviceIoControl(s) inside Vista trace copy file operation:

1. Device:0x14, Functions:260-262, Method:0

2. FSCTL_LMR_QUERY_DEBUG_INFO

3. IOCTL_LMR_DISABLE_LOCAL_BUFFERING (named above).

The single found info about is IOCTL_COPYCHUNK in MSDN, which is defined very pure.

I'd ask Mark, if possible, to describe how it works...

Overhead outside the transfer operation

Chris Quirke — Thu, 11 Jun 2009 11:23:48 GMT

If predicted theory isn't working, after studying exactly what happens during file operations, then perhaps there are other overheads at work.

The obvious one is resident av scanning, but there may be background processes that are triggered whenever the contents of a folder are changed; thumbnailers, indexers, SR/Previous Versions, etc.

Are those optimised to back off for long enough, so that they are not triggered into rebuilding their stuff every time a new file is added to a destination, or deleted from a source?

UI to balance copy vs. other tasks?

Chris Quirke — Thu, 11 Jun 2009 11:17:31 GMT

When there's a difficult trade-off decision to make, the best thing to do may be to leave users the opportunity to control this.

A visible way would be a slider that appears on the file operations dialog, when a copy or move operation will take over (say) 5 seconds. Pull the slider towards File Operations, and more memory is allocated to caching these; pull the slider towards Running Programs, cache memory is reduced to favor other running tasks. Effects to end with that operation (so that small copy ops don't "inherit" these settings with no UI to change them).

A less visible "power user" approach would be similar to holding down Shift when clicking No to file operation stalls (which has the "always No" effect) or when deleting, to bypass the 'bin. Hold Shift down when clicking Copy, Move or Paste, and more memory will be allocated to that operation to speed it up, XP style.

Perhaps add as an enhancement to W7 SP1, or as a Power Toy if the plumbing doesn't go too deep?

re: Inside Vista SP1 File Copy Improvements

Clovis — Sat, 09 May 2009 13:38:37 GMT

As someone said over a year ago, a file copy engine shouldn't be any more complex than open file, read, write, [read, write, [...]], close file. The system should handle the workload. That's what it's there for. Linux manages this, Mac OSX manages this, heck, DOS managed it.

So either the Vista algorithm is so over-complex and can't be simplified because of the egos involved (and I don't mean Mark R. here), or the Vista copy does something additional with the file contents during the copy.

Hopefully for Windows users this will be fixed in Windows 7.

re: Inside Vista SP1 File Copy Improvements

zzzy — Mon, 04 May 2009 02:16:17 GMT

It so goes that my Vista (client) would crash the entire XP (server) TCPIP stack when accessing these 3GB+ files. It would happen randomly but consistently enough to mine any attempts at completing one task or another.

Having tried all possible tricks on both sides, including:

(XP/server side)

- driver updates,

- increasing the Lanman IRPStackSize,

- fine tuning the NdisWanMTU,

- patching TCPIS.SYS to allow 50 concurrent half connects,

(Vista/client side)

- driver updates

- disabling TCP autotune (see above comment from Tim Bolton from Dec. 2 2008)

- fine tuning NdisWanMTU

- tweaking the NetworkThrottlingIndex,

I had to concede and consider I may be facing a network driver issue.

All until I had the great idea of installing Lighttpd on the XP/server side, and using Firefox as download manager on the Vista/client side.

Suddenly, not only did my transferred files become restartable, but no more crashes/errors/disconnects occurred. At the same time transfer rates improved from ~2200 kbps (Vista file copy between crashes) to ~2.1 MBps (please note capital B).

I have to conclude now that all the "fine tuning" of Vista file copy Microsoft folks have been fussing over turned up c**p (sorry Mark R., I really respect *your* work). It could very much be that file copy in both XP *and* Vista is irrevocably broken which is equally fine by me.

A decade of "fine tuning" by a well endowed R&D team couldn't beat work done by volunteer enthusiasts, both with repect to stability and performance.

Cheers

File copy under XP

dirbase — Wed, 08 Apr 2009 12:11:57 GMT

Sorry to comment again, but it seems to me that the following sentence in the paragraph "File Copy in Previous Versions of Windows" may require some precision:

"Explorer’s first read operation at event 0 of data that’s not present in memory causes the Cache Manager to perform a non-cached I/O, which is an I/O that reads or writes data directly to the disk *without caching it in memory*, to fetch the data from disk at event 1, as seen in the stack trace for event 1.In the stack trace, Explorer’s call to ReadFile is at frame 22 in its BaseCopyStream function and the Cache Manager invokes the non-cached read indirectly by touching the memory mapping of the file and causing a page fault at frame 8."

Looking at the CcCopyRead function, AFAIU, the missing page in physical memory is read from disk and copied *both* to the cache buffer and the user buffer (so it's indeed read directly from the disk into the user buffer but it also ends up in the system cache).

This is different from the case of a read-ahead, where the function CcPerformReadAhead, for a missing page in physical memory, will only copy the data from the disk to the cache buffer and not to the user buffer.