Welcome to TechNet Blogs Sign in | Join | Help

Inside Vista SP1 File Copy Improvements

Windows Vista SP1 includes a number of enhancements over the original Vista release in the areas of application compatibility, device support, power management, security and reliability. You can see a detailed list of the changes in the Notable Changes in Windows Vista Service Pack 1 whitepaper that you can download here. One of the improvements highlighted in the document is the increased performance of file copying for multiple scenarios, including local copies on the same disk, copying files from remote non-Windows Vista systems, and copying files between SP1 systems. How were these gains achieved? The answer is a complex one and lies in the changes to the file copy engine between Windows XP and Vista and further changes in SP1. Everyone copies files, so I thought it would be worth taking a break from the “Case of…” posts and dive deep into the evolution of the copy engine to show how SP1 improves its performance.

 

Copying a file seems like a relatively straightforward operation: open the source file, create the destination, and then read from the source and write to the destination. In reality, however, the performance of copying files is measured along the dimensions of accurate progress indication, CPU usage, memory usage, and throughput. In general, optimizing one area causes degradation in others. Further, there is semantic information not available to copy engines that could help them make better tradeoffs. For example, if they knew that you weren’t planning on accessing the target of the copy operation they could avoid caching the file’s data in memory, but if it knew that the file was going to be immediately consumed by another application, or in the case of a file server, client systems sharing the files, it would aggressively cache the data on the destination system.

 

File Copy in Previous Versions of Windows

In light of all the tradeoffs and imperfect information available to it, the Windows file copy engine tries to handle all scenarios well. Prior to Windows Vista, it took the straightforward approach of opening both the source and destination files in cached mode and marching sequentially through the source file reading 64KB (60KB for network copies because of an SMB1.0 protocol limit on individual read sizes) at a time and writing out the data to the destination as it went. When a file is accessed with cached I/O, as opposed to memory-mapped I/O or I/O with the no-buffering flag, the data read or written is stored in memory, at least until the Memory Manager decides that the memory should be repurposed for other uses, including caching the data of other files.

 

The copy engine relied on the Windows Cache Manager to perform asynchronous read-ahead, which essentially reads the source file in the background while Explorer is busy writing data to a different disk or a remote system. It also relied on the Cache Manager’s write-behind mechanism to flush the copied file’s contents from memory back to disk in a timely manner so that the memory could be quickly repurposed if necessary, and so that data loss is minimized in the face of a disk or system failure. You can see the algorithm at work in this Process Monitor trace of a 256KB file being copied on Windows XP from one directory to another with filters applied to focus on the data reads and writes:

 

 

Explorer’s first read operation at event 0 of data that’s not present in memory causes the Cache Manager to perform a non-cached I/O, which is an I/O that reads or writes data directly to the disk without caching it in memory, to fetch the data from disk at event 1, as seen in the stack trace for event 1:

 

 

In the stack trace, Explorer’s call to ReadFile is at frame 22 in its BaseCopyStream function and the Cache Manager invokes the non-cached read indirectly by touching the memory mapping of the file and causing a page fault at frame 8.

 

Because Explorer opens the file with the sequential-access hint (not visible in trace), the Cache Manager’s read-ahead thread, running in the System process, starts to aggressively read the file on behalf of Explorer at events 2 and 3. You can see the read-ahead functions in the stack for event 2:

 

 

You may have noticed that the read-ahead reads are initially out of order with respect to the original non-cached read caused by the first Explorer read, which can cause disk head seeks and slow performance, but Explorer stops causing non-cached I/Os when it catches up with the data already read by the Cache Manager and its reads are satisfied from memory.  The Cache Manager generally stays 128KB ahead of Explorer during file copies.

 

At event 4 in the trace, Explorer issues the first write and then you see a sequence of interleaved reads and writes. At the end of the trace the Cache Manager’s write-behind thread, also running in the System process, flushes the target file’s data from memory to disk with non-cached writes.

 

Vista Improvements to File Copy

During Windows Vista development, the product team revisited the copy engine to improve it for several key scenarios. One of the biggest problems with the engine’s implementation is that for copies involving lots of data, the Cache Manager write-behind thread on the target system often can’t keep up with the rate at which data is written and cached in memory. That causes the data to fill up memory, possibly forcing other useful code and data out, and eventually, the target’s system’s memory to become a tunnel through which all the copied data flows at a rate limited by the disk.  

 

Another problem they noted was that when copying from a remote system, the file’s contents are cached twice on the local system: once as the source file is read and a second time as the target file is written. Besides causing memory pressure on the client system for files that likely won’t be accessed again, involving the Cache Manager introduces the CPU overhead that it must perform to manage its file mappings of the source and destination files.

 

A limitation of the relatively small and interleaved file operations is that the SMB file system driver, the driver that implements the Windows remote file sharing protocol, doesn’t have opportunities to pipeline data across high-bandwidth, high-latency networks like WLANs. Every time the local system waits for the remote system to receive data, the data flowing across the network drains and the copy pays the latency cost as the two systems wait for the each other’s acknowledgement and next block of data.

 

After studying various alternatives, the team decided to implement a copy engine that tended to issue large asynchronous non-cached I/Os, addressing all the problems they had identified. With non-cached I/Os, copied file data doesn’t consume memory on the local system, hence preserving memory’s existing contents. Asynchronous large file I/Os allow for the pipelining of data across high-latency network connections, and CPU usage is decreased because the Cache Manager doesn’t have to manage its memory mappings and inefficiencies of the original Vista Cache Manager for handling large I/Os contributed to the decision to use non-cached I/Os. They couldn’t make I/Os arbitrarily large, however, because the copy engine needs to read data before writing it, and performing reads and writes concurrently is desirable, especially for copies to different disks or systems. Large I/Os also pose challenges for providing accurate time estimates to the user because there are fewer points to measure progress and update the estimate. The team did note a significant downside of non-cached I/Os, though: during a copy of many small files the disk head constantly moves around the disk, first to a source file, then to destination, back to another source, and so on.

 

After much analysis, benchmarking and tuning, the team implemented an algorithm that uses cached I/O for files smaller than 256KB in size. For files larger than 256KB, the engine relies on an internal matrix to determine the number and size of non-cached I/Os it will have in flight at once. The number ranges from 2 for files smaller than 2MB to 8 for files larger than 8MB. The size of the I/O is the file size for files smaller than 1MB, 1MB for files up to 2MB, and 2MB for anything larger.

 

To copy a file 16MB file, for example, the engine issues eight 2MB asynchronous non-cached reads of the source file, waits for the I/Os to complete, issues eight 2MB asynchronous non-cached writes of the destination, waits again for the writes to complete, and then repeats the cycle. You can see that pattern in this Process Monitor trace of a 16MB file copy from a local system to a remote one:

 

 

While this algorithm is an improvement over the previous one in many ways, it does have some drawbacks. One that occurs sporadically on network file copies is out-of-order write operations, one of which is visible in this trace of the receive side of a copy:

 

 

Note how the write operation offsets jump from 327,680 to 458,752, skipping the block at offset 393,216. That skip causes a disk head seek and forces NTFS to issue an unnecessary write operation to the skipped region to zero that part of the file, which is why there are two writes to offset 393,216. You can see NTFS calling the Cache Manager’s CcZeroData function to zero the skipped block in the stack trace for the highlighted event:

 

 

A bigger problem with using non-cached I/O is that performance can suffer in publishing scenarios. If you copy a group of files to a file share that represents the contents of a Web site for example, the Web server must read the files from disk when it first accesses them. This obviously applies to servers, but most copy operations are publishing scenarios even on client systems, because the appearance of new files causes desktop search indexing, triggers antivirus and antispyware scans, and queues Explorer to generate thumbnails for display on the parent directory’s folder icon.

 

Perhaps the biggest drawback of the algorithm, and the one that has caused many Vista users to complain, is that for copies involving a large group of files between 256KB and tens of MB in size, the perceived performance of the copy can be significantly worse than on Windows XP. That’s because the previous algorithm’s use of cached file I/O lets Explorer finish writing destination files to memory and dismiss the copy dialog long before the Cache Manager’s write-behind thread has actually committed the data to disk; with Vista’s non-cached implementation, Explorer is forced to wait for each write operation to complete before issuing more, and ultimately for all copied data to be on disk before indicating a copy’s completion. In Vista, Explorer also waits 12 seconds before making an estimate of the copy’s duration and the estimation algorithm is sensitive to fluctuations in the copy speed, both of which exacerbate user frustration with slower copies.

 

SP1 Improvements

During Vista SP1’s development, the product team decided to revisit the copy engine to explore ways to improve both the real and perceived performance of copy operations for the cases that suffered in the new implementation. The biggest change they made was to go back to using cached file I/O again for all file copies, both local and remote, with one exception that I’ll describe shortly. With caching, perceived copy time and the publishing scenario both improve. However, several significant changes in both the file copy algorithm and the platform were required to address the shortcomings of cached I/O I’ve already noted.

 

The one case where the SP1 file copy engine doesn't use caching is for remote file copies, where it prevents the double-caching problem by leveraging support in the Windows client-side remote file system driver, Rdbss.sys. It does so by issuing a command to the driver that tells it not to cache a remote file on the local system as it is being read or written. You can see the command being issued by Explorer in the following Process Monitor capture:

 

 

Another enhancement for remote copies is the pipelined I/Os issued by the SMB2 file system driver, srv2.sys, which is new to Windows Vista and Windows Server 2008. Instead of issuing 60KB I/Os across the network like the original SMB implementation, SMB2 issues pipelined 64KB I/Os so that when it receives a large I/O from an application, it will issue multiple 64KB I/Os concurrently, allowing for the data to stream to or from the remote system with fewer latency stalls.

 

The copy engine also issues four initial I/Os of sizes ranging from 128KB to 1MB, depending on the size of the file being copied, which triggers the Cache Manager read-ahead thread to issue large I/Os. The platform change made in SP1 to the Cache Manager has it perform larger I/O for both read-ahead and write-behind. The larger I/Os are only possible because of work done in the original Vista I/O system to support I/Os larger than 64KB, which was the limit in previous versions of Windows. Larger I/Os also improve performance on local copies because there are fewer disk accesses and disk seeks, and it enables the Cache Manager write-behind thread to better keep up with the rate at which memory fills with copied file data. That reduces, though not necessarily eliminates, memory pressure that causes active memory contents to be discarded during a copy. Finally, for remote copies the large I/Os let the SMB2 driver use pipelining. The Cache Manager issues read I/Os that are twice the size of the I/O issued by the application, up to a maximum of 2MB on Vista and 16MB on Server 2008, and write I/Os of up to 1MB in size on Vista and up to 32MB on Server 2008.

 

This trace excerpt of a 16MB file copy from one SP1 system to another shows 1MB I/Os issued by Explorer and a 2MB Cache Manager read-ahead, which is distinguished by its non-cached I/O flag:

 

 

Unfortunately, the SP1 changes, while delivering consistently better performance than previous versions of Windows, can be slower than the original Vista release in a couple of specific cases. The first is when copying to or from a Server 2003 system over a slow network. The original Vista copy engine would deliver a high-speed copy, but, because of the out-of-order I/O problem I mentioned earlier, trigger pathologic behavior in the Server 2003 Cache Manager that could cause all of the server’s memory to be filled with copied file data. The SP1 copy engine changes avoid that, but because the engine issues 32KB I/Os instead of 60KB I/Os, the throughput it achieves on high-latency connections can approach half of what the original Vista release achieved.

 

The other case where SP1 might not perform as well as original Vista is for large file copies on the same volume. Since SP1 issues smaller I/Os, primarily to allow the rest of the system to have better access to the disk and hence better responsiveness during a copy, the number of disk head seeks between reads from the source and writes to the destination files can be higher, especially on disks that don’t avoid seeks with efficient internal queuing algorithms.

 

One final SP1 change worth mentioning is that Explorer makes copy duration estimates much sooner than the original Vista release and the estimation algorithm is more accurate.

 

Summary

File copying is not as easy as it might first appear, but the product team took feedback they got from Vista customers very seriously and spent hundreds of hours evaluating different approaches and tuning the final implementation to restore most copy scenarios to at least the performance of previous versions of Windows and drastically improve some key scenarios. The changes apply both to Explorer copies as well as to ones initiated by applications using the CopyFileEx API and you’ll see the biggest improvements over older versions of Windows when copying files on high-latency, high-bandwidth networks where the large I/Os, SMB2’s I/O pipelining, and Vista’s TCP/IP stack receive-window auto-tuning can literally deliver what would be a ten minute copy on Windows XP or Server 2003 in one minute. Pretty cool.

Published Monday, February 04, 2008 6:47 PM by markrussinovich

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

# re: Inside Vista SP1 File Copy Improvements

Thanks Mark for an excellent excellent article. I only wish you'd write more often!

Monday, February 04, 2008 4:04 PM by John Röthlisberger

# re: Inside Vista SP1 File Copy Improvements

Next step: Ability to copy read-exclusive locked files in Explorer. If the user tries to copy a locked file, ask if they'd like to make a temporary snapshot and copy from that, if they have the necessary permissions to create one. Like HoboCopy does, but integrated into Explorer.

It beats failing halfway, giving the user the false impression that Windows is simply incapable of doing what every other OS can. Unreliable file copying has always been one of my biggest headaches in Windows. Something so basic shouldn't fail so easily and hopelessly.

Monday, February 04, 2008 4:16 PM by David

# re: Inside Vista SP1 File Copy Improvements

When you speak of "the Windows file copy engine" is this in the shell only?

Do applications that use CopyFileEx benefit from any of these changes?  Has this API changed since XP?

Monday, February 04, 2008 4:17 PM by tglascock

# re: Inside Vista SP1 File Copy Improvements

It would be interesting to see if the long standing File move on the same partition when the destination already exists has been fixed.

Normally, when you move a file on a partition, the OS just changes the directory links - it is quick and efficent, certainly much MUCH faster than a file copy (and file size has no impact on how long it takes).

However, if you copy file.txt from directory a to directory B, and directory B already contains file.txt something very different happens.  Yes, Vista looks at B's copy of file.txt, and yes, it can be slow if its video, but that isn't what I mean.  If you confirm that you want to overwrite the destiation, you get a slow block by block copy INSTEAD of the system just deleting B's copy and moving A's copy in the normal fashion.  Attempts to move multi-mb files can go from lightning fast to dog slow (worse than from hd to hd, as one hd is getting nailed with TWICE the traffic as it reads and writes each block - caching probably just makes it worse)

Any hope they fixed that?

Monday, February 04, 2008 4:22 PM by Xepol

# re: Inside Vista SP1 File Copy Improvements

The improvements apply to the Shell and the CopyFileEx engine. I've updated the conclusion to indicate that.

Monday, February 04, 2008 4:23 PM by markrussinovich

# re: Inside Vista SP1 File Copy Improvements

Biggest performance improvement you can make for same volume copies is hybrid differential copy. At first the copy is link to the original with different acl, then the modifications are stored (differential copy) and when there's enough modifications in the copy to affect performance then a true copy is created.

Another huge Vista issues have been

1. the time it takes to open explorer (clean install). The more drives you have in the computer the longer it takes. In 2003/XP Explorer SNAPS open (<100 ms) in Vista it's clearly slower, 200-300 ms atleast with some users reporting 1000 ms or more.

2. the inability to remember the explorer settings such as if you want to have every folder to have a detailed view akin to XP/2003 (columns: name,size,type,date modified).

You can set this from the customize folder however it doesn't apply it to all folder types globally AND it even forgets the setting quite soon.

3. Moving files in same volume is also perceivably slower (maybe 0.3 seconds or so but easy to notice)

Monday, February 04, 2008 4:26 PM by zzz

# re: Inside Vista SP1 File Copy Improvements

As always, a very interesting article. It was sounding like all the file copy problems of Vista had been resolved until the end of the article where you gave two examples where Vista SP1 performance would be less than what the original Vista was: file copies from Windows 2003 servers over a slow network, and large file copies on a Vista client. Unfortunately these are two very common scenarios, so it would seem that despite all the changes, users are still going to experience slow copies on Vista SP1 and probably even slower than on Vista RTM. Is this something that will be resolved, or is it just accepted that users will have to put up with it?

Monday, February 04, 2008 4:45 PM by stuart

# re: Inside Vista SP1 File Copy Improvements

<i>After much analysis, benchmarking and tuning. . . </i>

One would hope, that after what, 30+ years of OS-writing experience? . .  Microsoft would have this kind of thing down cold. . . :) - a little rigorous test process, and documentation, and they wouldn't have to keep re-inventing their own wheel every 6 years.

Anyway - another great article.  Once again, Process Monitor Pwns Vista.

Monday, February 04, 2008 5:47 PM by Neil Prestemon

# re: Inside Vista SP1 File Copy Improvements

How does the cut/paste algorithm difer from copying?

Monday, February 04, 2008 5:52 PM by Devin

# re: Inside Vista SP1 File Copy Improvements

Next add the ability to pause and resume copies :). Then merge that into a crash protection scenario...if explorer crashes while moving files...just resume no data loss :).

Monday, February 04, 2008 5:59 PM by Mark Farina, Jr.

# re: Inside Vista SP1 File Copy Improvements

"In 2003/XP Explorer SNAPS open (<100 ms)"

In my experience that is true in 2003 but not in XP.  Explorer in XP goes zzz for a while before displaying its window in the first place, zzz again before displaying the list of drives in My Computer, zzz again before expanding the C drive (if I have a C drive), etc.

"the inability to remember the explorer settings such as if you want to have every folder to have a detailed view akin to XP/2003 (columns: name,size,type,date modified)."

For me that's a pain too.  But Microsoft already answered that this behaviour is BY DESIGN.  As Microsoft pointed out, wording that used to say "apply to all folders" now says "apply to folder" and it is not supposed to apply the same view to other folders.  So the new bug is that Vista Explorer sometimes does part of what you and I want, for a short time, when it shouldn't be doing that at all.

XP's bug is what you and I thought it is, i.e. that it's supposed to apply the same view to all folders, but about twice a day it forgets its settings.

Monday, February 04, 2008 8:08 PM by Norman Diamond

# re: Inside Vista SP1 File Copy Improvements

"That’s because the previous algorithm’s use of cached file I/O lets Explorer finish writing destination files to memory and dismiss the copy dialog long before the Cache Manager’s write-behind thread has actually committed the data to disk"

In that case I'm sometimes enormously glad that XP or 2003 sometimes refuses permission to disconnect a USB hard drive.  When we think writing has finished, writing really hasn't finished.  But why doesn't this finish after, say, another 3 minutes?  Sometimes XP or 2003 never relinquishes, it just always keeps refusing permission to disconnect, so it's necessary to shut down.  Can the cache manager be told to finish writing its cache?

Also by the way, what happens in Vista when a user tells Vista to shut down?  Vista no longer gives applications a chance to delay the shutdown.  The user thinks writing has finished but the cache manager still needs 3 more minutes.  What happens when Vista forces the shutdown to complete within 20 seconds?

Monday, February 04, 2008 8:12 PM by Norman Diamond

# re: Inside Vista SP1 File Copy Improvements

To zzz: The problem with explorer view settings changing on their own has been fixed in SP1.

Monday, February 04, 2008 10:31 PM by Pavel Lebedinsky

# re: Inside Vista SP1 File Copy Improvements

Thank your analysis very much!!vista sp1 is very good!!!

Tuesday, February 05, 2008 12:17 AM by dvy

# re: Inside Vista SP1 File Copy Improvements

Next add queueing per local drives/destination in explorer. Number of times my girlfriend is copying music off a USB drive and has 20 odd file copy windows open...

Tuesday, February 05, 2008 12:39 AM by phx

# re: Inside Vista SP1 File Copy Improvements

Mark, what about the copy (command prompt), xcopy and robocopy in Vista? Which one of them use CopyFileEx API? Which command still copies using pre-Vista methods? Please answer this.

Also, for Windows 7 maybe you can add pausing and resuming and even better would be allowing priorities, like copying in the background using low priority I/O or a mission-critical copy etc.

As for the shell/Explorer some more things I don't like are (This is if the shell team is reading this)

1. In views like "List view", clicking on a white space between two horizontal columns, selects the item. Unselecting an item after doing whatever you have done and before doing another thing on it is very painful.

2. The Load/Save dialogs still show URLs!! Wow! Can I save my notepad text file to www.microsoft.com? What is this? Windows Live Workspace?

3. Autosort/Autorefresh behavior for Explorer needs getting used to, however if I paste 100+ files in a folder containing 100+ files, those files are scattered all over alphabetically! Give us an option to modify this behavior.

4. The size and free space is not shown on the status bar without selecting the files. Clearly this is a forgotten issue? Why not give us a TweakUI?

5. When sorting by any criteria, Vista first sorts in Descending order compared to XP which first sorted in Ascending order. Many times, (My)Computer and Recycle Bin turn up at the end of the list after sorting because of this!!

Tuesday, February 05, 2008 5:24 AM by someone

# re: Inside Vista SP1 File Copy Improvements

1. Good article as ever. The frustrating thing about this is that many Vista beta testers including myself pointed out the sluggishness of file copying, through mutiple bug reports with lots of votes.

2. Also, what about zip file extraction, which is also painfully slow on Vista RTM, and still slower than XP on Vista SP1. It is no surprise that the channel 9 videos often show MS Engineers with Winzip, though maybe that's for other features.

Tuesday, February 05, 2008 8:04 AM by Mark Anon

# re: Inside Vista SP1 File Copy Improvements

The Vista file copy has got to be the most frustrating and annoying part of Vista, especially over a network. How many times have I stared at the thing saying "0%... Calculating Time Remaining" and it just sits there. Horrible, horrible, horrible, how could they screw this up this badly?

Tuesday, February 05, 2008 8:48 AM by El Guapo

# re: Inside Vista SP1 File Copy Improvements

Thank you for being honest about the memory pressure issue.  This has always been a source of great frustration for me in Windows.  Why can't the cache manager cap the amount of memory it uses to a reasonable (and preferably tweakable) value?  I think it's really disgraceful for a file copy operation to "force other useful code and data out" of memory, especially to the extent that it does.  Valuable memory contents are discarded just to cache hundreds of MB of copied file data that is almost never accessed before it leaves the cache.  Why...

Tuesday, February 05, 2008 10:19 AM by Tom

# re: Inside Vista SP1 File Copy Improvements

Excellent article, as always!  This is vaguely related to file copying and performance, but one of the features I like most about Vista is Previous Versions.  I was quite surprised--and thrilled--to find out that this was accessible remotely via the admin share.  What a great admin tool--it has already saved the day several times!  

If I could make a request, it would be for a continual file change monitoring system (rather than preset intervals), at least for user files.  Maybe this would be better implemented within the application itself, but it would be awesome if Windows took this to the next level and had some kind of continual, point-in-time restore feature for user files.

Tuesday, February 05, 2008 11:52 AM by Ryan

# re: Inside Vista SP1 File Copy Improvements

slow file access/copy speeds with SP1 are only slightly better and still exist! 4-16 MB/s MAX. with any hd (disk to disk, disk to all external usb 2.0 disks every tried). all patches and latest drivers installed. f.i. with all sony vaio sz 1-7 notebooks so far. read all the issues: http://forums.microsoft.com/technet/showpost.aspx?postid=1358057&siteid=17&sb=0&d=1&at=7&ft=11&tf=0&pageid=36

Tuesday, February 05, 2008 12:06 PM by Ger Ger

# re: Inside Vista SP1 File Copy Improvements

@Mark Anon: I think that's got a lot to do with the 'Attachment Security' behaviour - if the ZIP was downloaded from the web and has the attachment security enabled ('This file came from another computer' in Properties), the extract engine applies attachment security to all the contents. This typically breaks CHM files.

WinZip 11 understands attachment security and if the ZIP is marked, will do the same, which is very slow. Clicking Unblock in Properties causes it not to apply attachment security, and extraction goes back to the expected speed.

WinZip 11 is still faster than the built-in ZIP feature, however.

Tuesday, February 05, 2008 12:34 PM by Mike Dimmick

# re: Inside Vista SP1 File Copy Improvements

"In Vista, Explorer also waits 12 seconds before making an estimate of the copy’s duration"

Question on this. Until SP1 for Vista comes out, is there a Registry modification we can make on Vista that would decrease this value? Thanks!

Tuesday, February 05, 2008 3:06 PM by Joseph Moore

# Send me an e-mail if they ever get it right

Vista sucks - plain and simple.   While file performance is one of the worst parts of Vista, network throughput is pathetic and gaming performance sucks quite a bit as well.   Sure you can buy faster hardware but then why would you throttle it with Vista - why not use a faster OS such as XP or Linux?

Yes, I truly hate Vista but I don't hate MS and I haven't given up on MS either.  I certainly don't think Vista can be fixed any more than a house full of mold should be fixed - far better to tear it down and start again.  I really hope MS learns SOMETHING from the spectacular failure that is Vista.

I bombarded them from numerous clients about how bad Vista performed and MS kept insisting - it must be your drivers, your hardware, your antivirus, your antispam, indexing should be turned off, aero should be turned off-  pretty much -make it even simpler than XP and you still won't get the performance of XP - I don't know if MS needs to hire just MIT people but I think they should stop hiring community college drop outs if they can't figure out with a ton of feedback that Vista is SLOW.

Tuesday, February 05, 2008 4:08 PM by boe

# re: Inside Vista SP1 File Copy Improvements

Excellent article Mark.

I was quite interested in the copy algortitm improvements on SP1, and now I have the clue.

That's very interesting, because some bad minds, thing the slowness during copy is a bug on Vista.

Tuesday, February 05, 2008 4:26 PM by Guti

# re: Inside Vista SP1 File Copy Improvements

One of the biggest improvements I noticed with Vista's copy operations (at least in Explorer) is the fact that if it encounters an error during the process, it doesn't cancel the entire operation as it would in previous versions of Windows. You may have to skip the file, but I'm glad it doesn't kill the entire transfer. I'll take the slower performance if I get a more stable copy operation.

It's the little improvements that nobody ever mentions that make Vista better than XP. Still, people continue to treat Vista like the plague. I'm not a big fan of Microsoft, but I have to admit that *gasp* I actually LIKE Vista. That's a first for a Microsoft OS in my case.

Tuesday, February 05, 2008 11:20 PM by Stephen

# re: Inside Vista SP1 File Copy Improvements

I'm running a Dutch Vista Ultimate version. I wonder if i could install SP1 when i change the language to English. As usual Mark's article is good.  Somehow i managed to fix 80% of the copy/move problems back in august with some tweaking and pre-sp1 updates.

Wednesday, February 06, 2008 5:31 AM by Johan

# re: Inside Vista SP1 File Copy Improvements

While Vista SP1 may copy faster, it seems like one cannot do anything while the system is copying. IE takes ages to load, things just seem slow. I hope that is addressed and fixed later, along with the Windows rot over time.

Wednesday, February 06, 2008 10:18 AM by Brandon Clinger

# re: Inside Vista SP1 File Copy Improvements

vista's Windows Explorer user experience is truly the most notable improvement over other windows versions as far as an everyday work flow is concerned. so many detailed and small improvements that i cannot live without it anymore. readjusting the folder view positioning automatically, etc. brilliant.

performance, though, needs to be improved.

Wednesday, February 06, 2008 10:30 AM by BBuGG

# re: Inside Vista SP1 File Copy Improvements

It's possible that a XX-billion dollar company can somehow miss and not realize the simplest of truths that a single individual home or business user knows with his/her pocketbook.

This is called missing the mark.

I'll make this very clear.

My statement is the single most correct, absolutely pure, direct, straightforward possible wording that can be realized.

Due to the strangest of chaotic influences, it's possible that the next release of Windows can actually manage to *at the very least* not meet these simple executive orders from a lowly end-user.

Here's the rules:

Do *NOT* ship us a new OS unless:

1) File copies and general IO ops are faster than or equal to the previous release

2) Application loading times are faster than or equal to the previous release

3) Frames per second in games/3d is faster than or equal to the previous release

4) CPU usage on 2d drawing is lower than or equal to previous release

5) UI responsiveness is faster than or equal to the previous release

6) Boot time is lower than or equal to previous release

7) General performance (ie, heap manager) is faster than or equal to previous release

Before shipping windows, go through a list of these.

If each point is not met,

Do *NOT* ship Windows.

I repeat.

If fps in games is lower, do *NOT* ship Windows. There's nothing else to analyze, consider, or figure out. We do not want the product if FPS has lowered.

How can it be any more clear?

It doesn't get any simpler.

Wednesday, February 06, 2008 4:09 PM by Vabokner

# Really necessary with copy engine?

I wonder, is it really necessary with a copy engine in an OS? I mean, Linux doesnt have one and fares well in benchmarks? Isnt a copy engine overkill? Whats the point?

Wednesday, February 06, 2008 4:13 PM by Kebabbert

# re: Inside Vista SP1 File Copy Improvements

Vabokner: maybe *you* should write the next version of Windows, then.  I think you'd quickly see why that is far, far from the level of simplicity you suggest.  

Wednesday, February 06, 2008 4:42 PM by bluvg

# re: Inside Vista SP1 File Copy Improvements

Re: Vabokner

I disagree that EVERYTHING has to be better in every aspect.  Sometimes, a cleaner, more extensible implementation will not be included due to a slight performance penalty which is incurred on older, but not noticed on newer, hardware.  Which is better, and why is one implemented over the other?  What are the prospects of bringing the more abstracted implementation up to performance par with the original implementation?  These are all questions which must be answered well, and I don't think you're really the person to answer them.  If you can intelligently express why the new version of Windows does not meet your needs, and express it to Microsoft, and your need seems weighty enough or common enough to make a difference in revenue to MS, then I think you'll find them QUITE responsive to your needs.  Don't be so presumptuous to think that Microsoft exists to meet your needs.  If your needs can be reasonably met, they'll do everything they can to do so.

Wednesday, February 06, 2008 4:43 PM by Charles

# re: Inside Vista SP1 File Copy Improvements

An interesting insight Mark. Once SP1 hits Technet I'll be rolling it out to test to see if we can deploy en mass finally...

Wednesday, February 06, 2008 7:03 PM by WigF1

# re: Inside Vista SP1 File Copy Improvements

Sorry to those who responded, if my points were unclear.

Obviously the points I mentioned aren't all that comprises windows. I didn't mean to say that.

I didn't mean to say that it's complete.

But they're a small slice of *bare minimum* of requirements.

The points are RELEASE BLOCKERS.

Microsoft should not even consider it vaguely possible, professional, or profitable to ship a release which misses these and many more points.

I didn't want to spam the blog with 100 points, so I gave a few examples that are transgressions against an OS release.

Vista broke various of the simple rules.

MS should fill the list out to the proper 100-point extent, and follow it tooth and nail.

"_we_ _cannot_ _ship_ _if_ _fps_ _is_ _lower_"

Wednesday, February 06, 2008 7:04 PM by Vabokner

# re: Inside Vista SP1 File Copy Improvements

"If each point is not met, Do *NOT* ship Windows."

That's excessive.  It's like saying that no planes should be allowed to fly until Biman/Garuda/USAir/whoever starts operating properly.

Let Vista ship.  Let XP ship.  Let Linux ship.  Let those who want to buy a computer with XP buy a computer with XP without paying a Vista monopoly tax.  Let those who want to buy a computer with Linux buy a computer with Linux without paying a Vista monopoly tax.

Imagine if Microsoft had to pay for a Canadian postage stamp every time Microsoft sends e-mail, and suppose the rate of loss of Microsoft's e-mail would be the same as letters that pass through Canada's post office.  Microsoft would be clamouring to end the monopoly.

Wednesday, February 06, 2008 7:05 PM by Norman Diamond

# re: Inside Vista SP1 File Copy Improvements

"While Vista SP1 may copy faster, it seems like one cannot do anything while the system is copying."

That is true.  When the OS finally gets fast enough to keep up with disk speeds, the new bottleneck is disk speeds.  Perhaps you're saying you want to specify which disk operations should be low priority (this time request that these Windows Explorer file operations should run slowly) and which disk operations should be high priority (this time request that these Internet Explorer file operations should run quickly ... together with these Windows Explorer file operations that load iexplore.exe into RAM ...).

Wednesday, February 06, 2008 7:08 PM by Norman Diamond

# re: Inside Vista SP1 File Copy Improvements

Re: Vabokner

Agree completely, its that simple.

@Charles:

"I disagree that EVERYTHING has to be better in every aspect."

It doesn't have to be better,  just not that much worse. 1% worse might bee ok, but a 10-30% drop in framerate for games scares a huge market.

Wednesday, February 06, 2008 7:48 PM by Erik

# Vista SP1 File Copy over gigabit LAN while listening to music

Mark, thanks so much for another timely and most enlightening article.

Have SP1 changes done anything to stop MMCSS from decimating throughput so severely on gigabit LANs?

Thursday, February 07, 2008 2:35 AM by Trevor Morris

# re: Inside Vista SP1 File Copy Improvements

Game speeds seem a little off topic but st they've been brought up ... isn't this very similar to what was experience when XP was released? Gamers stayed with 2000 et al for quite a while as game developers learnt to optimise for the new OS. It amazes me how quickly people forget the issues that always exist with the release of a new OS. We would all still be using DOS if initial speed was the sole criteria for releasing updates.

Thursday, February 07, 2008 5:20 AM by manta

# re: Inside Vista SP1 File Copy Improvements

Well game speeds are really off topic as they depend largely on graphics driver performance. It will take some more time until Nvidia and ATI/AMD have optimized their Vista drivers up to XP level.  

Thursday, February 07, 2008 7:45 AM by indeed356

# re: Inside Vista SP1 File Copy Improvements

Do you know if the "Out of memory" error when copying large numbers of files has been fixed too?

http://blogs.zdnet.com/hardware/?p=829

Thursday, February 07, 2008 12:41 PM by Martin

# re: Inside Vista SP1 File Copy Improvements

"Vista Improvements to File Copy"

The thing is that from many user's perspective, that statement is a lie; their experiences showed them that file move and copy operations were seriously degraded in Vista.

Maybe if you expressed it some other way to highlight the improved theory without implying an improvement to the end-user's experience.

To the end user, what you call "the perceived performance" is the performance.

SP1 "...can be slower than the original Vista release in a couple of specific cases." Who cares, people are comparing the speeds to non Vista OS not to pre-SP1 Vista.

Thursday, February 07, 2008 1:46 PM by Anon

# re: Inside Vista SP1 File Copy Improvements

Wow - quite a few people seem to fail at reading comprehension.  It's sad when he addresses the specific point a majority of people are deriding Vista about in his original blog.

At any rate, fantastic insight :)  It's good to know why Vista seemed a lot slower in the File Copying aspect.

Thursday, February 07, 2008 4:11 PM by Bryan

# re: Inside Vista SP1 File Copy Improvements

First things first.. THERE SHOULDN'T BE ANY PROBLEMS WITH FILE COPYING. This article is just an excuse for the problem. No commercial products should have this many flaws, especially an Operating System. I would expect flaws like these on an independent freeware OS such as ReactOS. I still cannot understand how can this product be marketed.. or even sold!

Thursday, February 07, 2008 4:26 PM by n0rt

# game speeds

I agree with indeed356. The major bottleneck is nVidia/ATI drivers. Some games do run faster under Vista, others don't. There are some known issues with Vista itself, but the most relevant bottlenecks seems to be drivers. nVidia dragged their feet on Vista drivers, and delaying Vista's release by a year would have allowed nVidia to postpone their Vista driver similarly. Better to release early so that the hardware OEMs are forced into coughing up some drivers. (I am still waiting for a decent 32-bit XP/2003 nVidia driver -- one that properly supports PAE again)

Thursday, February 07, 2008 4:27 PM by Rune

# re: Inside Vista SP1 File Copy Improvements

Nice writeup.  Now only if there were a way to 1) tune the behavior and 2) provide an override so a specific copy operation used specific copying behavior.

Thursday, February 07, 2008 4:30 PM by davidwr

# re: Inside Vista SP1 File Copy Improvements

Have any of theese improvements resulted in any regressions in other parts of the system? The copy engine sounds like some DRM stuff to protect streams. Why is it neccesary?

Thursday, February 07, 2008 4:42 PM by slorb

# VISTA IS A FAILURE GUYS, face it (be honest with yourselves)

So much for the "commit all memory to caching" (or SuperFetch/PreFetch stuff also): Guys @ Microsoft - "it's NOT working" fellas. People are NOT going with VISTA for reasons like this one, problems galore.

The only sales MS is generating is that from NEW PC's with VISTA pre-installed, & unless the person has NEVER used a PC before (a rarity today), they will also run into things like the rather foolish changes to say, control panel & UAC, which NO ONE REALLY LIKES period!

(Why on EARTH Microsoft ever changed from "CLASSIC VIEW" which everyone & their brother know by now, from Windows 3.x onwards basically)

VISTA is a failure guys.

Thursday, February 07, 2008 4:59 PM by Martyn Byll

# XP dismisses the copy dialog before completion ?!?

<quote>That’s because the previous algorithm’s use of cached file I/O lets Explorer finish writing destination files to memory and dismiss the copy dialog long before the Cache Manager’s write-behind thread has actually committed the data to disk</quote>

Are you saying XP dismisses the copy dialog before the file is copied to the disk? If that is true, it seems to be a HUGE violation of dialog intention.  A dialog shouldn't dismiss until the subject action is totally complete.

Something else is going on here anyway.  At MOST, XP for me would continue writing to the disk for 1 second after the copy file dialog dismissed.  Vista is tens of seconds or more slower than XP at doing file copies.  The perceived speed due to I/O caching is NOT the biggest difference in faster file copies for XP vs. Vista.

Thursday, February 07, 2008 5:10 PM by colinnwn

# re: Inside Vista SP1 File Copy Improvements

Linux and OSX releases get...

dun, dun, dun,

queue music...

FASTER

Each release. How unheard of?

Microsoft has lost its genius and they're just a bunch of newbies relative to the Cutler days of NT. They have 50 thousand employees or whatever, and they can't even manage to make an OS release faster than the previous one. This boggles the mind. Severe mismanagement.

The most ABSURD thing about the whole waterworld-OS situation-Windows-Vista, is that Microsoft employees ACTUALLY think we don't care about performance. I swear.

I am not kidding. That's really what they think. Straight from the horses mouth. They think that we won't notice, that general consumers are clueless. But what MS doesn't realize is that the informed users who are strongly looking at performance, are the ones originating the seed of opinion into the rest of the market.

Simple point: If the market didn't care about performance, then why does Intel have a strong business?

Microsoft vastly underestimates the complete domination of perception of their product by hard-numbers/sheer performance. They're being fed kool-aid from a bunch of marketing and analysis idiots. But it's not reality...

MS forgot speed.

Vista is too slow to be taken seriously.

I'm skipping Vista for my company of 60.

We'll see if a shining star can rise out of Microsoft to speed up their future with Windows 7.

Thursday, February 07, 2008 6:00 PM by Terry

# re: Inside Vista SP1 File Copy Improvements

It's amazing how most people are totally missing the point of the original post - local file copy in Vista RTM was never really broken! Developers of Vista made a brave choice of displaying actual copy speed to the users by disabling caching.

In XP and (I presume) in Vista SP1 the process worked like that:

1. file was read into memory cache

2. some time later (ideally when no other applications are writing to disk) memory cache was written to disk

Only step 1 was shown to the user as file copy. This made file copy to appear somewhat faster (as it actually was spread in time), but had two serious drawbacks:

a. if PC would suddenly lose power before completion of step 2, the file would not be completely copied and there will be data loss (and since step 2 wasn't shown to the user and could be postponed, that was a frequent source of problem)

b. if you were copying a huge file (larger then the amount of free RAM your PC had), the entire system would start swapping and slow down significantly (because of memory cache taking as much RAM as possible)

Vista solved both problems by actually displaying entire file copy to the user and not using memory cache. I guess people at Redmond hoped that modern hard drives are fast enough...

I'm not sure what approach is best for most people, but I liked Vista one much more. Too bad that it will be disabled in SP1 :-(

-------------------------------------

As for the rest of the comments, people are typically deluding themselves... h-m-m, where do I start...

- copy engine is part of every modern OS, including Linux or MacOS X (and "engine" is just a nice word for a set of functions ;-)

- and no DRM in sight

- neither MacOS X nor Linux are getting faster with new releases

- and FPS in Vista are mostly fault of driver writers, and not Microsoft's issue really

- etc

Thursday, February 07, 2008 7:02 PM by Alex

# re: Inside Vista SP1 File Copy Improvements

Oh come on... watching with Filemon you can see that Vista touches every file before copying a directory for creating an estimate of how long the copy-process will take and take a count of the files it has to copy.

On larger codebases (50.000+ small files) this created a situation here. We've actually measured the time that it takes to boot Windows XP to make a backup copy before booting back into Vista to continue development and that's actually FASTER than waiting for Vista to get the job done.

Matrices?! Caching?! A TEAM of people wrote this code?

But whatever, the thing that bugs ME most about Vista is that Microsoft wrote a huge lump of code degrading performance to implement encryption technology that actually does nothing but STOP the user from doing things with his very own computer at the request of Hollywood... but that's not the topic here either :-).

Thursday, February 07, 2008 9:10 PM by Joh

# re: Inside Vista SP1 File Copy Improvements

@alex:

MS's "brave decision" resulted in my pc being damn near unusable any time I'd move/copy/unzip any medium sized file for 5 to 15 minutes. Something I do dozens of times a day.

The fact that I'm less likely to lose that data if the power goes off during that operation, something that has only happened to me twice in 15 years of computing, impresses me not at all...

And the trade off of slowing down all these operations to speed up those rare occasions when I move a file 2 gigs or larger? Please...

I put up with it for one week in the hope that MS would put out a hotfix, then made the brave decision to delete Vista and switch to Linux, and will NEVER go back.

Friday, February 08, 2008 2:36 AM by sk

# re: Inside Vista SP1 File Copy Improvements

but my network isn't high latency or "slow", it's a GigE network between two machines that are about 12 inches apart

So if I copy a file to a hard disc not in my pool the transfer rate over my GigE network to WHS has shot up with Vista Ultimate and SP1, this used to be much slow pre-SP1 and I was lucky if I got over 10% utilistation, I now see around 30% utilisation!

Wow great I thought SP1 has sorted that annoying problem, well that until I tried copy a file from the hard disc (not in the pool) back to Vista, I now get around 7% utilisation, pre SP1 this was higher approx around 25/30% of GigE.

I'm just about ready to throw in the towel!

Friday, February 08, 2008 3:09 AM by Just me

# re: Inside Vista SP1 File Copy Improvements

This article lacks the uncompromising edge of all your other material, and instead comes out as "famous IT guy gives support for Vista SP1 using technical lingo" piece.

The reason for this is that everything in the article is purely *theoretical*. Caching *should* improve local copies... Unbuffered copies *should* improve network copies... We don't come to your blog for Microsoft sponsored guesswork! We come for the hard unfiltered facts.

Please take a minute to reply to the technical questions posed in the comments to the article.

Friday, February 08, 2008 3:16 AM by Anonymous Coward

# re: Inside Vista SP1 File Copy Improvements

Alex,

No, Linux and MacOS doesn't have a copy engine. Have a copy program. Well a lot... That can do (I fact, they do) all of that optimization tricks at will...

The thing is, thats a kernel operation?

I don't know windows kernel but if that's inside the kernel seems overkill to me.

Anyway, anyone can do a "SuperCopy" app that do the work better, but is pretty pathetic for a company that so simple operation can be done better.

My real question is why the kernel allows so abusive use of memory simply for file copying. I think that's the real root of the problem. In all systems I know, cache is a "surplus", something you have but you don't have to really count on it.

Friday, February 08, 2008 5:25 AM by mgb

# re: Inside Vista SP1 File Copy Improvements

For those saying that the only difference is in perceived copy speed, I am not sure that is correct.

On multiple occasions, i have run accross the situation where moving a folder with a large number of items (500 - 1000) in produces an estimated copy time of ~3 million hours. This has been mentioned by others.

Since installing SP1 beta, I have not had any ones that were that rediculous, but i have had at least one instance of over 1000 hours.

When you consider that the increased actual copy speed is partly due to the copy time estimation at the begginning, this just adds to the annoyance!

Friday, February 08, 2008 8:00 AM by Mark Anon

# re: Inside Vista SP1 File Copy Improvements

Mark Anon:

How can any zip extraction implementation be *slower* than the one in XP?  That beggars the imagination.  I expect some overhead in GUI tools like Explorer, because they do more than the CLI equivalent (my basis for comparison for zip extraction being info-zip).  For instance, they attempt to actually calculate progress, rather than just saying when each file is completed or whatever like a CLI app would do.  That's something many users want, and so it's worth a little overhead.  There are other things too, so like I said I expect some overhead.  It wouldn't bother me (much) if Explorer took twice or even four times as long to extract as info-zip.  But 50+ times as long (for a zipfile containing a large number of small-to-medium files in multiple nested directories) is just completely unreasonable, I don't care *what* extra tasks it thinks it's accomplishing.  And you say Vista's zip extraction implementation is even *slower*?  What's it doing, stopping after every 60KB of extracted data to recalculate the whole zipfile's SHA256 checksum?

Friday, February 08, 2008 8:18 AM by Jonadab

# re: Inside Vista SP1 File Copy Improvements

I'm confused.  First you say SMB2 does 64KB I/O's instead of 60KB, and then you show a picture of SP1 doing 1MB I/Os "from one SP1 system to another", with the cache manager doubling that in read-ahead.  Then a few paragraphs later, you say SP1 has smaller I/O's--just 32KB.  

Friday, February 08, 2008 8:40 AM by JS

# re: Inside Vista SP1 File Copy Improvements

Regarding Previous Versions, I agree, that's a major selling point of Vista, and something that sets Microsoft ahead of its most significant competitors.  I've never understood why versioning hasn't caught on more on other OSes and consider Microsoft's steps in this direction an instance of real leadership.  With that said, it could be improved, of course.  In fact, Microsoft could do worse than to clone VMS's versioning behavior as precisely as possible for Seven and worry about more innovative improvements subsequently.  VMS has had this since, approximately, the stone age, and it's really nice, but other systems (which in pretty much all other ways are much more up-to-date) for some odd reason have never really adopted it, until Vista.  OTOH, unless I have misunderstood badly, it's not really an Explorer function (at least, I should hope not) but part of the filesystem driver or somesuch, so there may not be a lot that the Explorer team can do about it, other than handling how versions are represented in the GUI.

Stephan, that's great to hear.  Not stopping in the middle of a large copy operation just because one file (usually one you don't actually need; e.g., when copying Documents and Settings for backup purposes on XP you run into locked system housekeeping files that you don't actually need) is uncopyable is a huge improvement.  If that's true, I am really looking forward to seeing Vista replace XP.  (No, I haven't deployed Vista yet.  I'll address that below.)  I do have a question, though.  Does it give you a separate "Can't do this one, skipping" dialog box for each file that fails, make a list and present you with a single dialog box at the end of the whole operation, or asynchronously populate a visible "skipped file" list in the progress dialog?

As far as being able to actually go ahead and copy the file, that would probably mean moving to an inode-oriented filesystem, which would have a *lot* of implications beyond just being able to make copies of open files.  I'm not saying it would ultimately be bad in the long run (indeed, there would be other advantages, e.g., not having to reboot when system files other than the kernel are updated), but it would be a major change for Windows, and major changes always mean bugs and incompatibilities in the short-term, so it's not something that would be done lightly.

We're holding off Vista deployment where I work until at least SP1 is available, because we have had bad experiences in the past with brand-new Microsoft OSes until the service packs start coming out.  (XP for instance was not really a viable replacement for 2K until SP1 came out, and it wasn't a really marked improvement until SP2, in my estimation.)  But that does not mean there aren't things about Vista that I'm really looking forward to.  There are.  (UAC, for all the flack it has received, is one of the biggies.  Other systems have had something like it for a long time, and while Microsoft's first implementation no doubt needed some adjustments, it is a huge step in the right direction.  I am very much looking forward to not having to log in as Administrator periodically just to get LiveUpdate to work correctly.)  And holding off deployment for a few months doesn't mean we think Vista is the plague (though I do have some coworkers who think upgrades are the plague).  It just means we aren't early adopters.  We understand that you never find all the bugs until you deploy to a bunch of real users, and we'd like to let other real users be the guinea pigs, as it were.  SP1 is something I consider to be very important, even though I haven't actually used Vista yet (nor seen it, really, except for a few minutes in a Microsoft travelling vehicle at a tech rally), because it brings Vista closer to the point where we *will* want to start deploying it.  Microsoft needs to understand that service packs are a key factor in broader uptake of new versions (though they are not the only factor; time is also important).

Friday, February 08, 2008 9:01 AM by Jonadab

# re: Inside Vista SP1 File Copy Improvements

Thanks Mark, for the great technical explanation. The number of passionate comments to this blog entry are very telling. I've been using Vista for 14 months now on 4 different machines (both x86 and x64)and while the file copy issues remain quite an annoyance, it's the overall lack of OS quality that really disappoints me. I've seen this in other, recently released Microsoft products too. I'll stand up here and suggest that something may be fundamentally broken at Microsoft. I noticed it personally when I was on campus recently for several weeks, as compared to a previous extended visit there 5 years ago.

Friday, February 08, 2008 11:54 AM by StuartR

# re: Inside Vista SP1 File Copy Improvements

Something does not sound right.  The file-copy problems I experienced were usually copies of a handful of very small files being copied from one directory to another on the same partition taking 10-20 minutes.  Usually the copy file dialog box would not show the files starting to copy for 7-8 minutes -- it would just sit and hang.  Once the files did start copying, it still took a painfully long time.

When I tested the same copies on the command line (using 4NT, which could make a difference), the files would copy in ~1 second.

The problem was *not* perception.  There was a real, serious, problem.  I've had to recommend to numerous people that they wait for SP1 before upgrading to Vista.

I've installed SP1 RC1 and I have not experienced those problems, so that's good.  But passing them off as simply perceived problems concerns me.  Were they actually fixed or not?  It seems so, but what was the cause?

Friday, February 08, 2008 1:48 PM by VistaUser

# re: Inside Vista SP1 File Copy Improvements

I hope the comment regarding 'a slower server 2003 file transfer experience' is changed. Windows Home Server users are already complaining bitterly regarding the abysmal file transfer speeds from Vista. It sounds as though their experience is going to get worse, not better. Not a good way to try and sell what is already a flawed system. File corruption and even slower files isn't a plus in my book.

Colin

Friday, February 08, 2008 2:23 PM by Colin Hodgson

# re: Inside Vista SP1 File Copy Improvements

Copy is slow in some cohfigurations. I think the same thing causing slow router performance in some systems is at fault. I call it 'long wire to router' problems. When the Cat5 cable is short, no packet loss, and no speed problem. When a longer wire (say 70 meters) is used, the delay caused by the natural 2/3rds speed of light in a copper wire cause the router to lose packets and make for very poor communication (say 300 bps, vs 100 mbps). When set to 'half-duplex' the router loses no packets.

I think there may be a posibility that the packet loss in the router is an example of some kind of over-run in the copy process. The destination may have USB, or some other connectors in the path.

Saturday, February 09, 2008 12:16 PM by Mark L. Ferguson

# re: Inside Vista SP1 File Copy Improvements

Who gives a rats ass if they improved the copy performance of Vista by 10% or even 20%? Why does anybody care?

Vista is a SLOW os.

So it's like getting a 15% better fuel-delivery-line installed on your crappy economy family wagon.

It really makes no difference. It won't convince us to drive around in a smoldering heap of junk.  

Saturday, February 09, 2008 3:47 PM by Sarb

# Pimping engine of WV Beetle...

Thanks Mark!

This article pretty much explaing why my WinXP system is soooo slow in copying large files (700MB+) between volumes of the same disk and who is behind all this clicking noise from HDD seeks that immidiately fills the room :)

Furthermore, it explained, why even it this situation, other task can access HDD relatively easy.

But I want to make accent on some other point:

From the end-user standpoint, despite all the effort being made to improve the situation, good old Norton Commander and it's derivatives perform 1 000 000 percent better that default XP/Vista copy engine.

Why you say?

Because end-user doesn't really care about all these kernel optimisations. End user cares about comfort, control and predictable results. And what we see for XP/Vista?

1. When calculating time-to-copy Explorer just hangs there displayig nothing to the user. Now very comfy. Couldn't the whole TEAM think of FIRST displaying "calculating the time" and THEN start the routine?

2. After having calculated the total size of all the data to be moved the system doesn't warn you if you have less space of destination drive - it just begins the process. Another frustration. And when the free space end it just tells you "Not enough space" and aborts the operation. See next paragraph.

3. When the copy process is aborted (for example, when copying multiple files and one of them becomes locked) - it just breaks and you have manually compare directories to see what was copied and what was not. And what do you expect now.

4. After the last bit of a huge file goes to the disk you get some small amount of time when explorer is unable to access the file because it's being checked by antivirus/antispyware/etc. What's the reason to check the file that was already checked when in was accessed by Explorer for reading?

Good old NC overcame most of this by just persistent file selection (non-processed files were not deselected) and smartly designed user dialogs in the middle of the process. And more modern file managers like FAR or Total Commander allow you to use cached copy where you can set cache size (for ex. 15% of free memory) which makes for copying dozens of small files and copying single large files much easier and faster as you do sequential reads and writes instead of HDD head-hopping back and forth.

What my point is? Now matter how good a kernel would be - I't of no use to the user without a proper usable interface.

This is why Windows beated Unix, OS/2 and other rivals in UserLand space - it was more pretty (while being pretty basic in comparison to those OSes).

Now we see, that while improving Windows internals, the Externals of the OS are still pretty much rudimentary. And most of UI development is directed now towards comfort&control, but to more important features, like 3D Window Flipping and Sidebar Slideshow from My Pictures.

So, if the whole TEAM works of File Copy - please, pass the message to them that they spend next week improving the user interface (or notify the appropriate team). That would increase sales much better, then another cache tweak that whould anyways be un-noticed by the user because of many other problems.

Saturday, February 09, 2008 3:56 PM by Apc

# re: Inside Vista SP1 File Copy Improvements

$20,000 + invested in so called Microsoft Certified equipment... and I can't copy a file from a local drive to a network drive at more than 100 KILO BYTES per second.

XP copies/loads over  20 MEGA BYES per second.

Upgrading effectively renders the computer unusable for business purposes.

If microsoft cant/wont fix this, then I expect a refund on all Vista software our firm has purchased.

Rumor has it that several suits under the federal deceptive trades act are being considered. At least they offer triple damage awards.

Our lost productivity installing, debugging, trouble shooting, backup, restore, re-install XP or Linux... has cost well over $10,000.

I'd bet hundreds of millions have been wasted by Vista users in similar ways.

Monday, February 11, 2008 12:04 AM by Ex VistaUser

# Caching priority?

There seems to be an awful lot of effort put into avoiding filling up the system's RAM with cached filesystem data. But why is this necessary? Linux systems will happily cache stuff in RAM, but those cache buffers have lower priority than application memory allocations. That way, you never have app code and data forced out of memory just to make room for caches.

With this simple design decision, you no longer need to worry about using too much RAM for cache. Linux has been doing this for years, decades. How hard would it be for Windows to do the same?

Monday, February 11, 2008 1:21 AM by Lawrence D'Oliveiro

# re: Inside Vista SP1 File Copy Improvements

It's curious that 3rd party programs like Total Commander are capable of achieving higher speeds when copying a file than explorer.exe (at least in my case). I have a laptop with Windows Vista Business 32-bit and a laptop with Windows XP Professional. When copying a single 700 MB file from XP (FAT32 partition) to Vista using Total Commander (run under Windows Vista), the Task Manager reports network usage to reach about 80-90% (100 Mbps network). When using explorer.exe, the network usage drops to 60% (plus it occasionally "spikes" from 2% to 60%) and the copy process is slower. Additionally, the hard disk heads on the XP laptop "go ape" (it might be the "out-of-order write operations" mentioned in the post). This does not occur when using Total Commander. Thumbnails on network folders are set to off. Setting autotuninglevel to disable under Vista had no effect as well as removing the RDC in Programs and Features.

I would assume that 3rd party programs use CopyFileEx API mentioned to copy the file. So how come Total Commander appears to be faster over network - even after putting aside the fact that it calculates the remaining time faster?

Monday, February 11, 2008 9:09 AM by Emkay1001

# re: Inside Vista SP1 File Copy Improvements

It's great to read some of the thought process behind the design... to a point.  The bottom line, though, is that Vista copy performance may be the single most complained about "feature".  How about, just for internal MS use, you guys do some benchmarks using different OSes.  Compare copying 5,000 files of mixed content and sizes between Vista, Vista SP1, XP SP2, XP SP3, Mac OS X, Win Server 2003, Win Server 2008, etc.  See if Vista isn't involved in most of the worst performances!

And Mark, while you're at it, could you please le