Blogs

The Case of the Failed File Compression

  • Comments 45
  • Likes

The other day Bryce tried to use Explorer’s Send To Compressed (zipped) Folder feature, seen below, to package up his latest Process Monitor source code updates to send me.

Instead of presenting compression progress dialog followed by an opportunity to edit the name of resulting compressed file, Explorer aborted the compression with this error:

Bryce was perplexed. The error didn’t seem to make any sense because he obviously had read permission to the files in the selection, which he’d just finished editing, and compressing files shouldn’t involve some kind of search that could result in a file not being found.  He retried the compression operation, but got the same error, this time after a different number of files had finished compressing.

I happened to walk into his office at this point and he showed me the behavior by trying a few more times, all with the same outcome.  Now both of us were perplexed. It was time to investigate, and the tool we called on for the job was, somewhat ironically, Process Monitor.

We launched Process Monitor, reproduced the failure, stopped the capture, and scanned through the thousands of operations in the trace looking for errors. We saw a slew of NOT FOUND errors near the start of the log, which are the generally innocuous result of an application checking for the pre-existence of a file. In fact, there were literally hundreds of them near the beginning of the log, all of which were queries for the file into which the compressed files would be placed:

That was disturbing, but not directly related to our troubleshooting effort, so I filed it away to look at later.

Several hundred events into the trace later we came across a SHARING VIOLATION error that bore a closer look:

When a process opens a file it can specify if and how it wants to share the file with other processes while it has the file opened. The three types of sharing are read, write and delete, and each is represented with a flag that a process passes to the CreateFile API. In the operation that failed, Explorer didn’t pass any of the flags, indicating that it didn’t want to share the file, as seen in the ShareMode field:

For an open to succeed, the sharing mode of the opener must be compatible with the sharing allowed by a process that already has the file opened, so the explanation for the error was that another process already had the file opened.

Looking back at the trace, the open operation immediately preceding the one with the error is an open of the same file by a process named Inort.exe. Inort’s close of the file isn’t visible in the screenshot because it comes long after Explorer’s failed attempt to open the file. That confirmed that Explorer’s unwillingness to share the file conflicted with Inort having the file open, despite the fact that Inort specified read, write and delete sharing in its open of the file.

Process Monitor had closed another case: Inort holding the file open when Explorer tried to open it was the cause of the sharing violation and almost certainly the reason for the misleading error message. Next we had to identify Inort so that we could come up with a fix or workaround. Process Monitor also answered that question with its image tooltip:

eTrust, Computer Associates’ Antivirus scanner, was apparently opening the file to scan it for viruses, but interfering with the operation of Explorer. Antivirus should be invisible to the system, so the error revealed a bug in eTrust. The workaround was for Bryce to set a directory filter that excludes his source directories from real-time scanning. 

I couldn’t reproduce the error when I went back to my office, so I suspected that I had a different version of Inoculan on my system than Bryce. Process Monitor’s process page on the event properties dialog for an Inort.exe event showed that Bryce had version 7.01.0192.0001 and I had the more recent 7.01.0501.000:

Why we have different versions isn’t clear since we’re both using images deployed and managed by Microsoft IT, but it appears that Computer Associates has fixed the bug in newer releases.

Now I turned my attention back to the inefficiencies of Explorer’s compression feature. I captured a Process Monitor trace of the compression of a single file and counted the associated operations. Just for this simple case, Explorer opened the target ZIP file 14 times, 12 of those before it had actually created the file and therefore with NOT FOUND results, and performed directory look ups of the target 19 times. It was also redundant with the source file, opening it 28 times and querying the file’s basic properties 17 times. It’s not like Explorer doesn’t give eTrust plenty of opportunities to cause sharing problems.

In order to verify that Explorer itself was at fault, and not some third-party extension, I looked at the stacks of various events by selecting the event and typing Ctrl+K to open the event properties dialog to the stack page:

Zipfldr.dll, the Explorer file compression DLL, was in most of the stack traces, meaning that the compression engine itself was ultimately responsible for the waste. Further, the number of repetitious operations explodes when you compress multiple files. There are clearly easy ways to improve the algorithm, so hopefully we’ll see a more efficient compression engine in Windows 7.

Update: I've learned that the compression engine has been updated in Vista SP1 to perform fewer file operations.

On a closing note, if you’d like to catch me at my next public speaking engagement, come to Wintellect’s Devscovery conference in Redmond, August 14-16, where I’m delivering a keynote on Vista kernel changes.

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • It's not really new that this feature in Windows sucks. I know only XP but you can make the test yourself: extract an archive with a lot of files once in e.g. WinRAR and then with Explorer. With Explorer it's unbelievably slow which seems to be caused by the lot of unnecessary operations you found. Maybe this is by design to still give vendors like WinZIP or WinRAR a reason to sell their product because it's vastly superior to the Windows built in feature...

  • I like the fact that Microsoft is NOT using OneCare for its employees:) . Also I find your articles where you troubleshoot stuff very interesting.

  • I can't say I'm surprised by your finding.

    I gave up on the compression built into XP looong ago since it's so unbearably slow.

  • Brilliant diagnosis!!!

  • Am I correct in assuming this was observed on a Vista OS?  The screen-captures seem to suggest that.

    I wonder if XP's built-in compression is any more or less efficient by comparison to Vista.

    I've always gone ahead and used third-party compression utilities on my XP/Vista systems simply out of habit.  But having one available on the system by default makes working on end-user's workstations a bit easier.

    This is an interesting look at that process.

    Thanks for the always fascinating detective work!

  • Mark, we've seen that problem before as well when we're trying to compress files for our nightly polling process on our XP machines.  They're used as POS terminals and have McAfee Virusscan loaded.  We kept getting sites that failed polling or had corrupt files until we figured out that McAfee was trying to open the files and scan them at the same time our polling process was running.  McAfee has yet to fix the bug in their app, so we just set an exclusion on the folder as well.

  • Not the first time you have found explorer performing the same file operation multiple times.  At least it's only into double figures; I seem to recall in a previous blog you found an instance of explorer performing the same operation 100's of times.

    Just think if Microsoft found and removed all this waste, we would probably being happily running XP on 500 Mhz machines and Vista on 1 Ghz machines.

    Keep up the good work Mark.

  • As always - excellent write-up.

    BTW

    http--blogs.technet.com-photos-markrussinovich-images-1702272-original.aspx

    is broken. No picture there!

  • Uhh, Mark, maybe its time to start using source control. Isn't that one of those standard Microsoft Quality Gates anyway?

  • Wow, that sucks. Note to self: Don't compress directly to remote network shares!

    Now that you work at Microsoft, I presume you get to look at the source code, so you can see the actual offending code. Do you file bugs against Windows when you find such issues?

  • Explorer is woefully slow at manipulating ZIP files, and this could be one of the reasons. At least in Windows Vista, it can handle 64-bit ZIPs and 64-bit files in a ZIP, which it can't in Windows XP.

    I thought I'd reported its slowness at extracting files in the Windows Vista beta, but since the main feedback site wasn't open to me as a public beta user, I couldn't see whether my feedback had even been submitted correctly. If you want a public beta to be useful, you have to make it possible for public beta users to submit feedback and continue the conversation.

    For the moment, WinZip is a much better solution (although its context menu entries are a pest when you've copied a large file in a Remote Desktop session, have the disk sharing option turned on, and right-click an Explorer window in your own desktop, because it downloads the file to work out which icons to show!)

  • I had almost exactly the same issue with WinRAR at most of the workstations at university labs. Whenever i asked it to compress a given setof files, it returned a dialog saying "no files to compress". There was a workaround to that which some students invented, but having read your case study of windows compression, i hope i will be able to address the issue downright :) Im an avid user of process explorer and process monitor at my home mostly using it to check activity of my PC and my software projects, though now you have given me an insight into how to troubleshoot with your magical softwares:) I wish i could attend your presentations on 14th/16th but im just like.... 10,000 miles away :( I hope we cud get a recording of it or notes online (youtube anyone?)

  • XP's compression engine is unbelievably inefficient.  Decompressing the Boost source code archive (a 23 MiB zip file, 85 MiB uncompressed, containing ~12000 files) takes 1.5 minutes on my machine with WinZip.  Using XP's decompressor, it takes a mind-numbing 45 minutes.

    Just how improved is it in Vista?  Is it on par with other compression software now?

  • Thanks Mark, that was very informative.  I love those kind of examples as I'm an avid user of all your tools.  Cheers.

  • Interesting that Microsoft IT do not use their own 'One Care' anti-virus solutions...

    When is Vista SP1 due?

    Mark - Thanks for another interesting blog. Keep up the good work.