Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
As a reader of this blog I suspect that you, like me, are the IT support staff for your family and friends. And I bet many of you performed system maintenance duties when you visited your family and friends during the recent holidays. Every time I’m visiting my mom, I typically spend a few minutes running Sysinternals Process Explorer and Autoruns, as well as the Control Panel’s Program Uninstall page, to clean the junk that’s somehow managed to accumulate since my last visit.
This holiday, though, I was faced with more than a regular checkup. My mom recently purchased a new PC, so as a result, I spent a frustrating hour removing the piles of crapware the OEM had loaded onto it (now I would recommend getting a Microsoft Signature PC, which are crapware-free). I say frustrating because of the time it took and because even otherwise simple applications were implemented as monstrosities with complex and lengthy uninstall procedures. Even the OEM’s warranty and help files were full-blown installations. Making matters worse, several of the craplets failed to uninstall successfully, either throwing error messages or leaving behind stray fragments that forced me to hunt them down and execute precision strikes.
As my cleaning was drawing to a close, I noticed that the antimalware the OEM had put on the PC had a 1-year license, after which she’d have to pay to continue service. With excellent free antimalware solutions on the market, there’s no reason for any consumer to pay for antimalware, so I promptly uninstalled it (which of course was a multistep process that took over 20 minutes and yielded several errors). I then headed to the Internet to download what I – not surprisingly given my affiliation - consider the best free antimalware solution, Microsoft Security Essentials (MSE). A couple of minutes later the setup program was downloaded and the installation wizard launched. After clicking through the first few pages it reported it was going to install MSE, but then immediately complained that an “error has prevented the Security Essentials setup wizard from completing successfully.”:
The suggestion to “restart your computer and try again” is intended to deal with failures caused by interference from an unfinished uninstall of existing antimalware (or a hope that whatever unexpected error condition caused the problem is transient). I’d just rebooted, so it didn’t apply. Clicking the “Get help on this issue” link provided some generic troubleshooting steps, like uninstalling other antimalware, ensuring that the Windows Installer service is configured and running (though by default it isn’t running on Windows 7 since it’s demand-start), and if all else fails, contacting customer support.
I suspected that whatever I’d run into was rare enough that customer support wouldn’t be able to help (and what would they say if they knew Mark Russinovich was calling for tech support?), especially when I found no help on the web for error code 0x80070643. My brother in law, who is also a programmer and tech support for his neighborhood was watching over my shoulder to pick up some tips, so the pressure was on to fix the problem. Out came my favorite troubleshooting tool, Sysinternals Process Monitor (remember, “when in doubt, run Process Monitor”).
I reran the MSE setup while capturing a trace with Process Monitor. Then I opened Process Monitor’s process tree view to find what processes were involved in the attempted install and identified Msiexec.exe (Windows Installer) and a few launcher processes. I also saw that Setup.exe launched Wermgr.exe, the Windows Error Reporting Manager, presumably to upload an error report to Microsoft:
I turned my attention back to the trace output and configured a filter that excluded everything but these processes of interest. Then I began the arduous job of working my way through tens of thousands of operations, hoping to find the needle in the haystack that revealed why the setup choked with error 0x80070643.
As I scanned quickly to get an overall view, I noticed some writes to log files:
However, the messages in them revealed nothing more than the cryptic error message shown in the dialog.
After a few minutes I decided I should work my way back from where in the trace operations the error occurred, so returned to the tree, selected Wermgr.exe, and clicked “Go to event”:
This would ideally be just after the setup encountered the fatal condition. Then I paged up in the trace, looking for clues. After several more minutes I noticed a pattern that accounted for almost all the operations up that point: Setup.exe was enumerating all the system’s installed applications. I determined that by observing it queried multiple installer-related registry locations, and I could see the names of the applications it found in the Details column for some of them. Here, for example, is one of the OEM’s programs, another help file-as-an-application, that I hadn’t bothered to uninstall:
I could now move quickly through the trace by scanning for application names. A minute later I stopped short, spotting something I shouldn’t have seen: “Microsoft Security Essentials”:
I knew I hadn’t seen it listed in the installed programs list in the Control Panel in my earlier uninstall-fest, which I confirmed by rechecking.
Why were there traces of MSE when it hadn’t been installed, and in fact wouldn’t install? I don’t know for sure, but after pondering this for a few minutes I came to the conclusion that the software my mother had used to transfer files and settings from her old system had copied parts of the MSE installation she had on the old PC. She likely had used whatever utility the OEM put on PC, but I would recommend using Windows Easy Transfer. But the reason didn’t really matter at this point, just getting MSE to install successfully, and I believed I had found the problem. I deleted the keys, reran the setup, and….hit the same error.
Not ready to give up, I captured another trace. Suspecting that setup was tripping on other fragments of the phantom installation, I searched for “security essentials” in the new trace and found another reference before the setup bailed. To avoid performing this step multiple more times, I went to the registry and performed the same search, deleting about two dozen other keys that had “security essentials” somewhere in them.
I held my breath and ran the installer again, but no go:
The error code was different so I had apparently made some progress, but a web search still didn’t yield any clues. I captured yet another trace and began pouring through the operations. The install made it way past the installed application enumeration, generating tens of thousands of more operations. I scanned back from where Wermgr.exe launched, but was quickly overwhelmed. I just couldn’t spot what had made it unhappy, and that was assuming that whatever it was would be visible in the trace. My brother-in-law was growing skeptical, but I told him I wasn’t done. I was motivated by the challenge as much as the fact that I couldn’t let him tell his work buddies that he’d watched me fail.
I decided I needed the guidance of a successful installation’s trace so that I could find where things went astray. When it’s an option, like it was here, side-by-side trace comparison is a powerful troubleshooting technique. I switched to my laptop, launched a Windows 7 virtual machine, and generated a trace of MSE’s successful installation on a clean system. I then copied the log from my mom’s computer and opened both traces in separate windows, one on the top of the screen and one on the bottom.
Scrolling through the traces in tandem, I was able to synchronize them simply by looking at the shapes that the operation paths make in the window and occasionally ensuring that they were indeed in sync by looking closely at a few operations. Though it was laborious, I progressed through the trace, at times losing sync but then gaining it back. One trace being from a clean system and the other with lots of software installed caused relatively minor differences I could discount.
Finally after about 10 minutes, I found an operation that differed in what seemed to be a significant way: an open of the registry key HKCR\Installer\UpgradeCodes\11BB99F8B7FD53D4398442FBBAEF050F returned SUCCESS in the failing trace:
but NAME NOT FOUND in the working one:
Another bit of the broken installation it seemed, but without any reference to MSE, so one that hadn’t shown up in my registry search. I deleted the key, and with some forced confidence told my brother-in-law that I had solved the problem. Then I crossed my fingers and launched the setup again, praying that it would work and I could get back to the holiday festivities that were in full swing downstairs.
Bingo, the setup chugged along for a few seconds and finished by congratulating me on my successful install:
Another seemingly unsolvable problem conquered with Sysinternals, application of a few useful troubleshooting techniques, and some perseverance. My brother-in-law was suitably impressed and had a good story for when he returned to the office after the break, and my mother had a faster PC with free antimalware service.
I followed up with the MSE team and they are working on improving the error codes and making the setup program more robust against these kinds of issues. They also pointed me at some additional resources in case you happen to run into the same kind of problem. First, there’s a Microsoft support tool, MscSupportTool.exe, that extracts the MSE installation log files, which might give some additional information. There’s also a Microsoft ‘fix-it tool’ that addresses some installation corruption problems.
I hope that your holiday troubleshooting met with similar success and wish that your 2012 is free of computer problems!
Mark; excellent write-up. We just need an anti-crapware removal utility - get on it Microsoft. hahahaha
Seriously, can't Microsoft hold OEM's to a higher standard? Why the need for a Signature series when that should be the bar all OEMs are required to be held accountable to have OEM installations of Windows. Consumers shouldn’t experience this type of frustration no matter what “Windows” machine they purchase. In the long run, it just hurts the image of Microsoft – even if the cause is from craplets.
Yes it is rare. And the process you went through is getting much rarer. But this was typical stuff in the XP days. Win 7 has fixed most of the gremlins that I as a "fixer" have had to deal with. But you can't FIX OEM's. They are absolutley unfazed by the problems that they systematically cause the average joe when he turns on his pc for the first time. They don't care, all they care about is the bottom line, and I have to believe they load pc's up with this stuff because they somehow someway, make an extra buck or more doing it. The problems that you went through were once very typical nightmares that the average joe had to go through. Once more. The OEM's don't give a crap about any of that as long as they can make an extra buck.
MS and the Signature Series and the Stores to come is all fine and dandy, but in the mean time Google and Apple have gained quite a comfortable foothold while MS has been sitting on it's ass seemingly not giving a crap about all of this as well.
I see 2012 as MS greatest chance at winning back some resepect. I see it like this. There are 3 pieces to this puzzle. Xbox, WinPho, and Win8. I can load up Xbox with as much crap is humanly possible and it still rolls along and let's me have fun with not even the chance of anything blowing up in my face. The same for WinPho. From what I've been reading, it's possible that Win8 might just be a lot closer to attaining that status.
That is all any consumer wants.
As far as your "issue" goes. My instinct would have been to go get Revo Uinstaller, and go to town on everything in the control panel.
But why should ANYONE have to do ANY of that in the first place?!!!
For the record: if anyone googles those HRESULTS and ends up here, they're both associated with Windows Installer
0x80070643 - ERROR_INSTALL_FAILURE
0x80070648 - ERROR_UNKNOWN_PROPERTY
The former should be self-explanatory; the latter's most commonly seen when calling MsiGetProductInfo or MsiGetPatchInfo for a product or patch whose ID is in the Windows Installer database but whose MSI or MSP has been deleted (generally by badly-coded cleanup tools or over-enthusiastic admins)
I must say, the mental image of Mark Russinovich sitting on hold with Tier 1 Microsoft support, listening to muzak on the phone made me giggle. :)
Each time I read you, I learn a lot.
I hadn't realised just how bad the crapware (sometimes bloatware) had become until I bought my brother a laptop recently. For the last 10 years I would never have considered booting a new computer without a Windows CD/DVD in the drive.
Good timing, Mark as I had the MSE install fail with the same error code on my end last night. After using the Sysinternals tools to troubleshoot it came down to everyone but me loosing access to the Start Menu->Programs folder so shortcuts could not be created and the install would fail. I added back the permissions to this directory so the installer could access it and this resulted in a successful install! :)
I would hate to the the first level support person who takes the call from Mark Russinovich. Eeek! :D
Mark, how on earth did an HKCR\Installer key go on the new computer? I think it deserves more investigation. It shouldn't take much of time and I think it would worthwhile.
Here's to hoping *cough* "somebody" *cough* could put a little pressure on the MSE guys to give more meaningful error messages... for me it's been the norm rather than an exception that MSE setup throws all kinds of unintelligible errors on OEM machines - even after (attempting to) remove the crapware they're loaded with.
It's a shame that we have to deal with this, because MSE itself is a really excellent piece of software :)
"when in doubt, run Process Monitor”. Another great post, Mark.
@snemarch: You have my support. But I guess in the end you just have to grin and bear.
My brother recently purchased a new computer, luckly I was there to set it up and before we did anything else, I removed all the crapware, installed the new applications and then transfered over the old data. Everything worked great. For my home computer, I build that myself and do a clean install of Windows from DVD, so no problems there. Thanks Mark for all your articles that encourage me to use ProcMon, ProcExp and other Sysinternals tools when I do run into problems.
Oh for the good old days when one could buy a new computer and (re)install everything themselves. Unfortunately these days that can be difficult or impossible. OEM's very rarely provide software CD/DVD's anymore. If you're lucky you'll get a "restore DVD", or hidden partition ,that'll restore the HD to the state (i.e. with all the crapware) it was when it left the factory.
I've encountered the same 0x80070643 error code when attempting to uninstall MSE from a machine that was found to still be infected with the 64-bit version of the ZeroAccess rootkit. Once the rootkit was unhooked from the system and changes that it makes reversed, the MSE uninstall proceeded without issue.