The Case of the Mysterious Code Signing Failures

  • Comments 28
  • Likes

I digitally sign code on a regular basis in the course of preparing Sysinternals executables for upload to the site. When you digitally sign a file, you encrypt the hash of the file with the private key of a public/private key pair. Someone can verify that you’ve signed the file by decrypting the encrypted hash with your public key and comparing the result with the hash of the file they calculate themselves.  The signing process is made simple with Signtool.exe, a utility that comes with the Platform SDK and the .NET Framework. You pass it your signing certificate, private key file, and target file as command-line arguments and it does the rest, appending the signed hash in the file as a final step.

The other day I went to sign an updated Sysinternals tool and ran into this error message:

It had been a week or so since the last time I had tried signing anything, but I couldn’t think of any changes I had made to the system that would have lead to this failure.  However, anyone that’s used computers for any length of time knows that they’re not really deterministic and that system configuration is often subject to spontaneous corruption. I resigned myself to never knowing the root cause and set out to resolve the problem.

The first thing I did was search for capicom.dll with the built-in Where utility, which looks for the file you specify in each of the directories listed in the PATH environment variable. The PATH environment variable is used for DLL searches, so I expected this step to confirm that I was missing Capicom.dll:

The output appeared to confirm it, but then I realized that because I was running on a 64-bit system and Signtool is a 32-bit executable, Where.exe wouldn’t look in the %SystemRoot%\Syswow64 directory, which is the directory in which 32-bit system DLLs are stored. When I manually looked in that directory I was surprised to find a copy of Capicom.dll:

Signtool must therefore not be looking for Capicom.dll in the directories listed in the PATH environment variable, so the question before me was, where was Signtool looking? I knew Process Monitor was the perfect tool to answer a question like that, so ran it, configured an Include filter for any Path ending in “capicom.dll” and then repeated the Nmake command that triggered the error:

The trace shows that, for some reason, Signtool only looks for Capicom.dll in two directories:  the Microsoft Shared sub-directory of the system’s Common Files directory, and the \Bin directory, which was where Signtool is located on my system.

To fix the problem I simply copied the Capicom.dll file from the \Windows\Syswow64 to the \Bin directory. I reran the make command and, as I expected, it succeeded. Process Monitor to the rescue!

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • Wow, that's good to know and several people have reported similar problems with Vista and CAPICOM. However, Microsoft's official position is that CAPICOM isn't supported in Vista.

    You are supposed to use the shiny new Certificate Enrollment API, which has finally been documented in the past week:

    I wonder if Microsoft will have a new signtool.exe for Vista, or whether they will use the unsupported CAPICOM interface as well. I hope it's the latter, because that I sure don't want to create a Vista-specific code path when CAPICOM works perfectly well on other versions of Windows.

  • Um, where.exe certainly is not built-in, at least not as far as XP SP2 or Windows 2003 Server. Nor is it part of the downloadable resource kit utilities.

    It is certainly an essential functionality for debugging things, but it's not builtin as far as I know. I use cygwin's which.exe.

  • Je viens de tomber sur cet article chez Mark Russinovich , qui décrit un problème possible avec SignTool,

  • Where.exe is included in Server 2003 SP1 (it might be in Server 2003 SP0, but I don't have a copy to check), Windows XP 64-bit for x64, and Windows Vista.

  • OK, guess I was wrong about 2003. It definitely isn't supplied with XP 32-bit though.

  • Sorry, Mark. I know I tested it in a 32 bit cmd.exe, but IIRC (it's been a few moons since I left) we considered invoking directly from 64-bit-land out of scope. Oops.

    I'm curious. What's the %path% on your box for a 32 bit EXE launched from 64-bit-land? I could have sworn that signtool looked anywhere in the path and tried to install capicom if it could be found.

    I'll check this out myself when I'm in front of my test boxes again tomorrow.

    Drew the ex-'softie, previous test owner of signtool.exe

  • For those running XP, you can find a "where" substitute here:

  • The app must be explicitly looking in those two dirs. LoadLibrary doesn't search other dirs if the caller includes a dir name in the first parameter.

  • FWIW, where.exe is also included with visual studio (at least in VS 2003, it's in the "Common7/Tools/bin" subdirectory).

    Problem is that which.exe (from cygwin) stops at the first hit that matches the searched executable, whereas where.exe lists all occurances.

  • Signtool.exe is NOT searching directly for capicom.dll. Capicom.dll contains a number of COM objects (obviously :) ). The signtool attempts to create a COM object located in capicom.dll and the COM subsystem looks for it (check MSDN for the exact order).

    I had a similar problem with one of my apps that was using capicom objects. Since Process Monitor exists only from 2 months, I used File Monitor to find the cause of this strange problem.

    <b>Mark</b>, try to set a filter only for signtool.exe without path filtering and you will find it! I suspect Visual Studio installer in messing thing up.

  • The path search order is indeed COM at work: it's first looking in the location where the COM registration indicates the DLL resides and if that fails it falls back and looks for it in the current directory. A full Process Monitor trace also shows the Registry accesses where OLE reads the COM DLL path.

  • Toma: Signtool actually registers capicom if it's in the same directory and not yet regsvr32'd. It's nonstandard, but true in this case.

  • Mark, yes it is normal that the full trace shows the registry access where OLE reads the COM DLL path, but ...

    On my machine the problem was that in the registry the full path was written in quotes, which by some reason COM does not understand. Actually quotes are not needed because the WHOLE value is supposed to contain only path, not path and parameters.

    I was not able to find which app registered capicom.dll in this way. I suspected Visual Studio, but can't prove it yet.

  • Where.exe is also available in VS2005 at C:\Program Files\Microsoft Visual Studio 8\Common7\Tools\Bin\Where.Exe

  • You must have uninstalled a program between the last time you signed your exe and the present. That is why capicom.dll was missing.

    I've often found that during an uninstall, I get a window telling me the uninstaller wants to remove a "shared file" but in all likelihood, no other program is using it.

    Usually I'm cautious and do not allow it to remove these shared files but a while ago I took a chance and allowed it. Soon after, every time booted or tried to open an MS Office file (doc, xls, etc.) the Office installer popped up and asked for its installation CD.

    Windows Installer has a built in feature which fixes broken installations. If you run a program installed using Windows Installer and later, when the program requests a resource (dll in our case) that is corrupt or missing, it fetches it from the installer.

    Apparently, when I allowed the uninstaller to remove the "shared file", it actually removed a file Office was using.

    In your case, you also experienced two conflicting installers.