The Case of the Slow Keynote Demo

re: The Case of the Slow Keynote Demo

Rob Girard — Tue, 26 May 2009 21:14:31 GMT

Mark, your posts are always amazing and I look forward to the days where one hits my RSS feed. Thanks for taking the time to share your troubleshooting techniques with us. I've used many of your methods to perform similar "magic" in much less time than my older, shotgun-style trial & error approaches. Your surgical approach to getting to the root of issues is surely making a positive impact at Microsoft. In a time where they need you and your drive for perfection more than ever, keep up the good work and try to "Right the Ship" that is MS. THANK YOU!!!

re: The Case of the Slow Keynote Demo

Michael Dragone — Tue, 26 May 2009 21:43:13 GMT

Excellent post. This is a common problem with E12 servers that aren't connected to the Internet (either on purpose or because of a network outage/problem) when they start up.

I've also had it happen with .NET 2.0 ASP.NET applications that have a signed assembly loaded, in which case you have to edit machine.config.

re: The Case of the Slow Keynote Demo

Shawn Lukaschuk — Tue, 26 May 2009 21:44:49 GMT

Hi Mark,

Interestingly, you could have solved the situation even faster by not using Process Monitor. A google or live search for the symptoms would have brought you to the same knowledgebase article.

The #1 return for keywords “signed windows application starts slowly” on Google and Live Search is the relevant KB Article.

And it’s the #7 result if you drop the word “signed”.

Way to miss the point, Shawn

Yonah — Tue, 26 May 2009 22:43:31 GMT

While it's easy to use Google to search for the answer, Mark was showing us the technical procedure involved in finding the answer ourselves. It's the thought process and deductive reasoning behind the search for the answer that makes this blog worth reading. The reader also has an opportunity to learn something. That's the value of doing your own research and fact finding, rather than always turning to someone else to find the answers.

re: The Case of the Slow Keynote Demo

mattjohnson2005@gmail.com — Wed, 27 May 2009 00:12:49 GMT

Great stuff, thanks for taking the time to post your method. It is GOOD to TEST 'LITTLE' CHANGES to the demo before going live! :)

re: The Case of the Slow Keynote Demo

Daniel — Wed, 27 May 2009 02:36:58 GMT

Funny thing. We ran into this issue when testing our final RTM bits with the "master" CD produced at our manufacturing facility.

It really, really sucked for a variety of reasons.

Ultimately, because there was no other workaround at the time (a few years ago), we had to remove the Authenticode certificates from our binaries.

This invalidated our Vista logo compliance status - which wasn't a huge deal. But it was really annoying trying to convince Microsoft that this was a severe bug in the .NET framework.

Eventually, someone at Microsoft independently got a clue and realized this was important enough to fix. Unfortunately, the workaround (modifying your app.config) is known only to the people who have the pleasure of having to chase this issue down. Since generatePublisherEvidence is not defaulted to false, you can be sure that everyone signing their apps will fall victim to this error.

So Dear Mark, I have to admit my perverse satisfaction in seeing you, a Microsoft employee, trip over this bug. Maybe the pain will induce your colleagues to consider this issue a little more important.

But, hey, thanks for blogging :)

re: The Case of the Slow Keynote Demo

WhatAboutSecurity — Wed, 27 May 2009 05:31:03 GMT

Great article Mark. However, this demonstrates how easy it is to create a security vulnerability. Just sign some "bad" code with a bogus cert, then include the .config file to turn off cert checking for the app. I certainly hope that you received some form of forced UAC message indicating that the application's cert validity wasn't being checked.

re: The Case of the Slow Keynote Demo

markrussinovich — Wed, 27 May 2009 06:02:46 GMT

@WhatAboutSecurity:

Turning off the check does not expose any security issue or create a 'vulnerability'. A security vulnerability allows for malicious software to gain execution without any authorization.

Signature checks serve as proof-of-origin for trust decisions (e.g. installing an activeX control) and integrity check, not as any indication that the software is non-malicious, free from exploitable defects, or carrying a malicous data payload.

The only code in general checked for signature validity during loading are ActiveX controls, .NET assemblies and device drivers. OS components are not verified except on demand. UAC elevations are not 'security' and the signature verification performed by the consent prompt is intended primarily to encourage ISVs to sign their code.

re: The Case of the Slow Keynote Demo

Aaron Tiensivu — Wed, 27 May 2009 07:36:36 GMT

I honestly wish generatePublisherEvidence enabled="false" was the default for most .NET apps. I understand the reasoning behind the verifications, but there seems that there should be a better way of going about this. As more applications become .NET based, I suspect that KB article is going to get hit more often. Anyone running Exchange 2007/2010 in a lab environment without internet connectivity already knows that KB by heart... most likely. :)

re: The Case of the Slow Keynote Demo

Chris Jackson — Wed, 27 May 2009 07:36:52 GMT

The Stock Viewer demo app is available for the world to use here: http://blogs.msdn.com/cjacks/archive/2008/01/03/stock-viewer-shim-demo-application.aspx. You can steal my demos too, just like Mark does! :-) Unless you digitally sign it yourself, though, you shouldn't need to apply this fix.

re: The Case of the Slow Keynote Demo

Kender — Wed, 27 May 2009 08:24:56 GMT

There are two thing here that I do not fully understand:

- why does it take 12 seconds to realize there's no internet connection?

- why does it try this six times?

re: The Case of the Slow Keynote Demo

Jagannath — Wed, 27 May 2009 09:19:24 GMT

Surprising that people in Microsoft search Web for problems related to Microsoft products, rather than searching the internal/external "Knowledge Base" articles.

re: The Case of the Slow Keynote Demo

Jeroen Ritmeijer — Wed, 27 May 2009 10:29:09 GMT

SharePoint is suffering from this problem as well. It really did my head in until I went through a similar exercise to troubleshoot the slowdown.

In addition to the web.config change there are other ways around it as well, including registry changes to turn this check off for all applications in one go.

This does not reduce the security of the system as, the way I understand it, .net is checking crl.microsoft.com to check if the certificate of the signed assembly has been revoked. As there is no internet connectivity on affected system a failed (and delayed) check is as good or as bad as no check at all.

I have blogged about it here:

http://www.muhimbi.com/blog/2009/04/new-approach-to-solve-sharepoints.html

This was a difficult one to troubleshoot, if an event log entry would have been written it would have literally saved me weeks.

re: The Case of the Slow Keynote Demo

Tim Heron — Wed, 27 May 2009 11:40:03 GMT

This is a particular problem for services written using the .NET Framework since if they do not start within 30 seconds then the SCM will kill them off. Other than applying the hotfix and workaround for all our customers (or trying to force our customers to download the revocation list weekly) the only way round this is to make sure our service executables are not signed. Of course when the SCM kills off an application in this manner the error message in the event log is abysmal.

re: The Case of the Slow Keynote Demo

Daniel — Wed, 27 May 2009 20:35:48 GMT

To Tim Heron,

Putting generatePublisherEvidence=>false in your app.config is the solution for you.

re: The Case of the Slow Keynote Demo

Tim Heron — Thu, 28 May 2009 00:16:18 GMT

Daniel - thanks for re-pointing out what is very clear in the article. A better solution is not to sign the executable, that way I do not need to rely on my customers installing a hotfix and modifying their .config files. You're probably thinking of a nice situation where I don't already have customers out there using the software. Thanks for you help.

re: The Case of the Slow Keynote Demo

Uri Cohen — Thu, 28 May 2009 17:15:49 GMT

Hi Mark, this is a nice coincidence.

I a Dev Lead in MED-V and was happy to find out it was not MED-V demo which caused the pain. 20 lines into the post I thought: "Authenticode!".

When Kidaro was acquired by Microsoft and we replaced our signatures from Kidaro's to Microsoft's, we run into this exact problem...

Took us more than 15 minutes to figure, though. :)

My personal feeling is that when both the caller and the callee are the same company (like in the case of stand-alone products and tools and unlike plug-ins, libraries which provide an API, etc.) Authenticode does not give you any actual benefit over 'regular' .net strong-name, and adds a lot of inpredictability. What do you think?

re: The Case of the Slow Keynote Demo

linuxGeek — Fri, 29 May 2009 08:22:08 GMT

Thanks for this case. I must confess im converted, i run linux. But i still find your article, for lack of a better word, entertaining!

/linuxGeek

WinTrustVerify()

Joe Dietz — Fri, 29 May 2009 17:33:25 GMT

The problem is that WinTrustVerify() was created entirely for internet explorer, it presumes internet connectivity and presumes that making a few HTTP queries has no side effects. It is a layer over cryptlib.dll and coordinates loading some 'subject' dlls that understand various signed file format. However the WinTrustVerify() layer does not expose the various flags to cryptlib.dll routines to disable things like autoroot update and full chain revocation checking (though this last one seems like it should work - it doesn't actually stop the lookup, just stops the policy enforcement). The ONLY solution I found to the problem is to hook the IAT address of LoadLibrary of my app's cryptlib.dll to point to a stub that filters out requests for 'cryptnet.dll'.

cryptnet.dll will also bring in quite a number of other libraries including a script host, win http, ldap, dns etc. in addition to hanging your app in a non-network connected setting.

re: The Case of the Slow Keynote Demo

Chris Sherlock — Sun, 31 May 2009 11:07:29 GMT

@Shawn Lukaschuk: yes, he could have done a Google search to find the issue. Of course, you'd have to know that the issue is to do with signing...

However, imagine if that was NOT the case. A few extra minutes to test the hypothesis would have solved quite a bit of time chasing a red herring.

re: The Case of the Slow Keynote Demo

Wayne Robinson — Thu, 11 Jun 2009 14:26:35 GMT

A real treat to read as always! Thank you for taking the time to write these blogs Mark :)

re: The Case of the Slow Keynote Demo

Mark Andereck — Sun, 14 Jun 2009 00:36:41 GMT

Thanks for the post Mark. I always enjoy being taught how to fish for myself. Sysinternals tools are used nearly daily by my colleagues and myself.

re: The Case of the Slow Keynote Demo

Mon, 15 Jun 2009 06:35:08 GMT

Great stuff, M! You never cease to amaze me!!! Keep it coming! V

re: The Case of the Slow Keynote Demo

Mark-Allen Perry — Thu, 25 Jun 2009 12:44:49 GMT

Many thanks, Mark. Great investigation tips again.

Mark-Allen

re: The Case of the Slow Keynote Demo

Dave — Wed, 01 Jul 2009 17:48:24 GMT

Great as usual Mark. It's always enjoyable to read your articles.

-djv

re: The Case of the Slow Keynote Demo

dunk sb shoes — Wed, 08 Jul 2009 10:53:54 GMT

Dear Mark, I have to admit my perverse satisfaction in seeing you, a Microsoft employee, trip over this bug. Maybe the pain will induce your colleagues to consider this issue a little more important.

re: The Case of the Slow Keynote Demo

generatePublisherEvidence — Fri, 10 Jul 2009 12:38:13 GMT

http://support.microsoft.com/kb/973072

re: The Case of the Slow Keynote Demo

Paul — Fri, 10 Jul 2009 18:54:32 GMT

Greate post. I have a similar issue with a .NET office add-in. The only solution we have found so far is to disable the Check for publisher's certificate revocation option in IE. Any idea how would could apply the app.config fix to Word, Excel, Outlook?

re: The Case of the Slow Keynote Demo

MS Sucks — Tue, 18 Aug 2009 19:56:37 GMT

The proper fix would be to detect if the machine is connected to the internet or not. It should not be timing out after 10 seconds, let alone retrying FIVE MORE TIMES, after one failed attempt. It should know immediately that it is not connected to the internet.

re: The Case of the Slow Keynote Demo

Ganesh — Fri, 25 Sep 2009 08:34:55 GMT

Hi,Mark great read - It was intresting that you zeroed in the issue by initially looking at the delays. Thanks for the insight.

re: The Case of the Slow Keynote Demo

Sid — Wed, 07 Oct 2009 02:04:37 GMT

Another great article. Great use of logic. Looking for the simple issue and zeroing in on it.