Recently, I came across something very interesting. I was working with a customer who was working with several internally developed applications that leveraged HTML files by creating links that would open them inside the user’s default browser. These applications can easily be virtualized with both App-V 5.0 and 4.6 (VAE, <LOCAL_INTERACTION_ALLOWED>)
What was happening was that these applications were behaving oddly when running virtualized with App-V. The applications would trigger the local browser (running outside the bubble) for these help documents (in this example, Internet Explorer 8.) While there were no issues with this particular function, every time a user would close one of the Internet Explorer windows containing one of these documents, the window would disappear as normal. Then, almost a second or two later, a window would pop up stated that the application had crashed.
Oddly enough, we knew pretty quickly that this had to be somewhat environmental because we could never prove these issues on a vanilla test machine. This was not due to a limitation or a potential code defect within the App-V virtualization engine. After rudimentary elimination of all factors (I.E Settings, App-V, GPO, branding from the IEAK) – we decided to just cut to the chase and debug it with WINDBG to determine why we could not reproduce the issue outside the customer’s environment.
Of course there are several ways to collect user dumps (process dumps.) In this case, the issue was happening on Windows 7 so often, the default AE (Application Experience) debugger - WER (Windows Error Reporting) will suffice. We configured WER to generate a user dump by making a few registry changes. We gave the customer the following .REG file to import into one of the offending machines.
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps]"DumpFolder"=hex(2):63,00,3a,00,5c,00,64,00,75,00,6d,00,70,00,73,00,00,00"DumpCount"=dword:0000000a"DumpType"=dword:00000002 This created a full user process dump and it put the location of the user dumps in the C:\Dumps folder.
Then once we had the user dump we took a look at the stack trace of the corrupting shutdown thread inside of WINDBG:
0:000> kChildEBP RetAddr 0013f714 77476a04 ntdll!KiFastSystemCallRet 0013f718 75656a36 ntdll!NtWaitForMultipleObjects+0xc 0013f7b4 75dfbd1e KERNELBASE!WaitForMultipleObjectsEx+0x100 0013f7fc 75dfbd8c kernel32!WaitForMultipleObjectsExImplementation+0xe0 0013f818 75e105df kernel32!WaitForMultipleObjects+0x18 0013f884 75e1087a kernel32!WerpReportFaultInternal+0x186 0013f898 75e10828 kernel32!WerpReportFault+0x70 0013f8a8 75e107a3 kernel32!BasepReportFault+0x20 0013f934 774a7f02 kernel32!UnhandledExceptionFilter+0x1af 0013f93c 7744e324 ntdll!__RtlUserThreadStart+0x62 0013f950 7744e1b4 ntdll!_EH4_CallFilterFunc+0x12 0013f978 77477199 ntdll!_except_handler4+0x8e 0013f99c 7747716b ntdll!ExecuteHandler2+0x26 0013f9c0 7744f98f ntdll!ExecuteHandler+0x24 0013fa4c 77476ff7 ntdll!RtlDispatchException+0x127 0013fa4c 5483ccd4 ntdll!KiUserExceptionDispatcher+0xf WARNING: Frame IP not in any known module. Following frames may be wrong.0013fd60 7748d690 <Unloaded_PseudoServerInproc.dll>+0xccd40013fd7c 7748e3d9 ntdll!RtlProcessFlsData+0x57 0013fe14 7748e12f ntdll!LdrShutdownProcess+0xbd 0013fe28 75e0bbd6 ntdll!RtlExitUserProcess+0x74 0013fe3c 775836dc kernel32!ExitProcessStub+0x12 0013fe48 77583371 msvcrt!__crtExitProcess+0x17 0013fe80 775836bb msvcrt!doexit+0xac 0013fe94 0103129e msvcrt!exit+0x11 0013ff1c 75dfed4c iexplore!__wmainCRTStartup+0x164 0013ff28 7749377b kernel32!BaseThreadInitThunk+0xe 0013ff68 7749374e ntdll!__RtlUserThreadStart+0x70 0013ff80 00000000 ntdll!_RtlUserThreadStart+0x1b There was this external DLL (not part of the standard Windows build) that was loaded into the stack and ad appeared to be partially or fully unloaded by the time that WER could capture the exception. We wanted to see if this DLL was being injected by way of AppInit_DLLs key. At the time, we did not know what this particular DLL was part of. All we knew was PseudoServerInproc.dll appears to be an unknown DLL that was injected into the IE process We went to AppInits_DLL in the registry and found the MFAHook which is a commonly known master hook DLL used in Citrix products. We disengaged all DLLS in that Key by using the following .REG file:
Windows Registry Editor Version 5.00[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows] "LoadAppInit_DLLs"=dword:00000000” The issue immediately went away. Now that we knew it was tied to one of ythe Citrix products being leveraged on the machine, we went back to the AppInit_DLL key to examine MFAHook. This will require us going further to investigate which specific hook DLL is the issue. We know the DLL was PseudoServerInproc.dll So we went into the Citrix configuration to get all of the specific hook agents and the processes they inject into and found our DLL under the following registry key: Key: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\HDXMediaStreamForFlashValue: FilePathNameData: C:\\Program Files\\Citrix\\ICAService\\PseudoServerInproc.dll
A subkey denotes the exe's it hooks into: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\HDXMediaStreamForFlash\iexplore.exe We found that by deleting it – this also fixed the issue. Upon further investigation with Citrix, we found that this was related to a known issue with one of there products from their VDI suite - AND - they already had a fix for this issue (which worked like a champ!~)More info here:http://forums.citrix.com/thread.jspa?threadID=307236&tstart=0
On a side note, if you want to use a more extensive tool for collecting these kind of crashes for analysis, I would highly encourage you to download ProcDump and configure it to be your default application experience debugger. You can enable it by importing the following .REG file:
Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug]"Auto"="1""Debugger"="C:\\procdump\\procdump.exe /accepteula -ma %ld C:\\Dumps""UserDebuggerHotKey"=dword:00000000
[HKEY_LOCAL_MACHINE\SOFTWARE\WoW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug]"Auto"="1""Debugger"="C:\\procdump\\procdump.exe /accepteula -ma %ld C:\\Dumps""UserDebuggerHotKey"=dword:00000000
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\DbgManagedDebugger]"Auto"="1""Debugger"="C:\\procdump\\procdump.exe /accepteula -ma %ld C:\\Dumps""UserDebuggerHotKey"=dword:00000000
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\DbgManagedDebugger]"Auto"="1""Debugger"="C:\\procdump\\procdump.exe /accepteula -ma %ld C:\\Dumps""UserDebuggerHotKey"=dword:00000000 This will register Procdump as the default post-mortem debugger instead of WER. It will intercept the exception and will store a dump file in the specified folder.