Thoughts from the EPS Windows Server Performance Team
A while ago, I got the opportunity to work on an interesting case where the customer’s Explorer process was showing a continuous increase in handle count. Using Process Explorer we could see that these handles were open to various Iexplore.exe processes, which were showing as terminated. Interestingly however, these Iexplore.exe processes were not being started by any user. They seemed to get created randomly, about one every half hour and almost immediately showing up as a terminated process handle under Explorer.exe.
So what was causing these processes to be launched? Putting these processes under a debugger with a breakpoint set on CreateProcess was an option, however we did not have access to the server and getting internet access on the server would be difficult. So I thought of giving Process Monitor a try. The idea was to get log captured for the processes Iexplore.exe and Explorer.exe for the operations process create, process start, and thread create. Also, we wanted to ensure that when we leave this running, Process Monitor did not fill up the pagefile, which is used as the default backing file.
So we did the following:
1. Launched Process Monitor with the following syntax “procmon /backingfile:E:\processlaunch.pml”
2. In the Filters menu, checked the option “Drop Filtered Events”.
3. Set filters for processes Explorer.exe and Iexplore.exe and also for operations process create, process start and thread create.
With this done, we let the server run for a couple of hours and got the logs. Here’s what we saw.
Now, looking at the thread stack for Explorer, process create, we see unknown module, with addresses 0x10003d2f,0x10002298,0x10002629.
First off 0x10000000 converts to 268435456. This is essentially greater than the 2 GB user mode, virtual address space limit. The box was running with the /3GB switch, so this is a valid user mode address; however Explorer.exe and Iexplore.exe are not /LargeAddressAware, which definitely looks suspicious.
Now looking at the stack information of the thread creation of Iexplore.exe we see the following:
Seems we have a binary, Linkinfo.dll, and its loading from %windir% directory. Now the file name seems genuine, however a legitimate version of a system file like Linkinfo.dll is supposed to be loaded from the System32 directory and not the %windir% directory. Also the box we were working with was running Windows Server 2003, and Windows Server 2003 file versions start as 5.2.3790.xxxx. This in combination with the load address of 0x10000000 makes this look out of the ordinary.
Doing a Bing search on Linkinfo.dll in the %windir% directory led me to this link:
Running a free Onecare online scan from the following link confirmed this, and was successful cleaning this up.
You may wish to review/remove the above paragraph - converting the default load address (in hex) set by a VS native DLL project (0x10000000) to base 10 and comparing that to the 2 GB user-mode virtual address limit seems not quite right.
Very good analysis of the problem, and it's nice to see the way you use process monitor and cross compare Windows OS startup switches.
What I cannot understand is that you say that you have no Internet connection ("getting internet access on the server would be difficult") and then you ran an online Onecare scan...
Good job anyway.
I really love these articles that show detailed troubleshooting processes to solve real problems. They are enlightening, good job. However, please allow me clear up something.
Actually, address 0x10000000 falls much below the 2 GB boundary, as it equals to 256 MB. This is the default preferred base address which the Microsoft linker and probably other linkers assign to DLL files, unless the /BASE switch is passed with a different address. The executable headers may have confirmed that this wouldn't be a coincidence and that the DLL wasn't relocated. The interesting part is how the rogue module wasn't noticed until the trace from Process Monitor was carefully analyzed. In a Windows XP-based system, the version number of the file might have fooled many people. Moreover, 32-bit processes which don't have set the LargeProcessAware flag always receive a space address of 2 GB, even if the system is running with the /3GB switch or it is a 64-bit system.
Thanks Ramon for the correction 0x10000000 = 256MB , bad Math on my part
This Article is really worth reading. It explains the OS internal process and very much useful to the guys who are into system performance tuning. for 0x10000000, logically, the memroy utilization for user mode process should not exceed the 2 GB limitation of virtual memory addressing space. As this may lead other process running sort of memory hance causing system unstability.
Great logical and structured troubleshooting !!