Applies to:

Windows Server 2008 R2

Windows 7

Windows Server 2008

Windows Vista

Windows Server 2003

Windows XP

 

In Task Manager, have you ever noticed ‘Handles’?

image

If you haven’t, don’t worry.

 

So what are handles?

Every single application and service (process) has to have handles to various Kernel system resources, such as semaphores, communication
ports (i.e. TCP ports, windows sockets (a.k.a. WinSock)), and files, that are accessible to all threads in the process.

Applications and services cannot directly access these Kernel resources.

A great write up by Mark Russinovich is available here Pushing the Limits of Windows: Handles

 

So how do I find out which process is consuming the most amount of handles?

image

In Task Manager, by default, you won’t be able to see the handles open by a process.

 

image

You will need to click on View > Select Columns…

image

You will need to enable handles.

image

Now, you should be able to see a column named ‘Handles’.

 

What are the thresholds (estimates, not a hard and fast rule)?

32-bit Windows XP, Server 2003, Vista, Server 2008, 7:

For most processes, if higher than 2,500 handles open, investigate.

Exceptions are:

System 10,000

lsass.exe 30,000

store.exe 30,000

sqlsrvr.exe 30,000

64-bit Windows XP, Server 2003, Vista, Server 2008, 7, Server 2008 R2:

For most processes, if higher than 3,000 handles open, investigate.

Exceptions are:

System 20,000

lsass.exe 50,000

store.exe 50,000

sqlsrvr.exe 50,000

 

image

Sort it by ‘Handles’, and you will get the biggest consumer of handles.

In this example, we are using the Microsoft Sysinternals TestLimit64.exe application.  We are able to see that it has over 16 million handles open.

 

Now the question becomes, what type of handles are open by the application?

There are tools such as the Microsoft Sysinternals tools such as Handle and Process Explorer that will let you look into the open handles.

In this blog, we are going to be using Process Explorer since it is UI driven and it is easier for most of the audience.

image

In Process Explorer, to see the handles used by a process, you will have to go thru a few steps.

image

Click on View > Select Columns…

image

Click on the ‘Process Performance’ tab > ‘Handle Count’

image

What you will notice is the new column called ‘Handles’

image

Move the ‘Handles’ columns next to ‘Working Set’

image

Sort it by the ‘Handles’ columns

Now, we have the process that is consuming the most amount of handles just like in Task Manager.

But we still don’t see what type of handles they are.

image

Click on View > Lower Pane View > Handles

Note:  This will enable the “Show Lower Pane” by default.

image

Click View > Show Unnamed Handles and Mappings

image

What you will see is the second pane at the bottom that has ‘Type’ and ‘Name’

image

In our sample application, they are ‘Event’ handles to ‘\Sessions1\1\BaseNamedObjects\TestLimit’

image

So, each handle that is open, will consume paged or nonpaged pool (kernel memory) which on 32-bit systems, there is a finite amount available.

Thus, handle leaks, can and will cause Windows clients and servers to hang.

In the example above, we could see that it consumes 112 Non-Paged pool memory(kernel memory).

Note:  To see what type of handles are the most used for the application, you will want to click on File > Save As > .txt and load it in Excel to sort by handle type. 

 

Ok, so now that we have the handle type and name, what’s our next steps?

If you are lucky, if the handle leak is related to a Microsoft process, you could do a search in our (Microsoft) Knowledge Base http://support.microsoft.com and find the root cause.

Keyword:  Handle count

If it is a 3rd party process, please check their Knowledge Base articles.

Hint:  A lot of times, the 3rd party vendors and Independent Software Vendors (isv’s) will not use the word ‘handle leak’, instead will use ‘memory leak’ in their KB’s.  Thus, if you don’t get any results using ‘handle leak’, change the keyword in your searches to ‘memory leak’.

 

After you search the Microsoft and 3rd party KB’s and end up empty handed.  To tackle a ‘new’ handle leak, we will need to:

Enable handle tracking on the application using Microsoft Application Verifier or use Gflags which is a part of of the Debugging Tools for Windows.

Once you have enabled any of the handle tracking tools, we will need to use one of the debuggers such as WinDbg Download and Install Debugging Tools for Windows

 

Option 1a)  Enable handle tracing on the process by using Global Flags (GFlags):

Go to c:\Program Files\Debugging Tools for Windows (x64)

image

Double click on gflags.exe

image

Click on the ‘Kernel Flags’ tab

Check the box next to ‘Enable bad handles detection’

image

Click on the ‘Image File’ tab

Next to ‘Image: (TAB to refresh)’, enter the path and the executable name.

For example:

c:\program files\Microsoft Sysinternals\testlimit64.exe

And then click on the ‘Tab’ key

Next to ‘Stack Backtrace: (Megs):’ enter ‘10’

Click on Apply

Click on Ok

 

OR

Option 1b) Enable handle tracing on the process by using Global Flags (GFlags):

 

 

image

Click on Next

image

Click on ‘I accept the terms in the License Agreement’

Click on Next

image

Click on Install

image

Click on Finish

 

image

Click on All Programs > Application Verifier (x64) > Application Verifier (x64)

 

image

You will see the “Application Verifier 4.0 (x64) tool above.

image

Click on File > Add Application

image

For example:

Point it to c:\program files\microsoft sysinternals\ folder

Next to ‘File name:’ testlimit64.exe

Click on Open

image

You will see this UI.

image

Expand ‘Basics’ and you will notice that there are a good amount of options (Exceptions, Handles, Heaps, Input/Output, Leak, Locks, Memory, Threadpool, TLS) checked by default.

image

Uncheck all, except for ‘Handles’ since it is what we are troubleshooting.

Click on Save

image

You will get this informational, click on OK

Click on Exit

 

image

Right click on WinDbg > Run as administrator

image

Click on File > Symbol File Path…

image

Under “Symbol Search Path”, type srv*c:\websymbols*http://msdl.microsoft.com/download/symbols

image

Click on “File, Attach to a Process…”

image

Scroll down to the application that you want to monitor for high handle consumption.

In our example here, we are clicking on “testlimit64.exe”

Click on Ok

image

Once the debugger (in this case, WinDbg) opens up.

image

Type “.logopen c:\temp\handleleak_testlimit64_MMDDYEAR_HHMMSS.txt” without the quotation marks and then press Enter.

Note:  Where MMDDYEAR is the Month, Date and Year.

Where HHMMSS is the Hour, Minute and Seconds.

and ApplicationName is the actual process name.

image

 

image

Type “!htrace enable” without the quotation marks and then press Enter.

image

Note:  Notice that you will get information such as “Handle” and that it’s opening (OPEN).

And you will also see the stack being called.

If you don’t have symbol resolution to the application that you are troubleshooting, then you will get an offset that is big instead of the function names.

In this example:

testlimit64+0x0000000000001b96

testlimit64+0x0000000000003037

image

Type “!htrace –snapshot” w/o the quotation marks, and then Press Enter.

image

You will get the following return informational “Handle tracing information snapshot successfully taken”

image

Type “g” without the quotation marks and then Press Enter.

Note:  g for Go.

image

Reproduce the issue.

image

Once the reproduction is done.

Click on Debug > Break (or just CTRL+Break)

image

Type “!htrace –diff” without the quotation marks and then press Enter.

image

Note:  By default, Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2 keep a history of 4000 handles open and close operations. 

With !htrace you can enable to keep a much higher history by doing the following:

image

Type “!htrace –enable 0x20000” without the quotation marks and then press Enter.

In this example, we are increasing the handle history to 128000 (decimal, 0x20000 hexadecimal).

image

 

Then go thru

!htrace –snapshot

g

Note:  Reproduce the leak, or let the app leak

CTRL-Break

!htrace –diff

image

Note: Notice that you will get information such as “Handle” and that it’s opening (OPEN).

And you will also see the stack being called.

If you don’t have symbol resolution to the application that you are troubleshooting, then you will get an offset that is big instead of the function names.

If this is not an internally built application, you will need to engage the application vendor for further troubleshooting.

 

Once you are done with the troubleshooting, make sure to disable GFlags and AppVerifier.

References:

Windows Internals 5th Edition

232830 How To Determine File Handle Ownership
http://support.microsoft.com/?id=232830

!handle
http://msdn.microsoft.com/en-us/library/ff563187(v=VS.85).aspx

!htrace
http://msdn.microsoft.com/en-us/library/ff563208(v=VS.85).aspx

Here are two ways of walking thru and using Handle Trace (!htrace) to monitor who is leaking handles:

Use !htrace to debug handle leak
http://blogs.msdn.com/b/junfeng/archive/2008/04/21/use-htrace-to-debug-handle-leak.aspx

If you learn better by visualization, Jeff Dailey did a great job in recording in MSDN's Channel 9:

Understanding handle leaks and how to use !htrace to find them
http://channel9.msdn.com/Blogs/jeff_dailey/Understanding-handle-leaks-and-how-to-use-htrace-to-find-them