Clint Huffman's Windows Performance Blog : PAL

Bling.ps1 (a functional proof of concept for PAL v2.0)

clinth@microsoft.com — Thu, 01 Jan 2009 02:15:24 GMT

Work on PAL v2.0 (PowerShell version) is coming along nicely. While working on it, I created a functional proof of concept PowerShell script called Bling.ps1 which will read a performance monitor log (BLG or CSV) and will create graphical charts for all of the counters in it. PAL v1.x only creates charts and stats for performance counters named in the threshold files, so this is a new feature. Oh, and BLING stands for BLg INto imaGes. :-)

Bling.ps1 v1.0 (Proof of Concept for PAL 2.0)
http://www.codeplex.com/PAL/Release/ProjectReleases.aspx?ReleaseId=21260

No installation is required, but dependent, free, products are required.

Required Products (free):
- PowerShell v1.0 or greater.
- Microsoft .NET Framework 3.5 Service Pack 1
- Microsoft Chart Controls for Microsoft .NET Framework 3.5

Syntax:
Bling.ps1 /LOG:[AbsolutePathToPerfmonLog[;AbsolutePathToPerfmonLog] /OUTPUTDIR:[AbsolutePathToOutputDirectory]

Note: Use full, absolute paths in each of the arguments.

Basic Example:
.\Bling.ps1 /LOG:C:\Users\clinth\Documents\SamplePerfmonLog.blg /OUTPUTDIR:C:\Users\clinth\Documents\Output

See the Readme.txt file for more details.

Note: Be sure to set PowerShell to unrestricted access by typing the following command in an elevated PowerShell command prompt:
Set-ExecutionPolicy Unrestricted

Special Thanks to Greg Varveris for his help with understanding the Microsoft Chart Controls for Microsoft .NET Framework 3.5.

All my posts are provided "AS IS" with no warranties, and confer no rights. For PFE Job Opportunities at Microsoft, please visit our website at: http://members.microsoft.com/careers/search/default.aspx - search for keyword “PFE”
“PFE: The best place to be at Microsoft”

Use the Hyper-V Counters for CPU Measurement

clinth@microsoft.com — Mon, 13 Oct 2008 19:31:40 GMT

As a reminder to everyone, use the perf counter “\Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time” to measure CPU in a Hyper-V environment.

If you are trying to measure CPU utilization in a Hyper-V environment and are a bit frustrated as to why the numbers don’t match up, then read the excerpt below. I spent a month with the BizTalk Product Team doing analysis of BizTalk in a Hyper-V environment. We had the virtual computers (guest) running at 100% CPU, yet Task Manager on the host computer (root partition) was about 1% on all of the CPUs.

Excerpt from http://msdn.microsoft.com/en-us/library/cc768535.aspx

Measure overall processor utilization of the Hyper-V environment using Hyper-V performance monitor counters – For purposes of measuring processor utilization, the host operating system is logically viewed as just another guest operating system. Therefore, the “\Processor(*)\% Processor Time” monitor counter measures the processor utilization of the host operating system only. To measure total physical processor utilization of the host operating system and all guest operating systems, use the “\Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time” performance monitor counter. This counter measures the total percentage of time spent by the processor running the both the host operating system and all guest operating systems.

Also, the PAL tool has a threshold file for Hyper-V with this information in it. http://www.codeplex.com/PAL

BizTalk Server 2006 R2 Hyper-V Guide (Full Guide)

http://msdn.microsoft.com/en-us/library/cc768518.aspx

The Need for a Performance Analysis of Logs Tool

clinth@microsoft.com — Mon, 22 Sep 2008 02:19:40 GMT

Introduction

Performance analysis of log files (*.blg files) in Microsoft Windows has primarily been a manual process for as long as I can remember. We [Microsoft] have made some strides in this area, but there is still little out there that analyzes log files *after* a problem has occurred. Furthermore, nearly all of the Microsoft tools require the user to capture performance data as the problem is occurring. While this is a great way to analyze a performance problem, it is sometimes impractical. This would be similar to asking a criminal to reenact a crime scene while we film it. Therefore, we need a tool which analyzes existing logs and pieces together the evidence after the performance issue (crime scene) has occurred. Until this happens, we must continue analyzing logs manually with our limited, individual knowledge. The knowledge is all around us, we just need to harness it. The Performance Analysis of Logs (PAL) tool (http://www.codeplex.com) is our first step towards the realization of this goal, but we cannot do it alone.

The Challenge

I go on-site with customer every week to assist with performance issues. Most of the time the customer has perfmon logs (*.blg), Event Logs (*.csv), IIS logs (*.log), and if we are lucky a Event Tracing for Windows (ETW) log (*.etl). The assumption is that network administrators should analyze these logs by hand assuming they know their environment best. The problem is that in order to *properly* analyze these log files, you would need have a working knowledge of Windows Architecture. The reality is that most people have a heavy work load and do not have time to thoroughly understand Windows Architecture enough to keep up with it. I know enough about Windows Architecture to put the above log files to use and to typically formulate an hypothesis, but even I struggle with understanding some of the issues that I see and need to rely on others who know more. Even when you have a social network of subject matter experts, the process of analyzing Windows performance can still be slow. Therefore, the knowledge these experts have needs to be consolidated into an analysis tool or a central location. Many people may say that performance analysis is an art versus a science. Well I say, let's take the art out of it and make it a science as much as we can.

One of the fruits of my team's labor towards this goal is the PAL tool (http://www.codeplex.com/PAL). The PAL tool (Performance Analysis of Logs tool) is a tool that takes in all of the variables needed to analyze a performance issue and generates a report on its findings. It does this by interviewing the user to find out more about the computer, then using those answers in its analysis. Now, you are probably asking me, “Why did we write our own tool when there are so many great performance analysis tools that Microsoft has written?” Well, let’s talk about a few of the more recent and relative tool that I am aware of and how PAL is different:

Microsoft Server Performance Advisor (SPA): SPA was written by the Windows Fundamentals team. It analyzes perfmon logs and ETL (Event Tracing for Windows log). It does a great job of aggregating data, but analyses are based on XPath statements which unfortunately is not flexible enough to consider all of the factors in analyzing performance. Furthermore, the problem must be reproduced while it’s gathering data. The tool should only be ran for short time periods due to the large amount of data it gathers, and finally takes in *all* of the data points in the logs as its average values – meaning if the problem occurred for 1 minute and the collection period is 20 minutes, then the averages are skewed. In any case, I am very fond of this tool and highly encourage it's use. As a matter of fact, I'm reusing many of it's concepts in the PAL tool today. SPA has the right idea, it just needs to be taken to the next level. You can download it at: http://www.microsoft.com/downloads/details.aspx?FamilyID=09115420-8c9d-46b9-a9a5-9bffcd237da2&DisplayLang=en

Microsoft Visual Studio Profiler: This tool is very good, but its focus is on application functions versus operating system performance. I firmly believe that this is one of the best approaches to performance analysis for applications because you can identify what functions that the application is waiting on. Unfortunately, you can only profile one process at a time, the profiling has a little bit of overhead, and you have to reproduce the problem as it is occurring. I wrote a white paper on how to do this in a production environment located here (http://go.microsoft.com/fwlink/?LinkId=105797). I recommend using both this tool and a performance counter analysis tool such as PAL.

Microsoft xPerf/xTrace: Written by the Windows Fundamentals team. This the latest/greatest tool out there for perf analysis. Unfortunately, it is currently lacking an intuitive UI and analyses around the data collected. With that said, they are rapidly improving it. Like many of the other tools you must be able to reproduce the problem as it is occurring. Furthermore, it only runs on Windows Vista and Windows Server 2008. Unfortunately, customers are not continuously logging ETW data, so this doesn’t help me much for post analysis. Finally, it only analyzes ETL – no other log format. This would be like asking a crime investigator to analyze a crime scene using only one type of evidence. If that type of evidence isn’t available, then no analysis can be done. xPerf is part of the Windows Performance Toolkit located here: http://www.microsoft.com/whdc/system/sysperf/perftools.mspx

Microsoft System Center Operations Manager (SCOM): I’m always impressed with the SCOM team and their product. They analyze perf data as they go and do a great job with providing guidance and trends on the data shown. While this great for customers who have SCOM… not all customers have it installed. Therefore, I am again left with manual analysis. Furthermore, SCOM might not have all of the data I need to analyze a problem.

Microsoft Log Parser: This is an incredible tool that parses many log types in an easy to use Sequel Query Language (SQL) syntax. It just doesn't do any analysis. Therefore, the PAL tool uses Microsoft Log Parser as its data access layer. Microsoft Log Parser can be downloaded at: http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&DisplayLang=en

There are certainly more tools out there, but my point is that none of the tools above meet the all of the needs of post log analysis. This is why the PAL tool initiative was started.

The Solution

I strongly feel that if you complain about something, then you need to offer a viable solution, so here are the requirements of a tool that would be of practical use in the field.

Consolidated Guidance: First, there needs to be a central repository of guidance on performance analysis. We have many great whitepapers out there, but the knowledge is spread out and it takes a great deal of time to read and understand them especially when you are trying to solve a problem. It’s like a guy bringing his car to an automotive repair shop and the mechanic hands the guy a huge book and says, “you can fix your car by reading this”. The guy will ask, “this is nice, but how to I fix my problem?” Shane Creamer’s Vital Signs workshop has done a great job with consolidating the basics into a short, 2-day workshop offered by my team (Microsoft Premier Field Engineering). The PAL tool has nearly all of the consolidated guidance from the Vital Signs workshop built-into its report, so when a threshold is broken the guidance is context sensitive to that threshold. If you are interested in the Vitals Signs workshop, then contact your Microsoft Technical Account Manager (TAM).

Log File Data Access Layer (DAL): Next, a simple to use data access layer is needed to analyze log files in a common way. The Microsoft Log Parser tool is a great tool for this, but it is based on legacy COM. No future versions of it are planned at this time, but I have asked the IIS product team to write a new version it. They are considering it. Currently, the PAL tool uses Microsoft Log Parser as its DAL, but inherits some of the same limitations of Log Parser because of this.

Analyze More Data Points: Some of our log analysis tools such as SPA reads in the entire log and generate an average, minimum, and maximum values from it. This doesn’t cut it when the problem occurs only in a small portion of the log because the problem is averaged out by the sheer size of the counter log. The PAL tool breaks down perfmon logs into smaller time slices and analyzes each time slice individually for better accuracy.

Dynamically Changing Thresholds and Interviewing the User: One of the assumptions most performance analysis tool make is that they assume the user knows how to change the threshold and what to change them to. Likewise, unless you are a Windows architecture guru, then you as the user assume the tool is using the appropriate thresholds. When you have both parties relying on the other to make the best decision, then this can cause confusion and misdiagnosis. In order for next generation tools to be effective, they need to have dynamically changing thresholds based on the environment. The point is that you need to have a tool that learns the customer’s environment and adjusts its thresholds appropriately even if this means simply asking the user for some additional input. For example, to determine if a computer is running out of paged pool or non-paged pool memory the PAL tool asks the user a series of questions to estimate the maximum sizes of these memory pools, then computes a respective 60% and 80% threshold for it. The PAL tool does this by running executable code at run-time using the user’s input as variables for the code to determine if the thresholds are broken. Using executable code as the thresholds and being able to ask the user questions about the environment makes the tool flexible enough to handle nearly any performance analysis challenge.

Reusable: Our next generation tools need to be reusable – meaning portions of it can be reused by other applications and tool. Luckily, tools like xPerf are modulized in this way, but I wanted to emphasize that this needs to continue. Currently, the PAL tool is a hybrid VB.NET/ VBScript and open source, so users can simply copy code they want to reuse.

Free and Public: Our next generation tools need to be free for all of our users. Many times when tools become intellectual property (IP), then they inherit licensing restrictions such as a cost to use or other restrictions. The PAL tool is a free, open source tool available at http://www.codeplex.com/PAL.

Extensibility of the Thresholds: As mentioned above, the thresholds need to be executable code to be flexible enough to handle complex analyses. In addition, the code needs to be open to where users can add to them or update existing ones. This is important because no one person can claim that they know all of the technical aspects of all performance problems in Windows. You have to allow the people who are experts in their field to have the empowerment to add, edit, and delete the thresholds in the tool. The PAL tool accomplishes this by using VBScript for the thresholds and the VBScript is embedded in XML based threshold files. Included with the PAL tool is an editor to make it easy for subject matter experts to edit PAL threshold files. Furthermore, with the help of other subject matter experts, we have several product specific threshold files namely Active Directory, IIS, MOSS, SQL Server, BizTalk Server, Exchange Server, and general Windows.

Low Requirements: Some of the tools written by other teams at Microsoft such as the Visual Studio Load Test tool require a back end SQL database in order to process the collected data. Many people in the field don’t have SQL Server running on their laptops, so our tools need to able to run on workstation class computers. The PAL tool simply requires Microsoft Log Parser and Office Web Components 11 both which are free downloads.

Conclusion

We need a tool that can analyze a wide variety of logs similar to how a crime scene investigator analyzes a crime scene by gathering the evidence from the scene (in this case the log files) and analyzing them with scientific precision. The PAL tool is a tentative solution to the problem and has enjoyed great success with it with over 2000 downloads per month. With that said, we cannot do this alone especially since this is not part of my regular job. While a few of the Microsoft product groups are starting to follow some of the concepts of the PAL tool, no product group at Microsoft that I know specializes in performance analysis in this fashion. The possibility of creating a tool with all of the aspects I mentioned above is difficult. With that said, the benefits of such a tool are clear – the better Microsoft Windows performs, the happier customers are with Microsoft products. In the end, my real intention is to simply make my parents computer run faster. ;-)

Moving Forward

If you want to assist with this effort, then please try out the PAL tool and help with the development of it. For more information on the PAL Tool, please go to http://www.codeplex.com/PAL.

How to use the PAL Tool for BizTalk Performance Analysis

clinth@microsoft.com — Wed, 03 Sep 2008 00:41:06 GMT

Sorry, I haven't blogged much in the past few months. I've been heads down on several exciting BizTalk projects which I'll blog about soon after this one. For now, I want to get more of the word out about the Performance Analysis of Logs (PAL) tool that I wrote in collaboration with other people on my team and how to use the tool to analyze BizTalk servers. I use the PAL tool when I conduct BizTalk Health Checks for customers. I'm a field guy (Microsoft Premier Field Engineer) who lives for performance issues and the tool takes the mundane work out of performance analysis. With that said, PAL is not a replacement for performance analysis - it is simply a really nice time saver!

The PAL (Performance Analysis of Logs) tool is a new and powerful tool that reads in a performance monitor counter log (any known format) and analyzes it using complex, but known, thresholds (provided). The tool generates an HTML based report that graphically charts important performance counters and throws alerts when thresholds are exceeded. The thresholds are originally based on thresholds defined by the Microsoft product teams, including BizTalk Server, and members of Microsoft support. This tool is not a replacement of traditional performance analysis, but automates the analysis of performance counter logs enough to help save you time. This is a VBScript and requires Microsoft Log Parser (free download). The tool is available at http://www.codeplex.com/PAL.

BizTalk Performance Counters

Microsoft BizTalk Server 2006 shipped with about 294 performance counters. This means a BizTalk Server implementation with at least 2 servers for redundancy means there are at least 588 BizTalk performance counters that may need to be analyzed. Therefore, a performance monitor log analysis tool is helpful with analyzing BizTalk performance counters.

Consider the Operating System Performance

Many BizTalk performance issues can be narrowed down by analyzing the resources of the operating system (CPU, disk, memory, and network). For example, if the tracking (DTADB) database file is heavily using a disk, then reducing the amount of tracking would be a logical step towards alleviating the bottleneck.

Usage

The PAL tool is can be used to create a Microsoft Performance Monitor (perfmon) template file from PAL threshold files and can analyze the perfmon log after its collection period. In this example, we will create, gather, and analyze a BizTalk performance monitor log file.

Collect a Microsoft Performance Monitor (perfmon) log

1. Export a Microsoft Performance Monitor Log from the BizTalk Server threshold file.

a. Selecting the Microsoft BizTalk Server Server 2006 threshold file and clicking the Export… button, then save the file. This is a perfmon template file.

2. Copy the perfmon template file to the BizTalk server(s) you have chosen for analysis. This includes the Microsoft SQL Servers hosting the BizTalk databases.

3. Create a new performance monitor log using the template exported in step 1. For more information on how to create performance monitor logs from a template file, please refer to the Windows help documentation.

a. Adjust the perfmon log settings if needed.

4. Start the new performance monitor log. Stop the perfmon log when the collection period is over.

5. Copy the perfmon log to the installation directory of the PAL tool.

Filling out the PAL Wizard Form

Counter Log Path: Specify the path to the Microsoft Performance Monitor (perfmon) log. The log can be in any of the known perfmon log formats such as BLG (binary) or CSV (text). Use the ellipsis button to browse for the perfmon log. If the perfmon log file is in the installation directory of PAL, then use the drop down arrow to select one of the perfmon log files.
Note: Multiple log files can be merged by separating them with semicolons, but this may produce unpredictable results. PAL is best used with logs containing data from a single computer.
Date/Time Range: This is the date/time range that you can restrict the analysis to. For example, if you did load testing on BizTalk during a specific time.
Threshold File Title: This is the threshold file that you want to use to determine thresholds that will be analyzed by PAL. If you do not see a product specific threshold file, then consider using the default System Overview threshold file. The System Overview threshold file analyzes the basic operating system performance counters only.
Question Variable Names: The question variables will be different depending on which threshold file is selected. Click on each question variable name and answer the question in respect to the perfmon log chosen earlier.
Analysis Interval: Specify the time (in seconds) that you want the PAL tool to analyze the perfmon log. Choose “AUTO” to have PAL automatically detect and choose an appropriate interval size. Choose “ALL” to have PAL analyze all of the data in the perfmon log. Be careful about choosing “ALL” because it is extremely resource intensive. The Analysis Interval determines how the perfmon log will be broken up when analyzed. For example, if you gathered a 24 hour log and choose a 1 hour analysis interval, then PAL will analyze each hour in the log for minimum, maximum, average, and thread values for that time interval. The data points in the charts generated by PAL are based on the analysis interval. “AUTO” is the default and is recommended.
Output Options: You can optionally specify an output directory for the output, the file name format for the HTML report, and an optional XML document output. The XML output is useful if you want to analyze the results using another tool.
The Queue: The queue is really just a batch file with line feeds for readability.
Execute:
- Execute: Execute what is currently in the queue.
- Add to Queue: Do not execute yet, but add the current configuration into the queue and restart the wizard allowing you to add more. This is useful if you intend to "batch" the processing while you go to lunch or overnight.
- Execute and Restart: This executes what is in the queue now and restarts the wizard. This is useful if you want to start processing one of the logs right away while answering the questions for the other logs.
Execute as a low priority process: This will run the PAL.vbs script at a low process priority so that when PAL.vbs is running, it will have the little to no impact on your computer. PAL processing can be very resource intensive. This option will make the processing take longer.

Interpreting the Report

The report generated by the PAL tool is simply an interpretation of the perfmon log data using generalized thresholds. The intention is to assist with performance analysis, but not to replace traditional performance analysis. Therefore, you should have a working knowledge of the BizTalk architecture and BizTalk performance analysis.

The thresholds used in the BizTalk threshold file include operating system thresholds (CPU, disk, memory, and network), Microsoft SQL Server counter thresholds, and BizTalk counter thresholds. The BizTalk thresholds generally focus on host throttling, adapter latency, service instance statistics (suspended, dehydrated, etc.), database sizes, and memory usage.

The PAL report is separated by categories and in each category is a collection of analyses. Each analysis focuses on a specific performance counter. If any of the thresholds are exceeded, then an alert is raised. The number of alerts in each analysis and the alert condition (typically Warning or Critical) typically indicate the severity of the results. For example, a critical alert in Pool Paged Bytes is a very serious condition and should be resolved immediately. When critical alerts occur, go to the analysis section associated to the alert. In the analysis section, there is typically a description of the analysis, a description of the thresholds used, and a link for more information on the topic.

The analyses in the report contain content describing the purpose of the analysis, why the thresholds are there, and references to more information. To learn more about interpreting the PAL report for BizTalk analysis, then read the following article:

Using the Performance Analysis of Logs (PAL) Tool
http://msdn.microsoft.com/en-us/library/cc296652.aspx

Identifying 32-bit Kernel Memory Issues

clinth@microsoft.com — Tue, 02 Oct 2007 00:55:00 GMT

I'm a Premier Field Engineer (PFE) and I go onsite with customers on a regular basis to conduct Health Checks. This is my first blog posting on TechNet, but I figured it would be on something important versus "Hello World". ;-)

More and more I am seeing customers who are not aware of kernel memory issues on the 32-bit Windows architecture. If you are running 32-bit Windows 2000 or 32-bit Windows 2003, then check the kernel memory. Lack of kernel memory can lead to system-wide hangs which seem unexplainable, so this is a serious issue. Always generally use and/or recommend 64-bit or Windows Server 2008 server to avoid these issues (Windows Server 2008 and Windows Vista have automatically adjusting kernel memory pool sizes). This issue and other performance issues are addressed in the Vital Signs workshop (written by Shane Creamer), which can be delivered by my team, Premier Field Engineering (PFE) - just contact your Technical Account Manager (TAM) if you are interested in this course.

Here is a kernel memory chart for Windows 2003 Server:

Memory

Default ( /PAE for 6-16GB )

/3GB

1GB

Free System PTE: 51k

Paged Pool: 282MB

Non Paged Pool: 212MB

Free System PTE: 32k

Paged Pool: 163MB

Non Paged Pool: 131MB

2GB

Free System PTE: 196k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 16k

Paged Pool: 262MB

Non Paged Pool: 131MB

3GB

Free System PTE: 195k

Paged Pool: 360MB

Non Paged Pool: 262MB

Free System PTE: 14k

Paged Pool: 262MB

Non Paged Pool: 131MB

4GB

Free System PTE: 106k

Paged Pool: 336MB

Non Paged Pool: 285MB

Free System PTE: 15k

Paged Pool: 258MB

Non Paged Pool: 154MB

6GB

Free System PTE: 186k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 239MB

Non Paged Pool: 131MB

8GB

Free System PTE: 182k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 225MB

Non Paged Pool: 131MB

12GB

Free System PTE: 175k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 196MB

Non Paged Pool: 131MB

16GB

Free System PTE: 167k

Paged Pool: 366MB

Non Paged Pool: 262MB

Free System PTE: 12k

Paged Pool: 169MB

Non Paged Pool: 131MB

Check your 32-bit servers to see if they are within 80% of the memory pool sizes for Pool Paged Memory and Pool non-Paged memory according to the chart above. The Free Page Table Entries (PTEs) listed in the chart above is the starting amount based on the specification of the hardware and boot.ini switches. Here are the performance counters to monitor:

Memory\Free System Page Table Entries

Memory\Pool Nonpaged Bytes

Memory\Pool Paged Bytes

For example, if I have a server that is a Windows 2003 SP1 (SP1 is needed to accurately see the Free PTE data in performance monitor) with 4GB of memory and /3GB switch turned on, then Free System PTE’s start off at 15,000 (according to the chart). If the value gets below 5,000, then the system could hang temporarily. This system has a Paged Pool memory maximum of 258MB and Non-Paged Pool maximum memory of 154MBs. If the respective “Memory\Pool Paged Bytes” and “Memory\Pool Nonpaged Bytes” counter values come within 80% of these maximum pool sizes, then the system could hang temporarily – in this case if the values go over 206MB’s and 123MB’s respectively, then it’s a critical issue.

Fix for Win2003 SP1 systems with /3GB and low on PTE’s: If the system is low on PTE’s, running Windows 2003, and using /3GB switch, then consider using the /USERVA switch to give back some of the memory to the kernel. Note, this only works for Free System PTE issues. For more information on the USERVA switch, go to:

How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB

http://support.microsoft.com/kb/316739

Lack of Paged Pool or non-Paged Pool Memory: If the system is low on Paged Pool or non-Paged pool memory, then first consider opening a support case with Microsoft to address this. Alternatively, you can use a free and public tool called Poolmon.exe to see what DLL’s are using kernel memory (see the article below). Most kernel memory leaks can be tracked back to a usermode process. To identify which user mode process is responsible, reboot the system (so you start off with a clean system), start a performance monitor log intending to run for a week or more capturing the Memory and Process objects, then analyze the perfmon log looking for memory leaks and/or handle leaks in one or more of the processes.

How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Mode Memory Leaks

http://support.microsoft.com/kb/177415

In summary, always consider 64-bit and always keep an eye on kernel memory.

In addition, I and a few of my colleages wrote a tool that can help identify these and many other performance issues by analyzing performance counter logs. The tool is located at http://www.codeplex.com/PAL. Codeplex.com is Microsoft's open source web site.