Welcome to Part Four of our Server Hang troubleshooting series.  Today we are going to discuss PTE depletion and Low Physical Memory conditions and how those two issues can lead to server hangs.  In our post on the /3GB switch we mentioned that in general, a system should always have around 10,000 free System PTE’s.  Although we normally see PTE depletion issues on systems using the /3GB switch, that does not necessarily mean that using the /3GB switch is going to cause issues – what we said was that the /3GB switch is intended to be used in very specific instances.  Tuning the memory further by using the USERVA switch in conjunction with the /3GB switch can often stave off PTE depletion issues.  The problem with PTE depletion is that there are no entries logged in the Event Viewer that indicate that there is a resource issue.  This is where using Performance Monitor to determine whether a system is experiencing PTE depletion comes into play.  However, Performance Monitor may not identify why PTE’s are being depleted.  In instances where a process has a continually rising handle count that mirrors the rate of PTE depletion, it is fairly straightforward to identify the culprit.  However, more often than not we have to turn to a complete dump file to analyze the problem.

Below is what we might see in a dump file in a scenario where we have PTE depletion when we use the !vm command to get an overview of Virtual Memory Usage:

*** Virtual Memory Usage ***
    Physical Memory:   2072331   ( 8289324 Kb)
    Page File: \?? \C: \pagefile.sys
       Current:   2095104Kb Free Space:   2073360Kb
       Minimum:   2095104Kb Maximum:      4190208Kb
    Available Pages:   1635635   ( 6542540 Kb)
    ResAvail Pages:    1641633   ( 6566532 Kb)
    Locked IO Pages:      2840   (   11360 Kb)
    Free System PTEs:     1097   (    4388 Kb)

    ******* 1143093 system PTE allocations have failed ******
    Free NP PTEs:        14833   (   59332 Kb)
    Free Special NP:         0   (       0 Kb)
    Modified Pages:        328   (    1312 Kb)
    Modified PF Pages:     328   (    1312 Kb)
    NonPagedPool Usage:  11407   (   45628 Kb)
    NonPagedPool Max:    32767   (  131068 Kb)
    PagedPool 0 Usage:   11733   (   46932 Kb)
    PagedPool 1 Usage:     855   (    3420 Kb)
    PagedPool 2 Usage:     862   (    3448 Kb)
    PagedPool 3 Usage:     868   (    3472 Kb)
    PagedPool 4 Usage:     849   (    3396 Kb)
    PagedPool Usage:     15167   (   60668 Kb)
    PagedPool Maximum:   40960   (  163840 Kb)
    Shared Commit:        3128   (   12512 Kb)
    Special Pool:            0   (       0 Kb)
    Shared Process:      25976   (  103904 Kb)
    PagedPool Commit:    15197   (   60788 Kb)
    Driver Commit:        1427   (    5708 Kb)
    Committed pages:    432175   ( 1728700 Kb)
    Commit limit:      2562551   (10250204 Kb)

In this particular instance we can clearly see that we have a low PTE condition.  In looking at the Virtual Memory Usage summary, we can see that the server is most likely using the /3GB switch, since the NonPaged Pool Maximum is only 130MB.  In this scenario we would want to investigate using the USERVA switch to fine tune the memory and recover some more PTE’s,  If USERVA is already in place and set to 2800, then it is time to think about scaling the environment to spread the server load.  For more granular troubleshooting, where we suspect a PTE leak that we cannot explain using Performance Monitor data, we can modify the registry to enable us to track down the PTE leak.  The registry value that we need to add to the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management key is as follows:

Value Name: TrackPtes
Value Type: REG_DWORD
Value Data: 1
Radix: Hex

Once we implement this registry modification we need to reboot the system to enable the PTE Tracking.  Once PTE Tracking is in place, we would need to capture a new memory dump the next time the issue occurs and analyze that dump to identify the cause of the leak.

To wrap up our post, we are going to take a quick look at a dump file of a server that is experiencing a low physical memory condition.  Below is the output of the !vm command (with a couple of comments that we’ve added in)

3: kd> !vm

*** Virtual Memory Usage ***
Physical Memory: 851843 ( 3407372 Kb) <----- Server has 3.4 GB physical RAM
Page File: \??\C:\pagefile.sys
Current: 3072000Kb Free Space: 2377472Kb
Minimum: 3072000Kb Maximum: 3072000Kb
Page File: \??\D:\pagefile.sys
Current: 4193280Kb Free Space: 3502716Kb
Minimum: 4193280Kb Maximum: 4193280Kb
Page File: \??\E:\pagefile.sys
Current: 4193280Kb Free Space: 3506192Kb
Minimum: 4193280Kb Maximum: 4193280Kb
Page File: \??\F:\pagefile.sys
Current: 4193280Kb Free Space: 3454596Kb
Minimum: 4193280Kb Maximum: 4193280Kb
Page File: \??\G:\pagefile.sys
Current: 4193280Kb Free Space: 3459764Kb
Minimum: 4193280Kb Maximum: 4193280Kb

Available Pages: 1198 ( 4792 Kb) <-------- Almost no free physical memory

ResAvail Pages: 795226 ( 3180904 Kb)
Modified Pages: 787 ( 3148 Kb)
NonPagedPool Usage: 6211 ( 24844 Kb)
NonPagedPool Max: 37761 ( 151044 Kb)
PagedPool 0 Usage: 11824 ( 47296 Kb)
PagedPool 1 Usage: 895 ( 3580 Kb)
PagedPool 2 Usage: 881 ( 3524 Kb)
PagedPool 3 Usage: 916 ( 3664 Kb)
PagedPool 4 Usage: 886 ( 3544 Kb)
PagedPool Usage: 15402 ( 61608 Kb)
PagedPool Maximum: 65536 ( 262144 Kb)
Shared Commit: 771713 ( 3086852 Kb)
Special Pool: 0 ( 0 Kb)
Free System PTEs: 7214 ( 28856 Kb)
Shared Process: 7200 ( 28800 Kb)
PagedPool Commit: 15402 ( 61608 Kb)
Driver Commit: 1140 ( 4560 Kb)
Committed pages: 2161007 ( 8644028 Kb) <------ Total committed pages is 8.6GB.  This amount is far larger than physical RAM, paging will be high.
Commit limit: 5777995 (23111980 Kb)
 
 
 
Total Private: 1363369 ( 5453476 Kb)

In this particular instance, the server simply did not have enough memory to keep up with the demands of the processes and the OS.  Paged and NonPaged Pool resources are not experiencing any issues.  The number of available PTE’s is somewhat lower than our target of 10,000.  However, if you recall from our earlier posts, if a server is under load, the number of Free PTE’s may drop below 10,000 temporarily.  In this case, as a result of the low memory condition on this server there were several threads in a WAIT state – which caused the server to hang. The solution for this particular issue was to add more physical memory to the server to ease the low physical memory condition.

And with that, we come to the end of this post.  Hopefully you’ve found the information in our last few posts useful.

- Sakthi Ganesh

Share this post :

Today we are continuing on with our Two Minute Drill series.  Our topic in this post is one that we discuss quite frequently with customers - namely the automation of creating Performance Monitor and Trace Logs.  Most administrators are comfortable creating Local and Remote Performance Monitor logs using the Performance Monitor MMC and the GUI tools.  However, there are some extremely powerful command line utilities that can be used to configure and capture Performance data.  Today we will be discussing the LOGMAN.ExE utility.  So without further ado ...

The LOGMAN.EXE utility can be used to create and manage Event Trace Session and Performance logs.  Many functions of Performance Monitor are supported and can be invoked using this command line utility.  Before we look at some examples of how to configure Performance logs using this utility, let's quickly cover some of the syntax.  Running LOGMAN /? from a command prompt brings up the first level of context sensitive help:

Basic Usage:  LOGMAN [create | query | start | stop | delete | update | import | export] [options].  The verbs specified determine what actions are being performed:

Running LOGMAN <verb> /? brings up context sensitive help for the verb specified.  There are also some options to be aware of:

So now that we have our basic commands, let's take a look at how we can use LOGMAN.EXE for one of our most common scenarios - capturing baseline Performance data for a system.  We've discussed the importance of capturing baseline server performance data in several previous posts.  In our example, we are going to capture a binary circular performance monitor log that has a maximum size of 500MB.  The reason we are going to use a binary circular log is that we can record the data continuously to the same log file, overwriting previous records with new data once the log file reaches its maximum size.  Since this will be a baseline performance log that will be constantly running, we want to ensure that we can capture a significant data sample, and not have the log file being overwritten in such a short timeframe that useful data is lost.  Put another way, we want to set our capture interval up so that we do not overwrite our data too quickly.  For the purposes of this example, we'll set up our log to capture data every two hours.  We want to save our data to a log file, so we will need to specify a log file location.  Given that we want to capture baseline data, there is a good possibility that we want to use the same settings on multiple servers so we'll need to ensure that we can repeat this process with a minimum of administrative fuss ...

So, to recap, we are going to capture our baseline performance log that is:

• a binary circular log that will be a maximum of 500MB in size
• configured with a capture interval of two hours
• saved to a file location
• configured with standard counters so that we can capture consistent baseline data across multiple servers if needed

The one piece of this equation that we have not specified is which counters we need to capture.  One of the key reasons to use LOGMAN.EXE is that we can specify which counters we want to capture in a standard configuration file and then use that configuration across to configure our capture for multiple servers.  Creating the configuration file is fairly simple - we are going to create a .CONFIG file that enumerates the counters that we want to capture, one per line.  An example is shown below:

"\Memory\Available MBytes"
"\Memory\Pool Nonpaged Bytes"
"\Memory\Pool Paged Bytes"
"\PhysicalDisk(*)\Current Disk Queue Length"
"\PhysicalDisk(*)\Disk Reads/sec"
"\PhysicalDisk(*)\Disk Read Bytes/sec"
"\PhysicalDisk(*)\Disk Writes/sec"
"\PhysicalDisk(*)\Disk Write Bytes/sec"
"\Process(*)\% Processor Time"
"\Process(*)\Private Bytes"
"\Process(*)\Virtual Bytes"

These are some fairly standard Performance Counters.  Let's save this file as Baseline.config on a folder on one of our file servers.  Now we have all of the pieces that we need to configure and capture our baseline.

logman create counter BASELINE -f bincirc -max 500 -si 2 --v -o "e:\perflogs\SERVERBASELINE" –cf "\\<FILESERVER>\Baseline\Baseline.config"

• logman create counter BASELINE: This creates the BASELINE Data Collector on the local machine
• -f bincirc -max 500 -si 2: This piece of the command specifies that we are creating a Binary Circular file, sets the Maximum Log file size to 500MB, sets the Capture Interval at 2 hours
• --v -o "e:\perflogs\SERVERBASELINE": In this part of the command, we turn off the versioning information, and set the Output Location and Filename.  The Performance Monitor log will be created with a .BLG extension
• –cf \\<FILESERVER>\Baseline\Baseline.config: Finally, we point the LOGMAN utility to the location of our standard counter configuration file

Once we run this command, we can run LOGMAN.EXE and use the QUERY verb to ensure that our Data Collector has been created:

The last thing we need to do is start our Data Collector set.  There are a couple of options here - the first is to run LOGMAN.EXE START BASELINE from the command line.  This will launch the Data Collector.  However, when we reboot our system, the Data Collector will not run.  If you create a startup script to run the command above to start the Data Collector set, then you can capture your performance data from the time that the server starts.

Before we wrap up our post, here is another common scenario.  You can create a Data Collector set on a full installation of Windows Server 2008 or Windows Vista.  Then export that Data Collector Set configuration to an XML Template.  You can then use the LOGMAN.EXE command with the IMPORT verb to import that Data Collector set configuration on a Windows Server 2008 Server Core system, then use the LOGMAN.EXE command with the START verb to start the Data Collector Set.  The commands are below:

• LOGMAN IMPORT -n <Data Collector Set Name> -xml <XML template that you exported>:  This will create the Data Collector Set named whatever name you choose when passing the -n parameter
• LOGMAN START <Data Collector Set Name>: Start the Data Collectiion process.

Finally, here are two more sample commands where we use LOGMAN.EXE for gathering Performance Monitor data for troubleshooting:

High CPU Issue

logman.exe create counter High-CPU-Perf-Log -f bincirc -v mmddhhmm -max 250 -c "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*" "\Thread(*)\*"   -si 00:00:05

In this example, we have a capture interval of five seconds, with a Maximum Log size of 250MB.  The Performance Counters that we are capturing are fairly generic.

Generic Performance Monitor Logging

logman.exe create counter Perf-Counter-Log -f bincirc -v mmddhhmm -max 250 -c "\LogicalDisk(*)\*" "\Memory\*" "\Network Interface(*)\*" "\Paging File(*)\*" "\PhysicalDisk(*)\*" "\Process(*)\*" "\Redirector\*" "\Server\*" "\System\*"  -si 00:05:00

In this example, we are using a five minute capture interval - the rest of the parameters are fairly straightforward.  Remember that in both of these cases, you will need to use LOGMAN.EXE with the START verb and specifying the name of the Data Collector Set to begin the capture.  These samples work on all Windows Operating Systems from Windows XP onwards.

And with that, we come to the end of this Two Minute drill.  Until next time ...

- CC Hameed

Share this post :

In our last post on Server Hangs, we discussed using the Debugging Tools to examine a dump file to analyze pool depletion.  Today we are going to look at using our troubleshooting tools to examine a server hang caused by a handle leak.  Issues where there are an abnormal number of handles for a process are very common and result in kernel memory depletion.  A quick way to find the number of handles for each process by checking the Task Manager > Processes.  You may have to add the handles column from View > Select columns.  Generally if a process has more than 10,000 then we probably want to take a look at what is going on.  That does not necessarily mean that it is the offending process, just a suspect.  However, there are instances where the process may be for a database or some other memory intensive application.  The most common instance of this is the STORE.EXE process for Exchange Server which routinely has well over 10,000 handles.  On the other hand if our Print Spooler process has 10,000 (or more) handles then we most likely have an issue.

Once we know there is a handle leak in a particular process, we can dump out all the handles and figure out why it is leaking.  If we want to find out from a dump if there is a process that has an abnormally large number of handles, we first have to list out all the processes and then examine the number of handles being used by the processes.  To list out all the processes that are running on the box using the Debugging Tools, we use the !process 0 0 command.  This will give us an output similar to what we see below:

0: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 8a5295f0  SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: 0acc0020  ObjectTable: e1002e68  HandleCount: 1056.
    Image: System

PROCESS 897e6c00  SessionId: none  Cid: 04fc    Peb: 7ffd4000  ParentCid: 0004
    DirBase: 0acc0040  ObjectTable: e1648628  HandleCount:  21.
    Image: smss.exe

PROCESS 89a26da0  SessionId: 0  Cid: 052c    Peb: 7ffdf000  ParentCid: 04fc
    DirBase: 0acc0060  ObjectTable: e37a7f68  HandleCount: 691.
    Image: csrss.exe

PROCESS 890f0da0  SessionId: 0  Cid: 0548    Peb: 7ffde000  ParentCid: 04fc
    DirBase: 0acc0080  ObjectTable: e1551138  HandleCount: 986.
    Image: winlogon.exe

PROCESS 89a345a0  SessionId: 0  Cid: 0574    Peb: 7ffd9000  ParentCid: 0548
    DirBase: 0acc00a0  ObjectTable: e11d8258  HandleCount: 396.
    Image: services.exe

The important piece of information here is the HandleCount.  For the purposes of this post, let’s assume that there is a problem with SMSS.EXE and that there is an unusually high HandleCount.  To view all of the handles for the process, the first thing we need to do is switch to the context of the process and then dump out all of the handles as shown below.  The relevant commands are:

• .process –p –r <processaddress> – this switches us to the context of the process
• !handle – this dumps out all of the handles

0: kd> .process –p –r 897e6c00                          
Implicit process is now 897e6c00
0: kd> !handle
processor number 0, process 897e6c00
PROCESS 897e6c00  SessionId: none  Cid: 04fc    Peb: 7ffd4000  ParentCid: 0004
    DirBase: 0acc0040  ObjectTable: e1648628  HandleCount:  21.
    Image: smss.exe

Handle table at e1674000 with 21 Entries in use
0004: Object: e1009568  GrantedAccess: 000f0003 Entry: e1674008
Object: e1009568  Type: (8a5258b8) KeyedEvent
    ObjectHeader: e1009550 (old version)
        HandleCount: 53  PointerCount: 54
        Directory Object: e10030a8  Name: CritSecOutOfMemoryEvent

0008: Object: 8910b370  GrantedAccess: 00100020 (Inherit) Entry: e1674010
Object: 8910b370  Type: (8a54c730) File
    ObjectHeader: 8910b358 (old version)
        HandleCount: 1  PointerCount: 1
        Directory Object: 00000000  Name: \WINDOWS {HarddiskVolume1}

000c: Object: e1af9828  GrantedAccess: 001f0001 Entry: e1674018
Object: e1af9828  Type: (8a512ae0) Port
    ObjectHeader: e1af9810 (old version)
        HandleCount: 1  PointerCount: 12
        Directory Object: e1002388  Name: SmApiPort

At this point we can continue to dig into the handles to determine if there is something amiss.  More often than not, this would be an issue for which systems administrators would be contacting Microsoft Support.  However, by using this method you can quickly determine whether the problem lies with a third-party component and engage that vendor directly.  Being able to provide them with a dump file that shows that their component is consuming an excessive number of handles can assist them in providing you with a quicker resolution.

That’s it for today.  In our next post on Server Hangs, we’ll look at how a lack of Available System PTE’s can cause server hangs.

- Sakthi Ganesh

Share this post :

Several months ago, we wrote a post on Troubleshooting Server Hangs.  At the end of that post, we provided some basic steps to follow with respect to server hangs.  The last step in the list was following the steps in KB Article 244139 to prepare the system to capture a complete memory dump for analysis.  Now that you have the memory dump, what exactly are you supposed to do with it?  That will be the topic of today’s post – more specifically, dealing with server hangs due to resource depletion.  We discussed various aspects of resource depletion including Paged and NonPaged pool depletion and System PTE’s.  Today we’re going to look at Pool Resource depletion, and how to use the Debugging Tools to troubleshoot the issue.

If the server is experiencing Non paged pool (NPP) memory leak or a Paged pool (PP) memory leak you are most likely to see the following event id’s respectively in the System Event log:

Type: Error 
Date: <date> 
Time: <time> 
Event ID: 2019
Source: Srv 
User: N/A 
Computer: <ComputerName> 
Details: The server was unable to allocate from the system nonpaged pool because the pool was empty. 

Type: Error 
Date: <date> 
Time: <time> 
Event ID: 2020 
Source: Srv 
User: N/A 
Computer: <ComputerName> 
Details: The server was unable to allocate from the system Paged pool because the pool was empty

Let’s load up our memory dump file in the Windows Debugging tool (WINDBG.EXE).  If you have never set up the Debugging Tools and configured the symbols, you can find instructions on the Debugging Tools for Windows Overview page.  Once we have our dump file loaded type !vm in the prompt to display the Virtual Memory Usage for the system.  The output will be similar to what is below:

kd> !vm
*** Virtual Memory Usage ***
                Physical Memory:      917085 (   3668340 Kb)
                Page File: \??\C:\pagefile.sys
                Current:   4193280 Kb  Free Space:   4174504 Kb
                Minimum:   4193280 Kb  Maximum:      4193280 Kb
                Page File: \??\D:\pagefile.sys
                Current:   4193280 Kb  Free Space:   4168192 Kb
                Minimum:   4193280 Kb  Maximum:      4193280 Kb
                Available Pages:      777529 (   3110116 Kb)
                ResAvail Pages:       864727 (   3458908 Kb)
                Locked IO Pages:         237 (       948 Kb)
                Free System PTEs:      17450 (     69800 Kb)
                Free NP PTEs:            952 (      3808 Kb)
                Free Special NP:           0 (         0 Kb)
                Modified Pages:           90 (       360 Kb)
                Modified PF Pages:        81 (       324 Kb)
                NonPagedPool Usage:    30294 (    121176 Kb)
                NonPagedPool Max:      32640 (    130560 Kb)
                ********** Excessive NonPaged Pool Usage *****
                PagedPool 0 Usage:      4960 (     19840 Kb)
                PagedPool 1 Usage:       642 (      2568 Kb)
                PagedPool 2 Usage:       646 (      2584 Kb)
                PagedPool 3 Usage:       648 (      2592 Kb)
                PagedPool 4 Usage:       653 (      2612 Kb)
                PagedPool Usage:        7549 (     30196 Kb)
                PagedPool Maximum:     62464 (    249856 Kb)
                Shared Commit:          3140 (     12560 Kb)
                Special Pool:              0 (         0 Kb)
                Shared Process:         5468 (     21872 Kb)
                PagedPool Commit:       7551 (     30204 Kb)
                Driver Commit:          1766 (      7064 Kb)
                Committed pages:      124039 (    496156 Kb)
                Commit limit:        2978421 (  11913684 Kb)

As you can see, this command provides details about the usage of Paged and NonPaged Pool Memory, Free System PTE’s and Available Physical Memory.  As we can see from the output above, this system is suffering from excessive NonPaged Pool usage.  There is a maximum of 128MB of NonPaged Pool available and 121MB of this NonPaged Pool is in use:

NonPagedPool Usage:    30294 (    121176 Kb)
NonPagedPool Max:      32640 (    130560 Kb)

Our next step is to determine what is consuming the NonPaged Pool.  Within the debugger, there is a very useful command called !poolused.  We use this command to find the Pool Tag that is consuming our NonPaged Pool.  The !poolused 2 command will list out NonPaged Pool consumption, and !poolused 4 lists the Paged Pool consumption.  A quick note here; the output from the !poolused commands could be very lengthy as they will list all of the tags in use.  To limit the display to the Top 10 consumers, we can use the /t10 switch:  !poolused /t10 2.

0: kd> !poolused 2
   Sorting by  NonPaged Pool Consumed
  Pool Used:
            NonPaged            Paged
 Tag    Allocs     Used    Allocs     Used
 R100        3  9437184        15   695744    UNKNOWN pooltag 'R100', please update pooltag.txt
 MmCm       34  3068448         0        0    Calls made to MmAllocateContiguousMemory , Binary: nt!mm
 LSwi        1  2584576         0        0    initial work context 
 TCPt       28  1456464         0        0    TCP/IP network protocol , Binary: TCP
 File     7990  1222608         0        0    File objects 
 Pool        3  1134592         0        0    Pool tables, etc. 
 Thre     1460   911040         0        0    Thread objects , Binary: nt!ps
 Devi      337   656352         0        0    Device objects 
 Even    12505   606096         0        0    Event objects 
 naFF      300   511720         0        0    UNKNOWN pooltag 'naFF', please update pooltag.txt

Once the tag is identified we can use the steps that we outlined in our previous post, An Introduction to Pool Tags to identify which driver is using that tag.  If the driver is out of date, then we can update it.  However, there may be some instances where we have the latest version of the driver, and we will need to engage the software vendor directly for additional assistance.

That brings us to the end on this post – in Part Three, we will discuss using Task Manager and the Debugging Tools to troubleshoot Handle Leaks which may be causing Server Hangs.

- Sakthi Ganesh

Share this post :

Something many of our readers have asked for is a quick overview of new and useful tools and features in Windows.  Today we are launching a new series, called our “Two Minute Drill” – these will be brief posts that provide a high-level view of tools and features.  We have already published several posts in this vein over the last year or so, but we are going to go back and add the “Two Minute Drill” tag to these posts to make them easier to find.  Hopefully you enjoy these posts, and keep providing feedback on other topics that you would like to see.  Enough with the Administrative, on with the Substantive …

Windows Vista and Windows Server 2008 introduced several new features to assist users and administrators in diagnosing performance-related issues.  Today, we are going to take a look at the System Diagnostics Report which provides a snapshot view of your system.  The report itself gathers data for 60 seconds, and then may take an additional 60-90 seconds to generate the report.  To generate a System Diagnostics Report, run the perfmon /report command from the Run line or from a command prompt.  Generating a System Diagnostics Report requires privilege elevation, so you will need to consent to the elevation prompt or provide credentials depending on how UAC is configured on your systems.  Below is the initial System Diagnostics window – the Report Status area shows the progress in generating the report:

Once the report has been generated, you will see a Report similar to the one below:

As you can see, the report is organized into different areas.  The first area to check is the Warnings section under the Diagnostic Results heading.  In this area of the report, different warning and informational events will be displayed.  In this particular instance, I have UAC turned off on my system, and the Security Center has generated an informational event.  You may also see information generated by the Windows System Assessment Tool on Windows Vista systems as shown below:

Device and Service failures are also displayed – for example, if a device is not working properly, or if a service has failed.  Additionally, if there are issues with the Event Logs themselves, you would see Warning messages, such as the one below:

As you can see there are hyperlinks that you can use to get additional information about resolving issues.  You can also expand each of the sections under the Basic System Checks section to get more details.

One of the most useful sections on this report is the Performance section.  This area provides process-level detail about the top consumers of resources for CPU, Disk, Network and Memory.  If there are any performance related issues, the status indicator would change, and you would need to expand out the relevant section in the  report for the component that reported a problem.

System Diagnostics Reports can be saved out in HTML format so that you can maintain an archive, post them to an intranet library, such as a Sharepoint portal, or email them to colleagues if needed.  If users have the rights to run these reports, they can provide them to their Help Desk, or IT Administrators for analysis and assistance.

Finally, you can view and create System Diagnostic Reports within the main Windows Reliability and Performance Monitor utility:

• Launch Windows Reliability and Performance Monitor (remember, that you will need elevated privileges)
• In the navigation tree, expand Data Collector Sets and expand System
• Right-click the System Diagnostics item and click Start.  This will begin the Data Collection Process.  Once the report has been generated, the System Diagnostics Report will appear in the main pane of the console.

You can also view historical reports from within Windows Reliability and Performance Monitor.  In the navigation tree, expand Reports, expand System, and then expand System Diagnostics.  Select the date for the report you wish to view.

That brings us to the end of this quick look at System Diagnostics Reports.  Now you have one more utility at your disposal to get system information for troubleshooting and reporting!

- Blake Morrison

Share this post :

Last week, we announced the release of Windows XP Service Pack 3 (SP3) to manufacturing (RTM).  Windows XP SP3 includes previously released updates for the OS and also a small number of new updates.  The current timeline for XP SP3 availability is as follows:  Release to the Web (RTW) on April 29, 2008.  Once released to the web, the final service pack will be available for download via Windows Update and the Microsoft Download Center.  Automatic Update Distribution for users at home is scheduled for the early part of this summer

Obviously the release of a Service Pack is a significant milestone.  There are some enhancements to Windows XP SP3 such as Network Access Protection (NAP) that will make it easier for enterprises to co-manage Windows XP and Windows Vista systems during their phased rollouts of Windows Vista.  A couple of things to note when installing XP SP3 – if you installed on of the XP SP3 Release Candidate (RC) builds on your XP system, you will need to uninstall the RC build before installing the final version of the service pack. 

Below is a table that outlines previously released functionality available to Windows XP in stand-alone updates.  These features are included in XP SP3 by default.

Finally – there are two important things to note:

• Windows XP SP3 does not include Internet Explorer 7.
• Windows XP SP3 is for x86 (32-bit) versions of XP only.  Windows XP Professional x64 editions are serviced by Windows Server 2003 SP2.  For more details see the information on Windows Server 2003 SP2

That’s it for today. 

Additional Resources:

• Windows XP Service Pack 3 Overview
• Microsoft KB Article 936929 – Release Notes for Windows XP SP3

- CC Hameed

UPDATE – 4/30:  As many of you have seen on various news web sites, the release of Windows XP SP3 to the Web has been delayed.  This was due to an compatibility issue that was discovered between Microsoft Dynamics Retail Management System (RMS) and Windows XP SP3, and Windows Vista SP1.

Share this post :

Over the last couple of weeks, we have worked with a few customers on some MMC issues that they have been experiencing when launching MMC Snap-ins on 64-bit Windows systems.  Most customers have reported that the issue normally surfaces after installing the Hotfix from KB942589 on Windows Server 2003 systems.  On the systems in question, there will be several SideBySide errors in the Event Logs, similar to the ones below:

Event Type: Error 
Event Source: SideBySide 
Event Category: None 
Event ID: 59 
Date: (Date) 
Time: (Time)
User: N/A 
Computer: (Computer Name)
Description: 
Generate Activation Context failed for C:\WINNT\SysWOW64\mmc.exe. Reference error 
message: The referenced assembly is not installed on your system. 

Event Type: Error 
Event Source: SideBySide 
Event Category: None 
Event ID: 59 
Date: (Date)
Time: (Time)
User: N/A 
Computer: (Computer Name)
Description: 
Resolve Partial Assembly failed for Microsoft.Windows.Common-Controls. Reference 
error message: The referenced assembly is not installed on your system. 

Event Type: Error 
Event Source: SideBySide 
Event Category: None 
Event ID: 32 
Date: (Date)
Time: (Time)
User: N/A 
Computer: (Computer Name)
Description: 
Dependent Assembly Microsoft.Windows.Common-Controls could not be found and Last 
Error was The referenced assembly is not installed on your system. 

As you can see, the error messages are not terribly descriptive and don’t really give any indication as to possible causes.  The initial assumption is that there is some issue with SideBySide on these systems (we’ll be covering SideBySide in more detail in a future post).  However, that is actually not the case.

If you use Process Monitor to monitor the startup of the MMC.EXE on a working and a non-working system, what you would actually find is that prior to the installation of the hotfix, MMC.EXE was being launched from the Windows\System32 folder.  However, after installing the hotfix, MMC.EXE was being launched from the Windows\SysWOW64 folder.  The MMC.EXE does not launch because it is attempting to use the 64-bit SideBySide files.  So … what happened?

Microsoft 64-bit operating systems can run both 32- and 64-bit MMC’s (MMC32 and MMC64).  However, a 64-bit MMC cannot run a 32-bit snap-in, nor can a 32-bit MMC run a 64-bit snap-in.  However, it is possible to control which version of the MMC is executed by using command-line parameters, or switches.  Stepping away from Windows Server 2003, and looking at 64-bit operating systems in general, let’s take a look at how this would work using the Computer Management snap-in.  Most administrators are familiar with right-clicking on the Computer icon and selecting the “Manage” option from the context menu.  When you launch the Computer Management snap-in in this fashion, the 64-bit version of MMC is launched.  However, to launch the 32-bit version, you could either run MMC.EXE –32 from the Run line, or run COMPMGMT.MSC –32 from the Run line.  You can confirm which version is running by looking in Task Manager:

Right-click on the Computer Icon and choose Manage	Run COMPMGMT.MSC from the Run Line
Run COMPMGMT.MSC –32 from the Run Line

So as you can see by looking in Task Manager, the 32-bit version of MMC is distinguished by the *32 suffix, denoting the application as 32-bit.  Thus there are ways to manipulate the version of MMC.EXE used.  When specifying a console file there are some considerations.  If MMC64 is started from the command line without the –32 or –64 switch, but the command line does contain a console file name, then MMC64 examines the console file before determining which version of MMC remains in execution using the following logic:

1. If all snap-ins referenced by the console file are available in 64-bit form, then MMC64 remains as the executing version and the rest of the logic below is ignored
2. If not all of the snap-ins are available in 64-bit form, but they are all available in 32-bit form then MMC64 starts MMC32, MMC32 executes the console file and MMC64 terminates, ignoring the rest of the logic below.
3. If the number of unavailable 64-bit snap-ins is a subset of the unavailable 32-bit snap-ins then MMC64 remains as the executing version and ignores the rest of the logic below
4. If the number of unavailable 32-bit snap-ins is a subset of the unavailable 64-bit snap-ins, then MMC64 starts MMC32 and MMC64 terminates, and ignores the last logical step below
5. If none of the previous logic steps yields a conclusion, then the user is presented with a prompt to either choose which version of MMC to execute, or to cancel the operation.

And that will do it for this post.  Hopefully this helps answer some of your questions.

Additional Resources:

• MSDN: Running 32-bit and 64-bit Snap-ins in 64-bit Windows

- Blake Morrison

Share this post :

Prior to Windows Vista, if you wanted to install optional components, Windows Updates or driver files on your system, the process was fairly simple.  In Windows Vista, the new componentization architecture, known as Component-Based Servicing (CBS) changes the way that these components are installed.  The CBS architecture is far more robust and secure than the installers in previous operating systems.  Users benefit from a more complete and controlled installation process that allows updates, drivers and optional components to be added while simultaneously mitigating against instability issues caused by improper or partial installation.  CBS allows components and features from IIS to Windows Media Player to be packaged as small modules that encompass the full functionality of the component.  In other words, each module contains all of the files, registry settings, and methods required for a full installation or removal of the component it contains.  The Core componentization services include the following:

• CBS (Component Based Servicing) - Also known as the trusted installer (TRUSTEDINSTALLER.EXE), which works at the package / update level
• CSI (Component Servicing Infrastructure) - Works at the deployment/component level
• DMI (Driver Management and Install) – Advanced driver installation processes
• CMI (Component Management Infrastructure) - Handles the advanced installers
• SMI (Systems Management Infrastructure) – Used to manage registry settings
• Kernel Transaction Manager (KTM) – Enables clients to use the transactional registry and file system

The Servicing Stack in Windows Vista consists of three levels:

• At the top of the stack are the top level clients, such as Windows Update, Programs and Features, and MSI, which deliver packages to a system.  The top-level clients are also responsible for control of user input and collection of user preferences during the servicing process.
• In the middle of the servicing stack is the Trusted Installer, CBS.  The top-level clients pass downloaded packages to CBS, which evaluates each individually to determine if they are applicable to the system.  For applicable updates, CBS provides the components to CSI, generates appropriate installation events, and registers packages with Programs and Features if needed.  Finally, CBS exposes the interfaces to enumerate and inventory the updates.
• At the bottom of the servicing stack is CSI, which uses the Kernel Transaction Manager (KTM) to do its work.

CSI is responsible for the actual installation of components.  To install components, CSI utilizes the Component Store (the %windir%\WinSXS folder) which is a collection of all components, manifests (%windir%\WinSXS\Manifests) and files on the system.  As new components are added, CSI moves them into the Component Store, and determines what state the component should enter.  Staging determines the current state of the package on the file system.  There are different staging activities that occur during the installation of a package:

• Identify any files that are missing from the package.  For a file to be installed, it must first be staged.  Some may already present in the system store, others may need to be transferred from installation media or downloaded from network locations
• Determine which files are required to install a package and identify files in other packages that may also be required
• Resolve dependencies and ensure that all required files are present before installation begins
• Complete installation

When a package is uninstalled, the process is reversed:

• The installer creates a list of any files in use, and other actions that require system reboot
• Remove the files or dependencies – files may either be removed from the system completely or may remain in the system store for future use.
• Files not in use may be removed from the system, and the system is rebooted to release and remove any files that were in use

When a newer version of a component is installed on the system, CSI queries the Component Store to determine what components are being updated.  When installing a component, CSI sets up a Primitive Operations Queue (POQ) which contains all files and registry keys that will be installed.  Advanced installers or generic commands are then executed to complete installation.  Advanced installers run in-process using CMI.

If there is a failure during installation of a component, CSI rolls back the entire installation.  If a file or process is in use during a component installation and cannot be replaced, generic commands and advanced installer actions are written to %windir%\WinSXS\Pending.xml, and then written to disk on the following reboot.  If several packages are installed at the same time, each additional package appends to Pending.xml.  Additional logging during this phase occurs in %windir%\Logs\CBS\CBS.log.

And that brings us to the end of our post on Component-Based Servicing.  Until next time ...

- CC Hameed

Share this post :

EDIT

• 4/23: Several Updates and Changes made to the article.  Article republished to ensure that RSS feeds have the updated version.

We’ve covered different pieces of WMI in previous posts, but we have yet to discuss the Windows Management Instrumentation Command-Line (WMIC) in detail.  The WMIC provides administrators easy access to the WMI infrastructure.  Prior to the WMIC, you could only access the management infrastructure via the providers or an API.  Since the ultimate goal of WMI is to make life easier for system and application administrators, this obviously presented a problem.

The WMIC environment allows interactive queries or scripting.  It is interoperable with existing shell and utility commands and can be extended by scripts and other administration-oriented applications.  WMIC is included with Windows XP and later operating systems.  However, since WMIC works locally and remotely, it is possible to run WMIC commands against Windows 2000 systems remotely so long as the particular action is supported by WMI on the target machine.  Beyond the ability to use WMIC against local and remote systems, there are other significant benefits:

• WMIC leverages existing OS WMI providers and can leverage additional providers that are added to the system
• WMIC can generate output in multiple formats such as .TXT, .CSV, .HTML and .XML
• WMIC has very powerful functionality natively available, however it is an extensible tool

So that should be enough of the marketing fluff around WMIC.  How exactly do you use it?  There are two modes of use for WMIC – Interactive or Scripting.  In Interactive mode, WMIC provides and ‘environment’ for scripting.  The environment allows you to enter commands and view the results in the immediate display – just like the standard command line interface you are accustomed to.  The Interactive environment also allows the use of the discoverable help which is context sensitive.  In the screenshots below, you can see the WMIC Interactive environment, which is launched by running WMIC from a command prompt, followed by examples of the help features:

So now let’s look at WMIC in action in Interactive mode.  In these examples, I am going to get information about two systems – the local and the remote.  There is  a command set to retrieve Operating System information as shown below:

So I know that I can probably use the OS GET command, but what data can I retrieve?  Let’s look at the context sensitive help:

That’s quite an exhaustive list.  I think I want to get the following information about my systems – Name of the system, what OS is installed and the build number:

By the way, those are not misspellings in the output – the symbols for ® and ™ don’t really translate into command line output.  So, this format is fairly readable, but if we were to add more properties, the output could become very confusing.  Enter the /format switch.  In this case, rather than the default table format, I want to see this information in list format:

Pretty easy, right?  So now let’s run the same command for a remote machine:

As you can see, we specify the name of the remote system with the /NODE switch.  If the value being passed has special characters such as a hyphen or a slash, you need to enclose the switch value in double quotes.  Also remember that you will need to create an exception on your system to allow WMI to be accessed remotely in the Windows firewall, or whatever local firewall you may be using.

So those are some basic information gathering pieces – but what about if we want to actually do something on the system – for example, launch an executable, such as the Calculator?  There is a command set called PROCESS that can be used to do just that:

And there we have our Calculator application launched, including the Process ID for CALC.EXE

As you can see, WMIC is very powerful – and that’s in Interactive mode where we have to enter one command at a time.  WMIC commands can also be run from scripts – let’s take a look at a fairly simple example to capture the same information that I captured in Interactive mode about my secondary system – BANGANALYZE-380.  I created a simple batch file called WMIC_Test.bat with a single command, as shown below:

wmic /node:"BANGANALYZE-380" /output:"c:\users\cchameed\documents\file1.htm" os get csname, name,buildnumber /format:hform

As you can see, I am redirecting the output to a .htm file and using a slightly different format.  The output of the file when viewed in a web browser, looks like this:

In this post, we have barely scratched the surface of what you can do with WMIC.  The resources below should help to get you on your way to truly leveraging the power of WMIC.  Until next time …

Additional Resources:

• MSDN: wmic (Windows)
• Microsoft KB290216: A description of the WMI Command-Line Utility (WMIC.EXE)
• Support WebCast: WMIC: A New Approach to Managing Windows
• MSDN: WMI Reference (Windows)

- CC Hameed

Share this post :

OK, a quick disclaimer right at the start.  The steps we discuss in this post assume that you are very comfortable with kernel debugging, and in particular live debugging.  This is a bit of a niche post, but it is a question that we get every so often, so we figured we’d share the information.  The information in this post follows on from our post last Friday on Pool Tags.  So without further ado, here we go …

Let’s say that you have a server that is leaking Pool memory (either Paged or NonPaged).  You’ve done troubleshooting and identified the leaking tag, but you want to examine the actual stack leading up to the allocation.  Using Special Pool and the Debugger Tools you can find out exactly who is allocating the tag.  Now, we’re not going to dive too deep into Special Pool itself, but in a nutshell, this feature configures Windows to request memory allocations from a reserved memory pool when the memory is allocated with a specific pool tag or is within a specified size range.

So let’s take a look at an example.  We’re going to use the Ddk tag for our example.  The Ddk tag is the default tag for driver-allocated memory.

• On the system where the tag is leaking, we navigate to the HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management key
• Create a REG_DWORD value called PoolTag.  In this value, we enter the tag value in little-endian format (i.e. backwards).  Remember that Pool Tags are four bytes(characters), so the tag is actually Ddk<space>.  When we convert this string into ASCII (and then into Hex) we get the following:

Character	ASCII Value	HEX Value
D	68	44
d	100	64
k	107	6b
<space>	32	20

Note:  When doing the conversion, remember that ASCII differentiates between upper and lower case, so you have to ensure that you take case sensitivity into account.

Therefore, according to our conversion table above, the basic conversion becomes 44646b20.  When we enter this into the registry, it has to be In little-endian format. Thus the value for the REG_DWORD Pooltag entry for the Ddk tag is: 206b6444

• Now that we have our registry value configured, it’s time to set up for the debug.  You’ll need a second machine (such as a laptop) and a null modem cable.  On the laptop, ensure that you have the latest version of the Debugging Tools for Windows.  A quick tip here – when installing the Debugging Tools, install them to a different location than the default, such as C:\Debug or C:\Debuggers.
• On the system with the leaking tag, edit the boot.ini file so that you can attach to the debugger.  Copy your default entry (give it a different display name) and add the following switches: /debug /debugport=com1 /baudrate=19200
• Create a batch file on the laptop, and save it with a .cmd extension – a sample is below:

@echo off
cd\debug
SET _NT_DEBUG_PORT=COM1
SET _NT_DEBUG_BAUD_RATE=19200
SET _NT_DEBUG_LOG_FILE_OPEN=c:\debug\debug.log
SET _NT_SYMBOL_PATH=SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols 
kd -server npipe:pipe=livedebug
cd\debug

When you run this batch file, you will see a message, “Waiting to Connect” – the laptop is waiting for the server to allow it to break in.  Reboot the server, and select your Debug option that you configured in the boot.ini earlier.  As the server boots up, you will start to see debug output in the command window on the laptop.  Log into the server as soon as possible.  Once you are logged in to the server, you’ll need to break into it from the laptop by pressing the CTRL+BREAK keys.  When the >kd prompt appears (it may take a few moments, and during this time your server will be halted), we’re ready to set our breakpoint on the nt!MmAllocateSpecialPool function.  The command is: bp nt!MmAllocateSpecialPool “;kb;g” and hit the ENTER key.  With our breakpoint set, it’s time to let the server get back to work.  In the command window, type in the letter ‘g’ (without the quotes) and hit ENTER.  Now all we have to do is wait for an allocation that matches the tag we entered to be allocated.  So in our example, every time the Ddk tag is allocated, the allocation stack will be written to the debug log (c:\debug\debug.log) that we specified above in our batch file which we can examine offline without having to perform a full dump of the server.

And that’s it.  Remember, that these are some very advanced troubleshooting steps that we are getting into – for most people, being able to identify the leaking tag is 99% of the challenge.

Additional Resources:

• Microsoft KB188831: How to use the Special Pool Feature
• MSDN: Special Pool (Driver Development Kit)
• MSDN: Special Pool (Debugging Tools for Windows)

- Aaron Maxwell

Share this post :

We’ve discussed