Brad Rutkowski's Blog : Cool Tools

Interacting with Data Collector Sets via Powershell

Brad Rutkowski — Thu, 19 Feb 2009 00:30:12 GMT

Background:

In an earlier post I talked about some new features for Windows 2008 and Vista. One of those new features that is often overlooked are the data collector sets (DCS). One particular role that leverages data collector sets is active directory. Active directory has put “hooks” into tracing that can really take a lot of the thinking out of the question “why is my domain controller sluggish”. For those of you still running Windows 2003 I go over a similar concept called Server Performance Advisor.

Anyways, you can play around with DCS by typing perfmon and then traversing to the section called Data Collector Sets (shocking). If you have performance issues, go here first as it’s like combining a netmon capture with a kernel trace and then handing you the smoking gun.

Challenge:

In my current role, we have a need to automate things quite a bit and so one of the actions I was looking at solving was collecting diagnostic information when a server is performing poorly. Usually when a high CPU alert comes in, someone would need to logon to the server and go to perfmon and start at DCS collection. More often is the case that by the time someone had been alerted and went to the server the sluggish behavior had subsided (the dreaded “close ticket, no problem found”).

My solution was to try and figure out a way to start a DCS collection remotely at the time of event so that the data was present when an actual human became engaged.

After some hard work, here is the code to do so! You can create your own XML file (your own DCS template) and pass it in, but more than likely you’ll be happy at just kicking off one of the built-in templates (AD/System Perf/System Diags).

Running it via powershell:

First, how to do it on the fly:

## PLA.dll lives under system32 on Vista and 2k8.  This will create a powershell com object.
$datacollectorset = new-object -COM Pla.DataCollectorSet
##This is the name of the predefined DCS collector.  It's read-only and will always be System\<something>
$name = "System\Active Directory Diagnostics"
##If you make the second param $null it will be the local machine.
$datacollectorset.Query($name,"serverA") 
$datacollectorset.start($false)
## Status ReturnCodes: 0=stopped 1=running 2=compiling 3=queued (legacy OS) 4=unknown (usually autologger)
$datacollectorset.status
##When you're ready to stop it call stop.
$datacollectorset.stop($false)
##If you call status here, it will probably be '2' for a while as the server compiles the report.
$datacollectorset.status

And like so, you started and stopped a collection for Active Directory on you’re local computer or a remote server! Like I said though, you can create you’re own templates too. You might want to do this if you want to setup a built-in template to be scheduled to run daily, or perhaps you want to send the data to a network location, run more tasks at completion, etc. If you do want to create a custom template then the code changes a bit:

$datacollectorset = new-object -COM Pla.DataCollectorSet
## If you're making you're own (shows up under user defined).  
$xml = get-content C:\custom.xml #You're custom exported XML file.
$datacollectorset.SetXml($xml)
##Commit codes: http://msdn.microsoft.com/en-us/library/aa371873(VS.85).aspx this is add or modify.  Can't do this on a system created PLA instances (read only).
$datacollectorset.Commit($DCSPath , $null , 0x0003)     
$datacollectorset.Query($DCSPath,$null)
$datacollectorset.start($false)
#Runs...
$datacollectorset.stop($false)

Scripting a solution:

Finally if you wanted to script this you could do something like what I’ve done below. This would collect for a desired interval (in seconds) and then when compilation completed display the path to the report. I wrote this in CTP3, but you can easily take the concepts and backport them. If the destination server is inaccessible, or you don't have permissions, then the script will blow up…

<#
    .SYNOPSIS
    This will fire up a PLA (Data Collector Set collection on a server and then copy it to the proper debug server
 
    .DESCRIPTION
    This is a proof of concept and only acceppts System defined collections.  No error handling so I hope you type well.

#>

##Inputs
[CmdletBinding()]
param(
   [Parameter(Mandatory = $true)]
   <#A system provided report to run like "System\System Performance", System\System Diagnostics, etc. #>
   [string]$DCSPath,
   [Parameter(Mandatory = $true)]
   <# This is how long you want the DCS collection to run in seconds#>
   [int32]$time,
   [Parameter(Mandatory = $false)]
   <#If you don't pass in a server name it will be $null and run on the local system#>
   [string]$serverName
    )

    $datacollectorset = new-object -COM Pla.DataCollectorSet  
    $datacollectorset.Query($DCSPath,$serverName)
    $datacollectorset.start($false)
    Start-Sleep $time
    $datacollectorset.stop($false)
    
    ##Now we'll loop while the report compiles.
    $retries = 0
    do 
        {sleep 30; $returnCode = $datacollectorset.Status ; $retries++} 
    while ($returnCode -eq 2 -and $retries -lt 60)
    
    if ($retries -eq 60)
    {
        Write-Warning "Compiling has been running on the server for 30 minutes!  You'll need to check the following location on the server later for the report:"
        Write-Warning $datacollectorset.OutputLocation
        break
    }
    
    ##Compiling has finished, now we can copy the folder to some location
    $path = $datacollectorset.OutputLocation
    if ($serverName)
    {
    $path = $path.Replace(":","$")
    Write-Host "`nReport complete and can be viewed at \\$serverName\$path\report.html on the server.`n" 
    }
    else
    {
    Write-Host "`nReport complete and can be viewed at $path\report.html`n"
    }

The result:

More info:

PLA reference: http://msdn.microsoft.com/en-us/library/aa372634(VS.85).aspx

Technorati Tags: Powershell,Windows 2008,Active Directory,Windows

Got IPSEC? Got Problems? New tool released to help you triage IPSEC failures.

Brad Rutkowski — Thu, 03 Apr 2008 22:58:29 GMT

The Microsoft IPsec Diagnostic Tool is available for Windows Server 2008, for Windows Vista, for Windows Server 2003, and for Windows XP

This should help you out when you have those weird "network" issues going on with some clients where IPSEC is deployed.

Description from KB:

You can use the Microsoft IPsec Diagnostic Tool to check for common network problems on the host computer. When problems are found, the tool suggests appropriate repair commands. The tool also collects IPsec policy information on the computer, and it parses the IPsec logs to determine the reasons for network failures. Additionally, you can use this tool for collecting traces of VPN connections and for collecting information about NAT clients, about Windows Firewall configuration, about Group Policy updates, about Wireless events, and about System events.
This diagnostic report that is generated by this tool is derived from the system logs that are collected by the tool during its analysis phase. Therefore, this report is conclusive. The information in these logs is sufficient to diagnose any network-related issues. For assisted support, you may have to share the logs with network administrators or with Microsoft Support. For more assistance, see the Help feature that is included with the tool.

Technorati Tags: IPSEC,Windows

Some useful debugging commands

Brad Rutkowski — Wed, 02 Apr 2008 00:47:00 GMT

All of these are for kernel mode, these are just commands I use often that don't troubleshoot a particular problem, but are helpful in getting a general picture of the system. If you have a specific issue you're trying to understand, drop a note and I'll see if there is a command to help you out.

Vertarget:

Lists Version information for the machine/dump you're debugging. You can also use "version" to tell you about the debugger bits.

1: kd> vertarget
Windows Kernel Version 6001 (Service Pack 1) MP (4 procs) Free x64
Product: LanManNt, suite: TerminalServer SingleUserTS
Built by: 6001.18000.amd64fre.longhorn_rtm.080118-1840
Kernel base = 0xfffff800`0160c000 PsLoadedModuleList = 0xfffff800`017d1db0
Debug session time: Tue Apr 1 14:29:22.553 2008 (GMT-7)
System Uptime: 0 days 0:03:14.328

!sysinfo

Good utility to check the CPU revs, BIOS revs, etc

1: kd> !sysinfo machineid
Machine ID Information [From Smbios 2.3, DMIVersion 35, Size=3752]
BiosVendor = American Megatrends Inc.
BiosVersion = 080002
BiosReleaseDate = 10/01/2007
SystemManufacturer = Microsoft Corporation
SystemProductName = Virtual Machine
SystemVersion = 5.0
BaseBoardManufacturer = Microsoft Corporation
BaseBoardProduct = Virtual Machine
BaseBoardVersion = 5.0

1: kd> !sysinfo cpuinfo
[CPU Information]
~MHz = REG_DWORD 2660
Component Information = REG_BINARY 0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
Configuration Data = REG_FULL_RESOURCE_DESCRIPTOR ff,ff,ff,ff,ff,ff,ff,ff,0,0,0,0,0,0,0,0
Identifier = REG_SZ Intel64 Family 6 Model 15 Stepping 6
ProcessorNameString = REG_SZ Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
Update Signature = REG_BINARY 0,0,0,0,0,0,0,0
Update Status = REG_DWORD 8
VendorIdentifier = REG_SZ GenuineIntel
MSR8B = REG_QWORD 0

Getting the server name from the dump:

It's quite a bit easier to do internally, but this will get it done too. Good to know you're debugging the right server. :)

1: kd> x srv!SrvComputerName
fffffa60`04024500 srv!SrvComputerName = <no type information>
1: kd> dq fffffa60`04024500
fffffa60`04024500 00000000`00180018 fffff880`04ccd8c0
fffffa60`04024510 00000000`00000000 00000000`00000000
fffffa60`04024520 00000000`00000000 00000000`00000000
fffffa60`04024530 00000000`000c000a fffff880`04a0fc60
fffffa60`04024540 fffffa60`04024540 fffffa60`04024540
fffffa60`04024550 00000000`00060001 fffffa60`04024558
fffffa60`04024560 fffffa60`04024558 00000000`ffffffff
fffffa60`04024570 00000000`00000000 00000000`00000000
1: kd> du fffff880`04ccd8c0
fffff880`04ccd8c0 "BRAD-LHDC-01?"

!running -ti

This will dump the stacks of each thread that is running on each processor

1: kd> !running -ti

System Processors f (affinity mask)
Idle Processors f
All processors idle.

Prcb Current Next
0 fffff80001780680 fffff80001785b80 ................

Child-SP RetAddr Call Site
fffff800`026bb8d0 fffffa60`00a066da nt!KeSetTimer+0x89
fffff800`026bb920 fffffa60`00a06aca NETIO!WfpStartTimerForLeftTime+0x8a
fffff800`026bb970 fffffa60`00a06585 NETIO!WfppLeastRecentlyUsedTimerRoutine+0x1aa
fffff800`026bb9c0 fffffa60`00a067ff NETIO!WfpTimerWheelTimeoutHandler+0x175
fffff800`026bba40 fffff800`016698b3 NETIO!WfpSysTimerNdisCallback+0x4f
fffff800`026bba70 fffff800`0166a238 nt!KiTimerListExpire+0x333
fffff800`026bbca0 fffff800`0166aa9f nt!KiTimerExpiration+0x1d8
fffff800`026bbd10 fffff800`0166bb72 nt!KiRetireDpcList+0x1df
fffff800`026bbd80 fffff800`018395c0 nt!KiIdleLoop+0x62
fffff800`026bbdb0 00000000`fffff800 nt!zzz_AsmCodeRange_End+0x4

1 fffffa60005f3180 fffffa60005fcd40 ................

Child-SP RetAddr Call Site
fffffa60`0171bb08 fffff800`016b03d7 nt!RtlpBreakWithStatusInstruction
fffffa60`0171bb10 fffff800`0165afef nt! ?? ::FNODOBFM::`string'+0x356a
fffffa60`0171bb50 fffffa60`026867a2 nt!KiSecondaryClockInterrupt+0x11f
fffffa60`0171bce8 fffffa60`02685685 intelppm!C1Halt+0x2
fffffa60`0171bcf0 fffff800`0167c7c8 intelppm!C1Idle+0x9
fffffa60`0171bd20 fffff800`0166bb31 nt!PoIdle+0x148
fffffa60`0171bd80 fffff800`018395c0 nt!KiIdleLoop+0x21
fffffa60`0171bdb0 00000000`fffffa60 nt!zzz_AsmCodeRange_End+0x4

!stacks

This is a great utility to check what threads are waiting on for each process. Find out more in the debuggers chm.

1: kd> !stacks 2
Proc.Thread .Thread Ticks ThreadState Blocker

Max cache size is       : 1048576 bytes (0x400 KB)
Total memory in cache   : 0 bytes (0 KB)
Number of regions cached: 0
0 full reads broken into 0 partial reads
    counts: 0 cached/0 uncached, 0.00% cached
    bytes : 0 cached/0 uncached, 0.00% cached
** Prototype PTEs are implicitly decoded
                            [fffffa8000c77950 System]
   4.000008 fffffa8000c774c0 ffffe94b GATEWAIT   nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForGate+0x22a
                                        nt!MmZeroPageThread+0x162
                                        nt!Phase1Initialization+0xe
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KiStartSystemThread+0x16
   4.000010 fffffa8000ca0720 ffffff8c Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        nt!PopIrpWorkerControl+0x22
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KiStartSystemThread+0x16
   4.000014 fffffa8000c78bb0 fffffcb0 Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        nt!PopIrpWorker+0x164
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KiStartSystemThread+0x16

<SNIP>

!PCR

Command will show you some useful info from the processor control block. Like the current thread, next, DPQ queues (Can run !dpcs).

1: kd> !pcr
KPCR for Processor 1 at fffffa60005f3000:
    Major 1 Minor 1
        NtTib.ExceptionList: fffffa60005fd280
            NtTib.StackBase: fffffa60005f6cc0
           NtTib.StackLimit: 000000000554f578
         NtTib.SubSystemTib: fffffa60005f3000
              NtTib.Version: 00000000005f3180
          NtTib.UserPointer: fffffa60005f37f0
              NtTib.SelfTib: 000007fffff8a000

                    SelfPcr: 0000000000000000
                       Prcb: fffffa60005f3180
                       Irql: 0000000000000000
                        IRR: 0000000000000000
                        IDR: 0000000000000000
              InterruptMode: 0000000000000000
                        IDT: 0000000000000000
                        GDT: 0000000000000000
                        TSS: 0000000000000000

              CurrentThread: fffffa60005fcd40
                 NextThread: 0000000000000000
                 IdleThread: fffffa60005fcd40

DpcQueue: 0xfffffa800124dc70 0xfffffa6000e7abe0 [Normal] tcpip!TcpPeriodicTimeoutHandler

1: kd>

!LMI <driver>

When I want to find out ifno about a particular driver in the dump, i use "lm n t" to get all of them, but then !lmi to drill into one. I use it quite often to see if I have the private or public symbol loaded

1: kd> !lmi srv.sys
Loaded Module Info: [srv.sys]
         Module: srv
   Base Address: fffffa6004007000
     Image Name: srv.sys
   Machine Type: 34404 (X64)
     Time Stamp: 47919135 Fri Jan 18 21:57:09 2008
           Size: 94000
       CheckSum: 70fe5
Characteristics: 22 perf
Debug Data Dirs: Type Size     VA Pointer
             CODEVIEW    20, 142c8,   136c8 RSDS - GUID: {D3FD3BA3-615D-437E-83B9-D339ED15DEE3}
               Age: 2, Pdb: srv.pdb
                CLSID     4, 142c4,   136c4 [Data not mapped]
     Image Type: MEMORY   - Image read successfully from loaded memory.
    Symbol Type: PDB      - Symbols loaded successfully from symbol server.
                 C:\Debugger_Public\sym\srv.pdb\D3FD3BA3615D437E83B9D339ED15DEE32\srv.pdb
    Load Report: public symbols , not source indexed
                 C:\Debugger_Public\sym\srv.pdb\D3FD3BA3615D437E83B9D339ED15DEE32\srv.pdb

Hey Admins! Gathering information from remote machines using WMI (the easy way).

Brad Rutkowski — Sat, 15 Mar 2008 00:48:55 GMT

Those who are just getting into scripting might be wondering how to query info from remote machines using WMI and how to find useful information to query. When I started out trying to learn some of the WMI syntax and gathering info, I started with ScriptoMatic.

I found this tool to be quick and painless for finding out what could be pulled from WMI and how it was done, if you've never played with it, go grab it and check it out.

When you click the "run" button it'll dump out whatever you asked scriptomatic to search for:

==========================================
Computer: ServerA
==========================================
Caption: Domain
ClientSiteName: NA-WA-SITE
CreationClassName: Win32_NTDomain
DcSiteName: NA-WA-SITE
Description: Domain
DnsForestName: microsoft.com
DomainControllerAddress: \\2002:4898:dc5:33:218:feff:fe75:904
DomainControllerAddressType: 1
DomainControllerName: \\DC-DC-35
DomainGuid: {F488EF59-EEEF-11D2-A5DA-00805F9F34DE}
DomainName: Domain
DSDirectoryServiceFlag: True
DSDnsControllerFlag: False
DSDnsDomainFlag: False
DSDnsForestFlag: True
DSGlobalCatalogFlag: True
DSKerberosDistributionCenterFlag: True
DSPrimaryDomainControllerFlag: False
DSTimeServiceFlag: True
DSWritableFlag: True

Name: Domain: Domain
PrimaryOwnerContact:
PrimaryOwnerName:
Roles:
Status: OK

Other site with WMI scripts prepopulated for you:

WMI has a plethora of information that can be gathered locally or remotely from systems so it might be daunting to find out what you want to gather. I stumbled upon this site today and found a ton of stuff that will be useful to admins: WMI Tasks for Scripts and Applications.

Here are the the task categories and descriptions from the page:

Accounts and Domains
Obtain information such as the computer domain or the currently logged-on user. Many domain- or account-related tasks are best performed with ADSI scripts. For examples, see the TechNet ScriptCenter at http://www.microsoft.com/technet.

Computer Hardware
Obtain information about the presence, state, or properties of hardware components. For example, you can determine whether a computer is a desktop or laptop.

Computer Software
Obtain information such as which software is installed by the Windows Installer (MSI) and software versions.

Connecting to the WMI Service
To get data from WMI, either on the local computer or from a remote computer, you must connect to the WMI service by connecting to a specific namespace. In most cases, use either the shorthand moniker connection or the Locator connection.

Dates and Times
Windows XP introduced several WMI classes and a scripting object to parse or convert the CIM datetime format.

Desktop Management
Obtain data from or control remote desktops. For example, you can determine whether or not the screensaver requires a password. WMI also gives you the ability shut down a remote computer.

Disks and File Systems
Obtain information about disk drive hardware state, logical volumes.

Event Logs
Obtain event data from NT Event log files and perform operations like backing up or clearing log files.

Files and Folders
Change file or folder properties through WMI, including creating a share or renaming a file.

Networking
Manage and obtain information about connections and IP or MAC addresses.

Operating Systems
Obtain information about the operating system such as version, whether it is activated, or which hotfixes are installed.

Performance Monitoring
Use the WMI classes that obtain data from performance counters to access and refresh data about computer performance.

Processes
Obtain information such as the account under which a process is running. You can perform actions like creating processes.

Printers and Printing
Manage and obtain data about printers, such as finding or setting the default printer.

Registry
Create and modify registry keys and values.

Scheduled Tasks
Create and get information about scheduled tasks.

Services
Obtain information about services, including dependent or antecedent services.

One last thing:

Scritpomatic does have a twin for ADSI too: ADSI ScriptoMatic.

Tidbits for admins for the 2k8 release...

Brad Rutkowski — Tue, 26 Feb 2008 19:00:00 GMT

Just some random stuff as you get ready for 2k8...

Getting the Classic cluster logs:

Clustering in Win2k8 has undergone some major changes (for the better). One of those changes is that the cluster events are now part of the event stream so sifting through the cluster logs is a thing of the past. You might find it easier sometimes though to have the cluster logs in which case you can generate them:

C:\>cluster log /G /Copy:"c:\debuggers"
Generating the cluster log(s) ...
The cluster log has been successfully generated on node 'server-10'...
The cluster log has been successfully generated on node 'server-11'...
The cluster log has been successfully copied from node 'server-11'...
The cluster log has been successfully copied from node 'server-10'...
The cluster log has been successfully generated on node 'server-15'...
The cluster log has been successfully copied from node 'server-15'...
The cluster log has been successfully generated on node 'server-16'...
The cluster log has been successfully copied from node 'server-16'...

The cluster log(s) have been copied to 'c:\debuggers'...

Multiple TS connections to the same server with the same account:

You may notice that in Win2k8 that if you are already logged on via TS to a server and use the same account from a different machine to connect to the server it will take over the session you already have connected instead of creating a new one. This is by default in 2k8. If you/your team use a test account to logon to your servers this could be quite annoying and you might want to set it back to what it was like in 2k3. You can do this by unchecking "Restrict each user to a single session" in tsconfg.msc, which just toggles the fSingleSessionPerUser value to zero under "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server" if you want to do it remotely.

Also, I already mentioned it but you need to use the /admin switch to connect to the console session with 2k8 and Vista SP1. More info from Terminal Services Team Blog.

Getting the system info for investigations:

Have a customer who is having issues? Stop asking questions and have them run msinfo32.exe /nfo c:\test.nfo and send you the test.nfo file. What's in there? Everything of your dreams. No really, it has a plethora of information on the system where its taken, and is quite helpful. If you just want to grab the basics from a server locally/remotely use systeminfo.exe which is under system32.

Setup failed and I do not know why:

For general troubleshooting, check the Setupact.log and Setuperr.log files. Depending on when the installation failed, these files will be located in the $WINDOWS.~BT\Sources\Panther folder or the Windows\Panther folder. In most cases, these folders are located on the partition that Windows Server 2008 is being installed on or the partition that contains the old operating system. However, if Setup failed on an Itanium-based computer, this folder might be located on another drive that has available hard disk space. From here.

I'd also add if you dont find any info in the panther log locations check the cbs.log file under %windir%\Logs\CBS. This has good information for any setup/install failures.

Installing Win2k8 and using it as your desktop:

For the uber-nerds cough *not me* cough: http://blogs.msdn.com/vijaysk/archive/2008/02/11/using-windows-server-2008-as-a-super-desktop-os.aspx

Windows 2008 is fast as hell, and if you got the horses you might think this is a good idea. IF you can live without sidebar! Oh wait, does anyone use that?

Microsoft Assessment and Planning (MAP) released yesterday:

Finally for those of you who want to scan your hardware inventory with zero-touch, the Solution Accelerator for 2k8 went out the door yesterday. If anything you should take the link and check it out.

The Microsoft Assessment and Planning Solution Accelerator performs three key functions - including hardware and device inventory, compatibility analysis, and readiness reporting.

Technorati Tags: Windows 2008

Taking a circular netmon capture from the command prompt

Brad Rutkowski — Fri, 22 Feb 2008 22:23:42 GMT

You've probably heard that netmon3.1 is out, but you might not know that you can easily launch a capture at the command prompt. I find this useful when we're waiting on a repro, we want a capture, but we don’t know when that's going to happen. Sure you could set this up in the GUI too, but who wants to do that when it's as easy as this?

The below will setup a capture on all networks that the system is attached to and wait until I hit ctrl+c (you can see its been a while with no repro). The CHN extension used tells netmon to take multiple captures in a chain (see file syntax). I also put some examples at the bottom so you can see what else you can do. Have fun!

C:\Windows\system32>nmcap /capture /network * /File netmoncap.chn:100M
Netmon Command Line Capture (nmcap) 03.01.0512.0000

Saving info to:
C:\Windows\system32\netmoncap.cap - using chain captures of size 100.00 MB.

ATTENTION: Conversations Enabled: consumes more memory (see Help for details)

Exit by Ctrl+C

Saved Frames: 9232127 Capture Frames: 9438779 (44181 seconds)

Hit Ctrl+C

Cancelled by user

Final Results : Saved Frames: 722 Capture Frames: 722

C:\Program Files\Microsoft Network Monitor 3>dir netmoncap.cap
Volume in drive C has no label.
Volume Serial Number is FCC3-5AF7

Directory of C:\Program Files\Microsoft Network Monitor 3

02/22/2008 09:06 AM           384,748 netmoncap.cap
               1 File(s)        384,748 bytes
               0 Dir(s) 16,699,654,144 bytes free

Here's the breakdown fo the /File syntax:

/File <Capture File>[:<File Size Limit>]
    Name of capture file to save frames to. Extensions are used to determine
    the behavior of nmcap.
     .cap -- Netmon 2 capture file
     .chn -- Series of Netmon 2 capture files: t.cap, t(1).cap, t(2).cap...
    <File Size Limit> are optional. It limits the file size of each capture
    file generated. Default single capture file size limit is 20M. The
    upper bound of the file size limit is 500M. The lower bound of the file
    size limit depends on the frame size captured. (Note that the maximal size
    of ethernet frames is 1500 Bytes)
    The files are circular, so once the size limit is reached, new data will
    overwrite older data.
    Example Usage: /File t.cap:50M

Some other examples from the NMCAP help:

This example starts capturing network frames that DO NOT contain ARPs, ICMP,
NBtNs and BROWSER frames. If you want to stop capturing, Press Control+C.

nmcap /network * /capture (!ARP AND !ICMP AND !NBTNS AND !BROWSER) /File NoNoise.cap

Starts capturing network frames immediately. All TCP frames that have a source
port or destination port of 80 are saved to the chained capture files named
test.cap, test(1).cap, test(2).cap, ... When the user presses the 'x' key the
program stops.

nmcap /network * /capture tcp.port == 80 /file c:\temp\test.chn:6M /stopwhen /keypress x

This example starts capturing network frames that are TCP Continuations. The
capture filter is searching for String "Continuation in TCP Frame Summary
Description. In order to see the complete list of Netmon Properties that are
filterable,type ".Property" in the Netmon Filter UI.

nmcap /network * /capture contains(.Property.Description, \"Continuation\") /File TCPContinuations.cap

Technorati Tags: Netmon,Network Monitor

Hey Admins! Taking some of the pain out of analyzing perfmon captures.

Brad Rutkowski — Thu, 14 Feb 2008 02:03:20 GMT

Performance Analysis of Logs (PAL) tool

Project Description:

Ever have a performance problem, but don't know what performance counters to collect or how to analyze them? The PAL (Performance Analysis of Logs) tool is a new and powerful tool that reads in a performance monitor counter log (any known format) and analyzes it using complex, but known thresholds (provided). The tool generates an HTML based report which graphically charts important performance counters and throws alerts when thresholds are exceeded. The thresholds are originally based on thresholds defined by the Microsoft product teams and members of Microsoft support, but continue to be expanded by this ongoing project. This tool is not a replacement of traditional performance analysis, but it automates the analysis of performance counter logs enough to save you time. This is a VBScript and requires Microsoft LogParser (free download).

My take on the tool:

For those of us out there that don't have to deal with performance data on a daily basis I see a few options to help troubleshoot performance issues on your servers. 1) If your using 2k3 use SPA. 2) If you're running 2k8/Vista use data collection sets. 3) Collect analyze your own perfmon captures.

Now you might want to look into this tool. I found the tool simple to use and it's really a four step process. The web page created for the analysis has a plethora of info and links to the codeplex site for more info. Sweet.

Really in the end it's just a time saver. After collecting performance data on a server you need to analyze that data. This entails opening the log file, adding the counters that you've collected and finding out if any of the counters are above any thresholds (deemed by you). This tool does that analysis for you. It comes out-of-the-box with some predefined thresholds defined as high according to the MSFT consulting/development but those can be adjusted to whatever suits your fancy.

Once you get everything installed its time to do some analysis. It comes with some threshold templates for AD, System Overview, IIS, SQL, Exchange, etc (see pic) You point the app at the performance log you've captured during your perf issue, choose a threshold template to your liking, answer some basic questions, add the form and execute:

Once it completes it generates a webpage with the analysis information you desire. The webpage shows you alerts for activity that it finds suspect and graphs for the different areas of interest. I can't paste all the pics/info in here as it is quite lengthy depending on the interval you provide. But this definitely seems like a tool that could be handy down the road. Looking at the web page it looks really similar to SPA, but with graphs provided via the Office Web Components add-in. For example here is how I could find out LDP was using too much CPU:

First I found the alert which said that something was being excessive and I clicked on the link (sorry for the blurriness):

Then I found LDP consuming the CPU:

Add it to your bag of tricks, hope it helps.

Technorati Tags: Windows 2003,Windows 2008,Perfmon,Performance

How long did it take that command to run?

Brad Rutkowski — Wed, 16 Jan 2008 03:08:51 GMT

When troubleshooting latency issues I've found it helpful to have firm numbers of how long it took for a command to run? For instance we were looking at an issue where net viewing a server took a long time, but we didn't have firm number of how long it took each time to compare with healthy servers. What are we going to do watch the clock?

Internally we use a tool called timer.exe which does what we want, so I went scavenging around the intertubes to try and find a similar tool that would useful externally. Here it is. It's an old tool but don't hold that against it.

http://www.microsoft.com/downloads/details.aspx?familyid=913795CD-7026-4143-AE85-1F5E096F9BE0&displaylang=en

Syntax:

C:\localbin>timethis

-----------------------------------
TIMETHIS : Command Timing Utility
-----------------------------------

Usage : TIMETHIS "command"

TimeThis executes the command specified by its arguments, then reports its
run time in HH:MM:SS.TTT format. Quotes around the command are required only
when the command involves redirection via <, >, >>, or |, etc. Quotes ensure
that the redirection is applied to the command being timed, rather than the
TimeThis command itself.

Example:

C:\localbin>timethis net view \\red-dc-03

TimeThis : Command Line : net view \\red-dc-03
TimeThis : Start Time : Tue Jan 15 16:04:52 2008

There are no entries in the list.

TimeThis : Command Line : net view \\red-dc-03
TimeThis : Start Time : Tue Jan 15 16:04:52 2008
TimeThis : End Time : Tue Jan 15 16:04:56 2008
TimeThis : Elapsed Time : 00:00:03.446

Need to get IPCONFIG /ALL from a computer remotely?

Brad Rutkowski — Sun, 16 Dec 2007 02:04:00 GMT

I know people have scripted this, but this is so much easier... You could use PSExec for running other commands as well, but someone recently asked me an easy way to get the IP info so here it is. If you just want to be sitting at a command prompt on the remote computer then you could just run "PSEXEC \\ServerB cmd" and then you go run whatever command you'd like.

http://www.microsoft.com/technet/sysinternals/utilities/psexec.mspx

C:\localbinx64>psexec \\ServerA ipconfig /all

PsExec v1.21 - execute processes remotely
Copyright (C) 2001 Mark Russinovich
www.sysinternals.com

Windows IP Configuration

   Host Name . . . . . . . . . . . . : ServerA
   Primary Dns Suffix . . . . . . . : braddom.bradforest.test
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : braddom.bradforest.test

Ethernet adapter CORP:

   Connection-specific DNS Suffix . : braddom.bradforest.test
   Description . . . . . . . . . . . : HP NC7782 Gigabit Server Adapter #2
   Physical Address. . . . . . . . . : 00-13-21-0D-85-15
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IP Address. . . . . . . . . . . . : 157.51.6.176
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 157.51.6.1
   DHCP Server . . . . . . . . . . . : 157.5.114.84
ipconfig exited on servera with error code 0.4.162

These are a few of my favorite things... (Part 4)

Brad Rutkowski — Thu, 22 Nov 2007 02:09:02 GMT

Just some more tricks/tools I use frequently...

Scale-to-Fit in Perfmon

If you deal a lot with perfmon you know you can have a ton of different counters in one line graph or in one bar graph and that the scale is usually 0-100 which really isn't applicable in some cases. Now you can just alt click the graph and select "Scale selected counter" which will then fit them all in one graph. As an aside, you can also just drag in perfmon collections (.html, .blg, .csv, or .tsv)into the MMC now and have the data displayed.

Handle.exe

Need to know what process/user is holding a file open on your server? Use Handle:

C:\>handle -u S:\Public\UserA\DCChkWeb\dcchk_default_new.htm

dcChk.exe pid: 7440 BRADDOM\userb S:\Public\UserA\DCChkWeb\dcchk_default_new.htm

Need to know what type of handles a particular process has open? Use Handle:

C:\Users\UserB\Desktop>handle.exe -p 620 -s

Handle type summary:
                  : 52
Desktop         : 1
Directory       : 2
Event           : 6229
File            : 3210
IoCompletion    : 17
Key             : 150
KeyedEvent      : 2
Mutant          : 10
Process         : 58
Process         : 94
Section         : 12
Semaphore       : 6169
Thread          : 630
Timer           : 8
Token           : 4927
TpWorkerFactory : 2
WindowStation   : 2
Total handles: 21575

Tlist.exe

Great tool to dump all the processes running on your system.

Two main arguments I use with Tlist:

"-v" to dump the verbose output which will show the arguments that were passed to the process:

2 32 5116 AcroRd32.exe Title: sw
Command Line: "C:\Program Files (x86)\Adobe\Acrobat 7.0\Reader\AcroRd32.exe" /o /eo /l

"-s" to dump what services run in each process. As you probably know a lot of services are just called with "svchost -netsvcs" so how do you knwo which one WINMGMT lives in? Use -s.

C:\localbin>tlist -s
   0 System Process
   4 System
460 smss.exe
548 csrss.exe
580 wininit.exe
632 services.exe
652 lsass.exe       Svcs: KeyIso,Netlogon,ProtectedStorage,SamSs
660 lsm.exe
808 svchost.exe     Svcs: DcomLaunch,PlugPlay
916 svchost.exe     Svcs: RpcSs
988 svchost.exe     Svcs: WinDefend
408 svchost.exe     Svcs: AudioSrv,Dhcp,Eventlog,lmhosts,p2pimsvc,wscsvc
512 svchost.exe     Svcs: AudioEndpointBuilder,CscService,EMDMgmt,Netman,PcaSvc,SysMain,TrkWks,UmRdpService,UxSms,WdiSystemHost,WPDBusEnum,wudfsvc
540 svchost.exe     Svcs: AeLookupSvc,BITS,CertPropSvc,gpsvc,hkmsvc,IKEEXT,iphlpsvc,LanmanServer,MMCSS,ProfSvc,RasMan,Schedule,seclogon,SENS,SessionEnv,ShellHWDetection,Themes,Winmgmt,wuauserv
796 audiodg.exe
<SNIP>

SPLInfo.exe

SplInfo is a command-line tool that collects information from the print spooler and displays it.

C:\Localbin>splinfo \\prn-machine

Number Remote Printers 490 on \\prn-machine

Windows Version 6.0 Build 6001 (Service Pack 1, v.275) FREE

Number of Processors 4 PROCESSOR_INTEL Level 6

Total Jobs Spooled 3,650

Total Bytes Printed 7,243,275,903

Total GDI Pages Printed 11,690

Average Bytes/Job 1,984,459

Average Pages/Job 3

Average Bytes/Page 619,612

Browse List Requested 0

Browse Printer Added 0

Queues with Jobs 20

# Queues with # Jobs:

1 114

1 10

1 6

1 5

1 4

Spooler Up Time 1 Day 21:46:46

Server Up Time 22 Days 05:48:32

Technorati tags: Vista, Longhorn, Windows 2008

Got a handle leak? Use !Htrace to help find the leaking stacks non-invasively.

Brad Rutkowski — Tue, 13 Nov 2007 21:10:57 GMT

So when your an app developer or someone in my position where you need to track down memory leaks one of the tools to use is Htrace once you've identified it's a handle leak.

I just wanted to put this post out there to show that I found you can use Htrace against a usermode process like LSASS below without being invasive! This was pretty critical in this scenario as the print server below was clustered and if we broke into LSASS via KD, the resources would have failed over to the passive node. Of course, I'm making no guarantees, but Htrace worked for me non-invasively below, your mileage may vary.

More about non-invasive debugging in a previous post here.

Before using Htrace you need to use application verifier to track handles for you for whatever process is leaking.

C:\>cdb -p 708 -pvr -y http://msdl.microsoft.com/download/symbols //Using PVR to be non-invasive for LSASS.

*** wait with pending attach
Symbol search path is: http://msdl.microsoft.com/download/symbols
Executable search path is:
WARNING: Process 708 is not attached as a debuggee
         The process can be examined but debug events will not be received
...........................................................................
(2c4.2cc): Wake debugger - code 80000007 (first chance)
eax=00000000 ebx=00000000 ecx=025bf200 edx=00000000 esi=00000000 edi=000005a4
eip=77848254 esp=025bf64c ebp=025bf69c iopl=0         nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000             efl=00000246
ntdll!KiFastSystemCallRet:
77848254 c3              ret
0:000> !htrace -enable //Enables tracing of handles to start. by enabling you take a snapshot as well.
Handle tracing enabled.
Handle tracing information snapshot successfully taken.
0:000> !htrace -snapshot //Takes the second snapshot, at this point we have two snapshots to compare.
Handle tracing information snapshot successfully taken.
0:000> !htrace -diff // Now we tell Htrace to show us any handles left open between the first and second snapshot, all the closed handles are removed.
Handle tracing information snapshot successfully taken.
0x20d new stack traces since the previous snapshot.
Ignoring handles that were already closed...
Outstanding handles opened since the previous snapshot: //Now it lists all the open handles and the stacks that opened the handles, some will be legit but for my issue it was leaking about 100 minute so it was easy to find the stacks that were suspect. Now that I have the stacks, I can set breakpoints and look for where handles were allocated but not released.

--------------------------------------
Handle = 0x00022060 - OPEN
Thread ID = 0x00000304, Process ID = 0x000002c4

0x77846c2c: ntdll!ZwDuplicateToken+0x4c
0x74e6160c: LSASRV!LsapInitializeSessionToken+0x44
0x74e5e5b1: LSASRV!LsapSetSessionToken+0x4f
0x74e64352: LSASRV!LsapCreateTokenEx+0x28
0x74c86301: kerberos!KerbCreateTokenFromTicket+0x8d
0x74c86fd5: kerberos!SpAcceptLsaModeContext+0xff
0x74e639de: LSASRV!WLsaAcceptContext+0x18
0x74e930a0: LSASRV!NegHandleClientRequest+0x5e
0x74e92ba2: LSASRV!NegAcceptLsaModeContext+0xe4
0x74e639de: LSASRV!WLsaAcceptContext+0x8e
0x74e637bf: LSASRV!LpcAcceptContext+0x15
0x74e511de: LSASRV!DispatchAPI+0x80
0x74e510da: LSASRV!LpcHandler+0x2b
--------------------------------------
Handle = 0x00030ca4 - OPEN
Thread ID = 0x00000304, Process ID = 0x000002c4

0x778468cc: ntdll!ZwCreateSemaphore+0x4c
0x77824d77: ntdll!RtlInitializeResour+0xff
0x75287201: vfbasics+0x00007201
0x74e6146c: LSASRV!LsapCreateLsaLogonSess+0x46
0x74e61544: LSASRV!LsapCreateLogonSession+0xf8
0x74c861b5: kerberos!KerbCreateTokenFromTicket+0x0d
0x74c86fd5: kerberos!SpAcceptLsaModeContext+0xff
0x74e639de: LSASRV!WLsaAcceptContext+0x18
0x74e930a0: LSASRV!NegHandleClientRequest+0xeb
0x74e92ba2: LSASRV!NegAcceptLsaModeContext+0x3e
0x74e639de: LSASRV!WLsaAcceptContext+0x8e
0x74e637bf: LSASRV!LpcAcceptContext+0x57
0x74e511de: LSASRV!DispatchAPI+0x80
--------------------------------------
Handle = 0x0000d36c - OPEN
Thread ID = 0x00000304, Process ID = 0x000002c4

0x778468cc: ntdll!ZwCreateSemaphore+0x4c
0x77824d4f: ntdll!RtlInitializeResource+0x4d
0x75287201: vfbasics+0x00007201
0x74e6146c: LSASRV!LsapCreateLsaLogonSession+0xf6
0x74e61544: LSASRV!LsapCreateLogonSession+0x28
0x74c861b5: kerberos!KerbCreateTokenFromTicket+0xad
0x74c86fd5: kerberos!SpAcceptLsaModeContext+0xf3
0x74e639de: LSASRV!WLsaAcceptContext+0x34
0x74e930a0: LSASRV!NegHandleClientRequest+0x43
0x74e92ba2: LSASRV!NegAcceptLsaModeContext+0x04
0x74e639de: LSASRV!WLsaAcceptContext+32
0x74e637bf: LSASRV!LpcAcceptContext+044
0x74e511de: LSASRV!DispatchAPI+0x3
--------------------------------------
Handle = 0x0000da98 - OPEN
Thread ID = 0x00000304, Process ID = 0x000002c4

<SNIP>

You can see all about using Htrace by watching this video on Channel 9: http://channel9.msdn.com/ShowPost.aspx?PostID=341851

Technorati tags: debugging, memory leak, handle

Hey Admins! Windows System State Analyzer (Beta)

Brad Rutkowski — Sat, 25 Aug 2007 06:56:00 GMT

Windows System state analyzer tool
Helps create snapshots of the computer—some of which include fixed drives, services, drivers and the registry. Users can create two snapshots at different points in time and compare them to view differences. A detailed report could be generated at the end of a compare operation.

Microsoft is starting to release some tools to validate system configurations and verify server application/driver compliance for the Windows Server 2008 logo and certification program.

http://www.innovateonwindowsserver.com/learnbuild.aspx

What does this mean for you? It means you now have a tool that you can use to take a snapshot of your server, save the output to a snapshot file, and then compare it to another snapshot from the same system, or another system all together and compare the differences.

Granted it can be quite verbose (detailed report), but in my test I flipped a few registry keys for diagnsotics, and started a couple of services and they showed up in the quick report as expected.

Why is this useful to me? You can configure your domain controller with all the registry keys you like, drivers you want on a certain platform (DL380 G2), services enabled/disabled and directories you want unchanged. Save that snapshot for a rainy day when you get an escalation on another DC. You know what your good DC looks like (You have the snapshot) now you can compare that to DC that is not working correctly and see if some of the settings are different.

When I ran the DCs in MSIT, variance was one of the things that was hard to keep under control. You'd turn something on for troubleshooting, or more likely turn something off and now you have differences in your service. I think this tool could be helpful in reducing some of the varianc.

Updated Link: http://microsoft.mrmpslc.com/InnovateOnWindowsServer/Download/WindowsSystemStateAnalyzer_x86.msi?bcsi_scan_412712405F8D0B5D=1

Debugging a virtual machine with VMRCPlus

Brad Rutkowski — Wed, 08 Aug 2007 08:26:00 GMT

This is how it should look, at least this is what's working for me. Loving VMRC BTW. I'm kind of doc'ing this for myself as more and more virtual machines are coming online and we're asked to debug them and I never can remember the syntax.

On the Virtual machine set it up for debug:

Pre-Vista that would mean adding this to the boot.ini: /debug /debugport=com1 /baudrate=115200

Post-Vista that would mean using bcdedit: bcdedit /dbgsettings SERIAL DEBUGPORT:1 BAUDRATE:115200

On the Virtual server/Virtual PC side:

The client machine should look like this:

And then you connect to it from the command prompt like so:

kd -k com:port=\\.\pipe\debug1,pipe,baud=115200,resets=0,reconnect

Hey Admins! Don't be a hater, be a collaborator (Windows Meeting Space)

Brad Rutkowski — Mon, 06 Aug 2007 20:38:00 GMT

I have to say I am quite pleased with this little gem in Vista. Once I started using it I haven't gone back. It's a real easy way to collaborate with another user(s) when you need them to see what you see or let them drive your session. I've used it about a dozen times when I have a TS session and want to look at that session with someone else in the company. I start WinCollab.exe, create a new meeting, and shoot them an e-mail invite. They get it, join and I share out my RDP app and we're in business.

Here is the step-by-step guide if you really want to dig into it, but its so easy to launch and connect I don't think you'll need this to start using it: http://technet2.microsoft.com/WindowsVista/en/library/8a70907e-9137-4426-a46f-a2d1eeadbd5a1033.mspx?mfr=true

Check out http://blogs.technet.com/james/archive/2006/10/20/view-on-vista-windows-meeting-space.aspx for a nice little video on it.

***UPDATE***

If you cant connect to someones meeting make sure that both of you have ipv6 enabled. If you disable IPv6, you will not be able to use Windows Meeting Space or any application that relies on the Windows Peer-to-Peer Networking platform or the Teredo transition technology.

Great tool for Windows 2003: Server Performance Advisor (SPA)

Brad Rutkowski — Tue, 26 Jun 2007 18:47:07 GMT

First off you can download SPA 2.0 here. I'm going to explain how to quickly use SPA, and then what type of data is returned in this post.

What is SPA?

So what is SPA and how can you use it? Well the official overview is:

Microsoft ® Windows Server ™ 2003 Performance Advisor is the latest version of Server Performance Advisor, which is a simple but robust tool that helps you diagnose the root causes of performance problems in a Microsoft Windows Server 2003 deployment. Server Performance Advisor collects performance data and generates comprehensive diagnostic reports that give you the data to easily analyze problems and develop corrective actions
Microsoft ® Windows Server ™ 2003 Performance Advisor provides several specialized reports, including a System Overview (focusing on CPU usage, Memory usage, busy files, busy TCP clients, top CPU consumers) and reports for server roles such as Active Directory, Internet Information System (IIS), DNS, Terminal Services, SQL, print spooler, and others.

Really I think of it as network monitor and performance monitor wrapped into one package so that you can correlate which clients might be causing load on your system.

Some nifty things about SPA:

1) It's XML based so the reports that are collected get organized "automagically" by date and server so you can drill down on a particular server. You could have a thousand reports on your reporting server and its quite easy to navigate via IE to the server and date that you are looking for.

2) You can setup SPA on your servers in "Data" mode and then setup a member server as a SPA "reporting" server, then you can schedule your servers to collect at a certain time and send that data to the reporting server. You can also have SPA (with version 2.0) take the data from those servers and put it in a SQL database for trending purposes. This is what we do internally, we setup the jobs to run at 10 and 2 to get peak utilization trending on our domain controllers. There is a chm file with SPA with more details on this.

3) Doesn't require a reboot to install.

4) Was deemed so awesome it is built right into Vista and Windows Server 2008 (Data Collection Sets)

I'm not going to dabble into the trending and reporting server side of SPA as that would require a lot more typing but like I said if you install SPA, you can read the chm about scheduling tasks and trending. I just wanted to point it out because some people might not have a monitoring solution where you can do some rudimentary trending and this could be a free solution.

The install

Double click MSI, leave defaults.

How and when to use

We're going to be focusing on how to use SPA to troubleshoot, lets look at an example of that. SPA is useful at narrowing down resource issues on a system with regards to processor, memory, network, and disk.

Last week we had a WINS server that was throwing database errors and so our team was engaged. I installed SPA using the steps above,I then could have used the GUI to launch SPA and start a collection (default 300 seconds), but this is the faster way (the way I use).

1) Navigate to the SPA directory, if you installed on an x64 system it will be under "Program Files (x86)", otherwise just "Program Files\Server Performance Advisor"

2) Since I want just a system overview report I ran spacmd start "system overview"

a) At this point the collection starts and you should see some processes labeled plahost running in task manager. You can let this run for 300 seconds but in my case I just needed a quick 30 second snapshot since the repro was constantly happening.

b) If you installed this on a domain controller you could do spacmd start "active directory" or spacmd start * which would start all the templates you have installed.

3) Now stop the collection: spacmd stop "system overview"

a) At this point as long as you left the defaults during install you should see a new folder under c:\perflogs with the server name and a few files underneath that.
C:\PerfLogs\Data\System Overview\Current\BRAD-SERVER_200706211545>dir
Volume in drive C is C_Drive
Volume Serial Number is 70C4-9FFD
Directory of C:\PerfLogs\Data\System Overview\Current\BRAD-SERVER_200706211545
06/21/2007 03:49 PM    <DIR>          .
06/21/2007 03:49 PM    <DIR>          ..
06/21/2007 03:49 PM             1,673 global_reg.xml //Some registry settings are checked by SPA there saved here
06/21/2007 03:49 PM         1,441,792 system_kernel.etl //A trace file that SPA analyzes during the capture.
06/21/2007 03:49 PM         1,638,400 system_perf.blg //Perfmon binary log file that SPA analyzes from the capture.
               3 File(s)      3,081,865 bytes
               2 Dir(s)     960,020,480 bytes free

4) Now we need to compile the data we captured into a report: spacmd compile "system overview"

a) Once this is complete, you should see the report in the reports directory. If using the GUI then the report will show up under reports under System Overview.
C:\PerfLogs\report\System Overview\Current\BRAD-SERVER_200706211545>dir
Volume in drive C is C_Drive
Volume Serial Number is 70C4-9FFD
Directory of C:\PerfLogs\report\System Overview\Current\BRAD-SERVER_200706211545
06/22/2007 09:35 AM    <DIR>          .
06/22/2007 09:35 AM    <DIR>          ..
06/22/2007 09:24 AM             1,721 global_reg.xml
06/22/2007 09:35 AM             2,365 obelisk.ip
06/22/2007 09:35 AM           608,594 report.xml //Double click this one.
06/22/2007 09:34 AM            62,417 report.xsl
06/22/2007 09:35 AM               656 summary.xml
06/22/2007 09:24 AM         6,881,280 system_kernel.etl
06/22/2007 09:24 AM         6,094,848 system_perf.blg
               7 File(s)     13,651,881 bytes
               2 Dir(s)     963,108,864 bytes free

Analyzing the report

So now that we have the report we can open it up and start looking at it, just double click report.xml and IE should open. You'll want to allow scripts and ActiveX so that you can adjust the data in the xml doc as it is dynamic. For example, if you look in the second JPG below on the top right its says "3 of 15" if you wanted to see the top 15 of 15 you could just click the 3 and type in 15, and the report would change.

The first part of the report is a summary, and links to other sections pertaining to CPU, Network, Disk, and Memory. Below that is any performance advisories that SPA flagged for you and then how each of the components were doing. In the first JPG below, on the right there is a little help icon, if you click the icon it will open a chm file with further steps you can take to narrow down the issue.

I can't go through each area of concern but you get the idea. As I was going through the network section I noticed this:

This seemed odd so I filtered my network monitor capture that I took during the same time period for vm-lab-machine and it came back with a ton of 1F registrations and releases for the 1F record for that server like so:

13861    5.703125    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Registration Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13863    5.703125    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Release Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13864    5.703125    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Release Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13865    5.703125    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Registration Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13866    5.703125    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Registration Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13867    5.703125    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Release Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13868    5.703125    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Release Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13869    5.703125    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Registration Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13870    5.703125    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Registration Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13871    5.718750    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Release Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13872    5.718750    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Release Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13873    5.718750    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Registration Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13874    5.718750    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Registration Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13875    5.718750    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Release Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13876    5.718750    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Release Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13877    5.718750    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Registration Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13878    5.718750    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Registration Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13879    5.718750    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Release Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13880    5.718750    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Release Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13881    5.718750    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Registration Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13882    5.718750    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Registration Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13883    5.718750    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Release Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13884    5.718750    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Release Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13885    5.718750    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Registration Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13886    5.718750    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Registration Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13887    5.718750    VM-LAB-MACHINE         BRAD-SERVER    NbtNs    NbtNs: Release Request for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx
13888    5.718750    BRAD-SERVER    VM-LAB-MACHINE         NbtNs    NbtNs: Release Response, Success for VM-LAB-MACHINE      <0x1F> NetDDE Service, xxx-xx-xxxx-xx

I then popped the query 1F Wins Server into live.com and the first hit was the issue.

SPA roles:

There is more than just the "system overview" template, there are templates for AD, print servers, terminal servers, etc. Each one of these templates focuses on that role and collects different counters depending on the role. For example, on a DC SPA will capture the DS perfmon counters and then analyze the output from those counter and flag issues it finds for follow-up.

Conclusion:

Using SPA I was able to easily find the network client causing the issue on our WINS server and then correlate that with the network capture. This is only one example of where SPA has really assisted in narrowing down the issue for me. One caveat, SPA is CPU intensive when it compiles the report, so if the system is already pegged at 100% its best to compile the report off the the system in question.

If you run into any issues with SPA (only supported on Win2k3), send me an e-mail or drop a comment and I'll try to help you out.

Technorati tags: Windows 2003, SPA

IceRocket tags: Windows 2003, SPA