Brad Rutkowski's Blog : Windows Server 2008

win32_processor and cim_processor CurrentClockSpeed shows lower value than actual processor speed

Brad Rutkowski — Thu, 16 Apr 2009 22:50:05 GMT

Was looking at an issue today where a whole bunch of our servers were showing clock speeds that didn’t match the max clock speed. These servers were showing up on our exBPA reports and thus landed in my lap to investigate. At first I assumed that something most be wrong with the report and so I logged onto a couple and sure enough they didn’t match the max speed:

PS C:\> get-wmiobject win32_processor | select=object currentclock*,max* | format-table -automatic

CurrentClockSpeed MaxClockSpeed
----------------- -------------
1999 2332
1999 2332

So with a bit of digging on the internet I found a page referencing the Enhanced Intel SpeedStep Technology:

SpeedStep is a trademark for a series of dynamic frequency scaling technologies (including SpeedStep, SpeedStep II, and SpeedStep III) built into some Intel microprocessors that allow the clock speed of the processor to be dynamically changed by software. This allows the processor to meet the instantaneous performance needs of the operation being performed, while minimizing power draw and heat dissipation.

Well that seemed promising, so I found out that you can turn this off in power options in the control panel, setting the minimum processor state to 100%:

PS C:\> gwmi win32_processor | select-object currentclock*,max* | ft -au

CurrentClockSpeed MaxClockSpeed
----------------- -------------
2332 2332
2332 2332

In the end I just turned it back on as this does save power and will dynamically increase to full usage when needed. Another mystery solved…

Technorati Tags: Windows 2008

Interacting with Data Collector Sets via Powershell

Brad Rutkowski — Thu, 19 Feb 2009 00:30:12 GMT

Background:

In an earlier post I talked about some new features for Windows 2008 and Vista. One of those new features that is often overlooked are the data collector sets (DCS). One particular role that leverages data collector sets is active directory. Active directory has put “hooks” into tracing that can really take a lot of the thinking out of the question “why is my domain controller sluggish”. For those of you still running Windows 2003 I go over a similar concept called Server Performance Advisor.

Anyways, you can play around with DCS by typing perfmon and then traversing to the section called Data Collector Sets (shocking). If you have performance issues, go here first as it’s like combining a netmon capture with a kernel trace and then handing you the smoking gun.

Challenge:

In my current role, we have a need to automate things quite a bit and so one of the actions I was looking at solving was collecting diagnostic information when a server is performing poorly. Usually when a high CPU alert comes in, someone would need to logon to the server and go to perfmon and start at DCS collection. More often is the case that by the time someone had been alerted and went to the server the sluggish behavior had subsided (the dreaded “close ticket, no problem found”).

My solution was to try and figure out a way to start a DCS collection remotely at the time of event so that the data was present when an actual human became engaged.

After some hard work, here is the code to do so! You can create your own XML file (your own DCS template) and pass it in, but more than likely you’ll be happy at just kicking off one of the built-in templates (AD/System Perf/System Diags).

Running it via powershell:

First, how to do it on the fly:

## PLA.dll lives under system32 on Vista and 2k8.  This will create a powershell com object.
$datacollectorset = new-object -COM Pla.DataCollectorSet
##This is the name of the predefined DCS collector.  It's read-only and will always be System\<something>
$name = "System\Active Directory Diagnostics"
##If you make the second param $null it will be the local machine.
$datacollectorset.Query($name,"serverA") 
$datacollectorset.start($false)
## Status ReturnCodes: 0=stopped 1=running 2=compiling 3=queued (legacy OS) 4=unknown (usually autologger)
$datacollectorset.status
##When you're ready to stop it call stop.
$datacollectorset.stop($false)
##If you call status here, it will probably be '2' for a while as the server compiles the report.
$datacollectorset.status

And like so, you started and stopped a collection for Active Directory on you’re local computer or a remote server! Like I said though, you can create you’re own templates too. You might want to do this if you want to setup a built-in template to be scheduled to run daily, or perhaps you want to send the data to a network location, run more tasks at completion, etc. If you do want to create a custom template then the code changes a bit:

$datacollectorset = new-object -COM Pla.DataCollectorSet
## If you're making you're own (shows up under user defined).  
$xml = get-content C:\custom.xml #You're custom exported XML file.
$datacollectorset.SetXml($xml)
##Commit codes: http://msdn.microsoft.com/en-us/library/aa371873(VS.85).aspx this is add or modify.  Can't do this on a system created PLA instances (read only).
$datacollectorset.Commit($DCSPath , $null , 0x0003)     
$datacollectorset.Query($DCSPath,$null)
$datacollectorset.start($false)
#Runs...
$datacollectorset.stop($false)

Scripting a solution:

Finally if you wanted to script this you could do something like what I’ve done below. This would collect for a desired interval (in seconds) and then when compilation completed display the path to the report. I wrote this in CTP3, but you can easily take the concepts and backport them. If the destination server is inaccessible, or you don't have permissions, then the script will blow up…

<#
    .SYNOPSIS
    This will fire up a PLA (Data Collector Set collection on a server and then copy it to the proper debug server
 
    .DESCRIPTION
    This is a proof of concept and only acceppts System defined collections.  No error handling so I hope you type well.

#>

##Inputs
[CmdletBinding()]
param(
   [Parameter(Mandatory = $true)]
   <#A system provided report to run like "System\System Performance", System\System Diagnostics, etc. #>
   [string]$DCSPath,
   [Parameter(Mandatory = $true)]
   <# This is how long you want the DCS collection to run in seconds#>
   [int32]$time,
   [Parameter(Mandatory = $false)]
   <#If you don't pass in a server name it will be $null and run on the local system#>
   [string]$serverName
    )

    $datacollectorset = new-object -COM Pla.DataCollectorSet  
    $datacollectorset.Query($DCSPath,$serverName)
    $datacollectorset.start($false)
    Start-Sleep $time
    $datacollectorset.stop($false)
    
    ##Now we'll loop while the report compiles.
    $retries = 0
    do 
        {sleep 30; $returnCode = $datacollectorset.Status ; $retries++} 
    while ($returnCode -eq 2 -and $retries -lt 60)
    
    if ($retries -eq 60)
    {
        Write-Warning "Compiling has been running on the server for 30 minutes!  You'll need to check the following location on the server later for the report:"
        Write-Warning $datacollectorset.OutputLocation
        break
    }
    
    ##Compiling has finished, now we can copy the folder to some location
    $path = $datacollectorset.OutputLocation
    if ($serverName)
    {
    $path = $path.Replace(":","$")
    Write-Host "`nReport complete and can be viewed at \\$serverName\$path\report.html on the server.`n" 
    }
    else
    {
    Write-Host "`nReport complete and can be viewed at $path\report.html`n"
    }

The result:

More info:

PLA reference: http://msdn.microsoft.com/en-us/library/aa372634(VS.85).aspx

Technorati Tags: Powershell,Windows 2008,Active Directory,Windows

Display warning text when someone logs onto your servers

Brad Rutkowski — Thu, 25 Sep 2008 03:25:00 GMT

This works for Windows 2003 and Windows 2008. We use it during our reliability study to let the server owners know that they shouldn't reboot their boxes without a good reason. You can use it for whatever you’d like. :)

The two keys to set:

reg add "\\brad-dc-01\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeCaption /t REG_SZ /d "MSIT Reliability Study" /f

reg add "\\brad-dc-01\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeText /t REG_SZ /d "This server is part of the MSIT Windows 7 Reliability Study. The server should not be rebooted. If the server is experiencing a bug, please contact DCOPERATE to triage and they will escalate as needed. If you are rebooting the server for a hotfix, private fix, or other legitimate reason, please document it properly in the shutdown tracker so that the statistics are accurate." /f

Hop to loop it and apply it en masse:

Open CMD with your alt creds and do the following:

C:\Windows\system32>for /f %a in (machines.txt) do (

More? reg add "\\%a\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeCaption /t REG_SZ /d "MSIT Reliability Study" /f

More? reg add "\\%a\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeText /t REG_SZ /d "This server is part of the... (HUGE LONG STRING) ... " /f

More? )

How to turn it off:

C:\Windows\system32>for /f %a in (machines.txt) do (

More? reg add "\\%a\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeCaption /t REG_SZ /d "" /f

More? reg add "\\%a\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeText /t REG_SZ /d "" /f

More? )

The result:

Another way of doing this is to set "Interactive logon: Message text for users attempting to logon" in secpol.msc...

Technorati Tags: Windows 2003,Windows 2008

Getting Access Denied when trying to query root\MSCluster namespace remotely against Windows 2008.

Brad Rutkowski — Mon, 08 Sep 2008 21:00:56 GMT

Ran into a weird issue where I was getting access denied when trying to query nodes remotely in powershell. The query was working fine against Windows 2003 cluster names and worked locally when ran on a Windows 2008 cluster node, it just didn’t work remotely.

Against 2k3:

PS C:\Debuggers> gwmi -q "Select name from MSCluster_Node" -namespace root\mscluster -computername Server-2k3-01 | Select-Object Name

Name
----
Server-2k3-01
Server-2k3-02

Against 2k8:

PS C:\Debuggers> gwmi -q "Select name from MSCluster_Node" -namespace root\mscluster -computername Server-2k8-01
Get-WmiObject : Access denied
At line:1 char:5
+ gwmi <<<< -q "Select name from MSCluster_Node" -namespace root\mscluster -computername Server-2k8-01

I also tried the query outside of powershell to eliminate that form the equation with the same results and it still failed. So why the difference? Well looking around on the target, I noticed this event in the event log:

Log Name:      Application
Source:        Microsoft-Windows-WMI
Date:          9/5/2008 10:17:52 AM
Event ID:      5605
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Server-2k8-01
Description:
Access to the root\mscluster namespace was denied because the namespace is marked with RequiresEncryption but the script or application attempted to connect to this namespace with an authentication level below Pkt_Privacy. Change the authentication level to Pkt_Privacy and run the script or application again.

Doing a little research I ran across this article explaining the event and what needs to happen to run the query properly:

http://technet.microsoft.com/en-us/library/cc727103.aspx

In VBScript that means adding: authenticationLevel=pktPrivacy to your query. In Powershell (I’m using 2.0) you just add the authentication switch to get it to work. Now the query works on downlevel as well as 2k8:

PS C:\Debuggers> gwmi -q "Select name from MSCluster_Node" -namespace root\mscluster -computername Server-2k8-01 -Authentication PacketPrivacy | Select-Object Name

Name
----

Server-2k8-01
Server-2k8-02
Server-2k8-03
Server-2k8-04
Server-2k8-05

PostScript:

You can do a whole bunch of cool stuff with powershell check it out! Here’s just a little query to tell me each node and ‘t state:

PS C:\Debuggers> gwmi -q "Select * from MSCluster_Node" -namespace root\mscluster -computername TK5-CLUS-01 -Authentication PacketPrivacy | Select-Object Name,State | Format-Table -au

Name           State
----              -----
tk5-clus-01     0
tk5-clus-02     0
tk5-clus-03     0
tk5-clus-04     1
tk5-clus-05     0
tk5-clus-06     0

Domain doesn't know about my computer account? I vouch for my computer, you can trust me...

Brad Rutkowski — Fri, 01 Aug 2008 22:31:11 GMT

Had an issue where a server would not allow logon via termian services each time you attempted to logon it would return this:

Soooooooooo, what to do here?

First, we made sure the account existed in the directory since that's why it appeared to be complaining. So I opened LDP and verified it existed, and that all "checked out" with being healthy (stare and compare against a good object).

Second thing we did was crank up netlogon debug logging (nltest dbflag) and see what it showed. It was complaining of a lot of stuff but nothing conclusive unfortunately. So at that point it was time to move to event viewer. The "nice" thing about this issue was that the server was accessible via the network with the same account that was failing to TS so I could do some of the investigation remotely.

One event in particular struck me:

Log Name:      System
Source:        Microsoft-Windows-Security-Kerberos
Date:          7/31/2008 4:11:24 PM
Event ID:      3
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      BRAD-SRV-01.braddom.bradforest.com
Description:
A Kerberos Error Message was received:
on logon session
Client Time:
Server Time: 23:11:24.0000 7/31/2008 Z
Error Code: 0x7 KDC_ERR_S_PRINCIPAL_UNKNOWN
Extended Error: 0xc0000035 KLIN(0)
Client Realm:
Client Name:
Server Realm: braddom.bradforest.COM
Server Name: host/BRAD-SRV-01.braddom.bradforest.com
Target Name: host/BRAD-SRV-01.braddom.bradforest.com@braddom.bradforest.COM
Error Text:
File: 9
Line: d86
Error Data is in record data.

Using err.exe I resolved the error code and found there was a collision:

C:\localbin>err 0xc0000035
# for hex 0xc0000035 / decimal -1073741771 :
STATUS_OBJECT_NAME_COLLISION ntstatus.h
# Object Name already exists.
# 1 matches found for "0xc0000035"

At this point it's time to look for a collision of "host/BRAD-SRV-01.braddom.bradforest.com" in the forest. The easiest way to do it is use a nice script called querySPN.vbs.

C:\localbin>querySPN.vbs HOST/BRAD-SRV-01.braddom.bradforest.com braddom.bradforest.com
Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.

CN=VL Account,CN=Users,DC=braddom,DC=bradforest,DC=com
Class: user
User Logon: VLSBST
-- host/BRAD-SRV-01.braddom.bradforest.com <----------------------------------------------------------------- Bingo the SPN is registered for two objects!

CN=BRAD-SRV-01,CN=Computers,DC=braddom,DC=bradforest,DC=com
Class: computer
Computer DNS: BRAD-SRV-01.braddom.bradforest.com
-- TERMSRV/BRAD-SRV-01.braddom.bradforest.com
-- TERMSRV/BRAD-SRV-01
-- HOST/BRAD-SRV-01
-- HOST/BRAD-SRV-01.braddom.bradforest.com <-----------------------------------------------------------------

Once we removed the SPN from the user account, logons began to immediately work.

-B

Windows Update fails with 8000FFFF (E_UNEXPECTED)

Brad Rutkowski — Thu, 03 Jul 2008 22:07:02 GMT

Quick Solution: Check the permissions on the root of C: and ensure that BUILTIN\Users have Read access.

Long Story:

8000FFFF == E_UNEXPECTED, not very helpful…

Had a client where windows update was continually failing with the error code 8000FFFF. When looking in the Windows Update log we’d see errors like this:

WARNING: PTError: 0x80248014
Handler FATAL: CBS called Error with 0x8000ffff, <— Checked the CBS.log file but that didn’t give any clues.
Handler FATAL: Error source is 106.
DnldMgr Error 0x8000ffff occurred while downloading update; notifying dependent calls.
AU        # WARNING: Download failed, error = 0x8000FFFF
AU        # WARNING: Download failed, error = 0x8000FFFF
AU      WARNING: BeginInteractiveInstall failed, error = 0x8024000C
CltUI   WARNING: AU directive Interactive Progress is exiting due to error 8024000C

And in the event viewer upon each run we’d see these events:

Log Name:      Application
Source:        ESENT
Date:          7/2/2008 3:05:16 PM
Event ID:      491
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      XXXX
Description:
Catalog Database (1560) Catalog Database: An attempt to determine the minimum I/O block size for the volume "C:\" containing "C:\Windows\system32\CatRoot2\" failed with system error 5 (0x00000005): "Access is denied. ". The operation will fail with error -1032 (0xfffffbf8).

Log Name:      Application
Source:        Microsoft-Windows-CAPI2
Date:          7/2/2008 3:05:16 PM
Event ID:      257
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      XXXX
Description:
The Cryptographic Services service failed to initialize the Catalog Database. The ESENT error was: -1032.

After seeing this data I did a stare and compare between my root permissions and his and found that he’d modified the c:\ permissions on his system:

His machine:
c:\temp\xcacls c:
C:\ NT AUTHORITY\SYSTEM:(OI)(CI)F
BUILTIN\Administrators:(OI)(CI)F

Mine:
C:\>xcacls c:\
c:\ BUILTIN\Administrators:F
    BUILTIN\Administrators:(OI)(CI)(IO)F
    NT AUTHORITY\SYSTEM:F
    NT AUTHORITY\SYSTEM:(OI)(CI)(IO)F
    BUILTIN\Users:(OI)(CI)R <— This is the key one missing that was causing the headache.
    NT AUTHORITY\Authenticated Users:(OI)(CI)(IO)C
    NT AUTHORITY\Authenticated Users:(special access:)
                                     FILE_APPEND_DATA

The Cryptographic Services runs under “Network Service” which would require Users to have read access. I added BUILTIN\Users with read access to C and all worked again.

Hopefully this post will guide others with similar issues to the solution quickly.

Technorati Tags: Vista,Windows Update,WSUS,Windows 2008

Staring at a blank desktop, due to Interactive missing from Users group

Brad Rutkowski — Fri, 30 May 2008 01:51:36 GMT

Ran into an issue this week that was strange. When you TS’d to the box it would just show a blank background and nothing else. If you tried to launch task manager it would just fail silently to the user (actually access denied in the debugger). My user account was in the admin group and the server was completely accessible remotely with administrative perms. It was just when I (or anyone) tried to logon to the server locally or through TS that it was messed up. Another piece of the puzzle was that if you disabled UAC and rebooted the server the issue no longer repro’d.

So what was there with UAC and logging onto this server?

When logging on this event was triggered:

Log Name:      Application
Source:        Microsoft-Windows-Winlogon
Date:          5/27/2008 5:13:28 PM
Event ID:      4006
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      XXXX
Description:
The Windows logon process has failed to spawn a user application. Application name: . Command line parameters: C:\Windows\system32\userinit.exe.

Turns out that they removed the Account "NT AUTHORITY\INTERACTIVE" from the Users group on the machine. We added that account back into the users group and like magic it worked again. I'm working on getting a KB filed and written for this issue, but until then at least people can find it if they notice this event in the event log.

Reference:

http://technet2.microsoft.com/WindowsVista/en/library/00d04415-2b2f-422c-b70e-b18ff918c2811033.mspx?mfr=true

UAC Architecture

While the Windows Vista logon process externally appears to be the same as the logon process in Windows XP, the internal mechanics have greatly changed. The following illustration details how the logon process for an administrator differs from the logon process for a standard user.

Windows Vista logon process

When an administrator logs on, the user is granted two access tokens: a full administrator access token and a "filtered" standard user access token. By default, when a member of the local Administrators group logs on, the administrative Windows privileges are disabled and elevated user rights are removed, resulting in the standard user access token. The standard user access token is then used to launch the desktop (Explorer.exe).

HatTip to Ben on my Team who actually figured this out after I tried to debug it for 3 days...

Technorati Tags: Vista,Windows 2008,UAC,Winlogon

Stuff to check out for Windows 2008

Brad Rutkowski — Wed, 26 Mar 2008 19:41:25 GMT

Just got released yesterday:

X86: http://www.microsoft.com/downloads/details.aspx?FamilyID=9ff6e897-23ce-4a36-b7fc-d52065de9960&DisplayLang=en

X64: http://www.microsoft.com/downloads/details.aspx?FamilyID=d647a60b-63fd-4ac5-9243-bd3c497d2bc5&DisplayLang=en

Overview

Microsoft Remote Server Administration Tools (RSAT) enables IT administrators to remotely manage roles and features in Windows Server 2008 from a computer running Windows Vista with SP1. It includes support for remote management of computers running either a Server Core installation or the full installation option of Windows Server 2008. It provides similar functionality to Windows Server 2003 Administration Tools Pack.
After you install this item, you may have to restart your computer. This update is provided to you and licensed under the Windows Vista License Terms.

Once you install the KB, you need to enable the RSAT tools by doing the following:

1. Click Start, click Control Panel, and then click Programs.

2. In the Programs and Features area, click Turn Windows features on or off.

3. If you are prompted by User Account Control to allow the Windows Features dialog box to open, click Continue.

4. In the Windows Features dialog box, expand Remote Server Administration Tools.

5. Select the remote management tools that you want to install.

6. Click OK.

Other notables:

1) Windows Server® 2008 Network Shell (Netsh) Technical Reference What can you do in Netsh in Win2k8? How do I add IP info? How do I adjust the firewall? How do I connect to a remote server via netsh? You get the point.

2) Active Directory Database Mounting Tool Screencast Great screen cast on how to take a snapshot of your DIT, mount it, and view an offline copy via dsa.msc

3) IIS7 Media Pack Bit Rate Throttling Module For media files, Bit Rate Throttling implements a dynamic per-file throttling capability to provide intelligent progressive downloading.

Technorati Tags: RSAT,Windows 2008,Vista

2 things: Tell if a server is server core remotely. Tell if a server is a VM remotely. (C#)

Brad Rutkowski — Fri, 07 Mar 2008 23:54:49 GMT

I've been messing around over the last week making a tool that will frisk a remote machine. It's been a fun project, a couple of items I got hung up on were if the machine was server core and if it was a VM.

I mean who would knowingly TS to a server if they knew it was server core? As for the VM, it's nice to know before-hand so you dont request a debugger to be attached to a virtual server ;).

Well here are some snippets for those two things, hope it helps those trying to do similar queries...

Server Core:

Basically you just need to look at the OperatingSystemSKU value and if it E (hex) or 14 (decimal) then its server core. This and all the other SKU numbers are listed here: http://msdn2.microsoft.com/en-us/library/ms724358.aspx

System.Management.ConnectionOptions objconn = new System.Management.ConnectionOptions();
        objconn.Impersonation = System.Management.ImpersonationLevel.Impersonate;
        objconn.EnablePrivileges = true;
        System.Management.ManagementScope exmangescope = new System.Management.ManagementScope(@"\\" + srvName + @"\root\cimv2", objconn);
        System.Management.ObjectQuery objquery = new System.Management.ObjectQuery("SELECT * FROM Win32_OperatingSystem");
        System.Management.ManagementObjectSearcher objsearch = new System.Management.ManagementObjectSearcher(exmangescope, objquery);
        System.Management.ManagementObjectCollection queryCollection = objsearch.Get();
        foreach (System.Management.ManagementObject stringer in queryCollection)
        {
            serverCoreval = stringer["OperatingSystemSKU"].ToString();
            //Console.WriteLine(serverCoreval);
        }

Virtual Machine:

If the VM is either Vista/Windows 2008 it's a simple reg query:

if (buildInt >= 6000)
{
sysInfo = RegistryKey.OpenRemoteBaseKey(RegistryHive.LocalMachine, srvName).OpenSubKey(@"SYSTEM\CurrentControlSet\Control\SystemInformation").GetValue("SystemProductName").ToString();

    if (sysInfo.Contains("Virtual"))
    {
        vmCheck = 1;
    }

}

If the VM is downlevel then it's a WMI query

else if (buildInt == 3790)
{
    System.Management.ConnectionOptions objconn = new System.Management.ConnectionOptions();
    objconn.Impersonation = System.Management.ImpersonationLevel.Impersonate;
    objconn.EnablePrivileges = true;
    System.Management.ManagementScope exmangescope = new System.Management.ManagementScope(@"\\" + srvName + @"\root\cimv2", objconn);
    System.Management.ObjectQuery objquery = new System.Management.ObjectQuery("SELECT * FROM Win32_ComputerSystem");
    System.Management.ManagementObjectSearcher objsearch = new System.Management.ManagementObjectSearcher(exmangescope, objquery);
    System.Management.ManagementObjectCollection queryCollection1 = objsearch.Get();
    foreach (System.Management.ManagementObject stringer in queryCollection1)
    {
        sysInfo = stringer["Model"].ToString();
        //System.Console.WriteLine(sysinfo);
    }
    if (sysInfo.Contains("Virtual"))
    {
        vmCheck = 1;
    }

Technorati Tags: windows 2008,Server Core,C#,Virtualization

Tidbits for admins for the 2k8 release...

Brad Rutkowski — Tue, 26 Feb 2008 19:00:00 GMT

Just some random stuff as you get ready for 2k8...

Getting the Classic cluster logs:

Clustering in Win2k8 has undergone some major changes (for the better). One of those changes is that the cluster events are now part of the event stream so sifting through the cluster logs is a thing of the past. You might find it easier sometimes though to have the cluster logs in which case you can generate them:

C:\>cluster log /G /Copy:"c:\debuggers"
Generating the cluster log(s) ...
The cluster log has been successfully generated on node 'server-10'...
The cluster log has been successfully generated on node 'server-11'...
The cluster log has been successfully copied from node 'server-11'...
The cluster log has been successfully copied from node 'server-10'...
The cluster log has been successfully generated on node 'server-15'...
The cluster log has been successfully copied from node 'server-15'...
The cluster log has been successfully generated on node 'server-16'...
The cluster log has been successfully copied from node 'server-16'...

The cluster log(s) have been copied to 'c:\debuggers'...

Multiple TS connections to the same server with the same account:

You may notice that in Win2k8 that if you are already logged on via TS to a server and use the same account from a different machine to connect to the server it will take over the session you already have connected instead of creating a new one. This is by default in 2k8. If you/your team use a test account to logon to your servers this could be quite annoying and you might want to set it back to what it was like in 2k3. You can do this by unchecking "Restrict each user to a single session" in tsconfg.msc, which just toggles the fSingleSessionPerUser value to zero under "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server" if you want to do it remotely.

Also, I already mentioned it but you need to use the /admin switch to connect to the console session with 2k8 and Vista SP1. More info from Terminal Services Team Blog.

Getting the system info for investigations:

Have a customer who is having issues? Stop asking questions and have them run msinfo32.exe /nfo c:\test.nfo and send you the test.nfo file. What's in there? Everything of your dreams. No really, it has a plethora of information on the system where its taken, and is quite helpful. If you just want to grab the basics from a server locally/remotely use systeminfo.exe which is under system32.

Setup failed and I do not know why:

For general troubleshooting, check the Setupact.log and Setuperr.log files. Depending on when the installation failed, these files will be located in the $WINDOWS.~BT\Sources\Panther folder or the Windows\Panther folder. In most cases, these folders are located on the partition that Windows Server 2008 is being installed on or the partition that contains the old operating system. However, if Setup failed on an Itanium-based computer, this folder might be located on another drive that has available hard disk space. From here.

I'd also add if you dont find any info in the panther log locations check the cbs.log file under %windir%\Logs\CBS. This has good information for any setup/install failures.

Installing Win2k8 and using it as your desktop:

For the uber-nerds cough *not me* cough: http://blogs.msdn.com/vijaysk/archive/2008/02/11/using-windows-server-2008-as-a-super-desktop-os.aspx

Windows 2008 is fast as hell, and if you got the horses you might think this is a good idea. IF you can live without sidebar! Oh wait, does anyone use that?

Microsoft Assessment and Planning (MAP) released yesterday:

Finally for those of you who want to scan your hardware inventory with zero-touch, the Solution Accelerator for 2k8 went out the door yesterday. If anything you should take the link and check it out.

The Microsoft Assessment and Planning Solution Accelerator performs three key functions - including hardware and device inventory, compatibility analysis, and readiness reporting.

Technorati Tags: Windows 2008

Taking a circular netmon capture from the command prompt

Brad Rutkowski — Fri, 22 Feb 2008 22:23:42 GMT

You've probably heard that netmon3.1 is out, but you might not know that you can easily launch a capture at the command prompt. I find this useful when we're waiting on a repro, we want a capture, but we don’t know when that's going to happen. Sure you could set this up in the GUI too, but who wants to do that when it's as easy as this?

The below will setup a capture on all networks that the system is attached to and wait until I hit ctrl+c (you can see its been a while with no repro). The CHN extension used tells netmon to take multiple captures in a chain (see file syntax). I also put some examples at the bottom so you can see what else you can do. Have fun!

C:\Windows\system32>nmcap /capture /network * /File netmoncap.chn:100M
Netmon Command Line Capture (nmcap) 03.01.0512.0000

Saving info to:
C:\Windows\system32\netmoncap.cap - using chain captures of size 100.00 MB.

ATTENTION: Conversations Enabled: consumes more memory (see Help for details)

Exit by Ctrl+C

Saved Frames: 9232127 Capture Frames: 9438779 (44181 seconds)

Hit Ctrl+C

Cancelled by user

Final Results : Saved Frames: 722 Capture Frames: 722

C:\Program Files\Microsoft Network Monitor 3>dir netmoncap.cap
Volume in drive C has no label.
Volume Serial Number is FCC3-5AF7

Directory of C:\Program Files\Microsoft Network Monitor 3

02/22/2008 09:06 AM           384,748 netmoncap.cap
               1 File(s)        384,748 bytes
               0 Dir(s) 16,699,654,144 bytes free

Here's the breakdown fo the /File syntax:

/File <Capture File>[:<File Size Limit>]
    Name of capture file to save frames to. Extensions are used to determine
    the behavior of nmcap.
     .cap -- Netmon 2 capture file
     .chn -- Series of Netmon 2 capture files: t.cap, t(1).cap, t(2).cap...
    <File Size Limit> are optional. It limits the file size of each capture
    file generated. Default single capture file size limit is 20M. The
    upper bound of the file size limit is 500M. The lower bound of the file
    size limit depends on the frame size captured. (Note that the maximal size
    of ethernet frames is 1500 Bytes)
    The files are circular, so once the size limit is reached, new data will
    overwrite older data.
    Example Usage: /File t.cap:50M

Some other examples from the NMCAP help:

This example starts capturing network frames that DO NOT contain ARPs, ICMP,
NBtNs and BROWSER frames. If you want to stop capturing, Press Control+C.

nmcap /network * /capture (!ARP AND !ICMP AND !NBTNS AND !BROWSER) /File NoNoise.cap

Starts capturing network frames immediately. All TCP frames that have a source
port or destination port of 80 are saved to the chained capture files named
test.cap, test(1).cap, test(2).cap, ... When the user presses the 'x' key the
program stops.

nmcap /network * /capture tcp.port == 80 /file c:\temp\test.chn:6M /stopwhen /keypress x

This example starts capturing network frames that are TCP Continuations. The
capture filter is searching for String "Continuation in TCP Frame Summary
Description. In order to see the complete list of Netmon Properties that are
filterable,type ".Property" in the Netmon Filter UI.

nmcap /network * /capture contains(.Property.Description, \"Continuation\") /File TCPContinuations.cap

Technorati Tags: Netmon,Network Monitor

I PTE the fool: !SYSPTES 4 works in Vista SP1/WS08

Brad Rutkowski — Thu, 21 Feb 2008 03:21:00 GMT

System Page Table Entry (PTE) issues are some of the top support issues for servers that run large server applications and have a relatively large amount of Random Access Memory (RAM). PTEs are structures used to track pages of RAM, similar to the way a telephone number is used to track a telephone to a specific location.

You can now track down those 3f bugchecks using !sysptes using the public symbols. Usually when we hit a server running out of system PTEs, it will just tip over and we don't see an actual bugcheck.

Prior to Vista SP1/Windows 2008 if you tried to run SYSPTES 4 on a server with public symbols you'd get this error message: "Unable to get System PTE individual lock consumer information". Well Windows Dev has fixed the bug. Below is an example of what we would typically see, and then how we'd use !sysptes to narrow down who is consuming the space.

You find this sort of output in !VM:

0: kd> !vm 1
*** Virtual Memory Usage ***
Physical Memory: 999242 ( 3996968 Kb)
Page File: \??\C:\pagefile.sys
Current: 927744 Kb Free Space: 884312 Kb
Minimum: 927744 Kb Maximum: 927744 Kb
Page File: \??\E:\pagefile.sys
Current: 3072000 Kb Free Space: 3024624 Kb
Minimum: 3072000 Kb Maximum: 3072000 Kb
Available Pages: 265887 ( 1063548 Kb)
ResAvail Pages: 933615 ( 3734460 Kb)
Locked IO Pages: 1679 ( 6716 Kb)
Free System PTEs: 500 ( 2000 Kb)

********** Running out of system PTEs **************

******* 416179544 system PTE allocations have failed ******

Free NP PTEs: 1630 ( 6520 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 478 ( 1912 Kb)
Modified PF Pages: 477 ( 1908 Kb)
NonPagedPool Usage: 8814 ( 35256 Kb)
NonPagedPool Max: 32351 ( 129404 Kb)
PagedPool 0 Usage: 10590 ( 42360 Kb)
PagedPool 1 Usage: 994 ( 3976 Kb)
PagedPool 2 Usage: 958 ( 3832 Kb)
PagedPool 3 Usage: 972 ( 3888 Kb)
PagedPool 4 Usage: 931 ( 3724 Kb)
PagedPool Usage: 14445 ( 57780 Kb)
PagedPool Maximum: 54784 ( 219136 Kb)
Shared Commit: 4163 ( 16652 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 10930 ( 43720 Kb)
PagedPool Commit: 14485 ( 57940 Kb)
Driver Commit: 1963 ( 7852 Kb)
Committed pages: 747541 ( 2990164 Kb)
Commit limit: 1952440 ( 7809760 Kb)

So what now? If you don't have the registry value set below, well for all intents and purposes you SOL. So reboot, set the value, and then wait for repro. Usually when we see the issue it comes back pretty quickly as some driver is eating up the space.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
trackptes REG_DWORD 0x1

Once the waiting is over and the system tips over, run !sysptes 4 and it will tell you what is allocating the PTEs and how many per call. The “!SYSPTES 4” command only lists driver PTE allocations. This is because, historically, drivers have made the most use and misuse of system PTEs. Sometimes you'll find one heavy hitter with a huge count (like in the article linked below), or in other instances you might find a certain sequence allocating many times, in either case you now have a clue as to who is using the PTEs and can either investigate that driver via break points, or contact the vendor who is eating up all the PTEs.

Cleaned up a bit for sanity's sake:

2: kd>!sysptes 4

VA MDL PageCount Caller/CallersCaller

f0769080 fce7fb18 2         mrxsmb+0x2bed2/mrxsmb+0x2da71
efbbb8b8 fce0f658 2         rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
f1c17080 fd0eb7a8 2         mrxsmb+0x2bed2/mrxsmb+0x2da71
eff41820 fd41bc70 2         rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
f1d10080 fcd91950 2         mrxsmb+0x2bed2/mrxsmb+0x2da71
f027f108 fd051f88 2         rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
efbf7080 fd7f3e80 2         mrxsmb+0x2bed2/mrxsmb+0x2da71
f1cede10 fce71460 2         rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
ef8a8080 fcedde80 2         mrxsmb+0x2bed2/mrxsmb+0x2da71
ef963730 fc9c2868 2         rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
f0281080 fccc52c0 2         mrxsmb+0x2bed2/mrxsmb+0x2da71
f1dfaff8 fd156650 2         rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
f0141080 fc6e82c0 2         mrxsmb+0x2bed2/mrxsmb+0x2da71
ef8f3508 fd003a30 2         rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
ef637080 fee1dde0 2         mrxsmb+0x2bed2/mrxsmb+0x2da71
eff3fa18 fd304050 2         rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b

The actual process of tracking down the PTEs is thoroughly explained here: Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues

So next time your system is acting up, get in there with Live KD and see what's going on!

Hey Admins! Taking some of the pain out of analyzing perfmon captures.

Brad Rutkowski — Thu, 14 Feb 2008 02:03:20 GMT

Performance Analysis of Logs (PAL) tool

Project Description:

Ever have a performance problem, but don't know what performance counters to collect or how to analyze them? The PAL (Performance Analysis of Logs) tool is a new and powerful tool that reads in a performance monitor counter log (any known format) and analyzes it using complex, but known thresholds (provided). The tool generates an HTML based report which graphically charts important performance counters and throws alerts when thresholds are exceeded. The thresholds are originally based on thresholds defined by the Microsoft product teams and members of Microsoft support, but continue to be expanded by this ongoing project. This tool is not a replacement of traditional performance analysis, but it automates the analysis of performance counter logs enough to save you time. This is a VBScript and requires Microsoft LogParser (free download).

My take on the tool:

For those of us out there that don't have to deal with performance data on a daily basis I see a few options to help troubleshoot performance issues on your servers. 1) If your using 2k3 use SPA. 2) If you're running 2k8/Vista use data collection sets. 3) Collect analyze your own perfmon captures.

Now you might want to look into this tool. I found the tool simple to use and it's really a four step process. The web page created for the analysis has a plethora of info and links to the codeplex site for more info. Sweet.

Really in the end it's just a time saver. After collecting performance data on a server you need to analyze that data. This entails opening the log file, adding the counters that you've collected and finding out if any of the counters are above any thresholds (deemed by you). This tool does that analysis for you. It comes out-of-the-box with some predefined thresholds defined as high according to the MSFT consulting/development but those can be adjusted to whatever suits your fancy.

Once you get everything installed its time to do some analysis. It comes with some threshold templates for AD, System Overview, IIS, SQL, Exchange, etc (see pic) You point the app at the performance log you've captured during your perf issue, choose a threshold template to your liking, answer some basic questions, add the form and execute:

Once it completes it generates a webpage with the analysis information you desire. The webpage shows you alerts for activity that it finds suspect and graphs for the different areas of interest. I can't paste all the pics/info in here as it is quite lengthy depending on the interval you provide. But this definitely seems like a tool that could be handy down the road. Looking at the web page it looks really similar to SPA, but with graphs provided via the Office Web Components add-in. For example here is how I could find out LDP was using too much CPU:

First I found the alert which said that something was being excessive and I clicked on the link (sorry for the blurriness):

Then I found LDP consuming the CPU:

Add it to your bag of tricks, hope it helps.

Technorati Tags: Windows 2003,Windows 2008,Perfmon,Performance

Some docs to help get you ready for Windows 2008.

Brad Rutkowski — Tue, 12 Feb 2008 01:23:40 GMT

Yes you are going to want Windows 2008 in your shop. We've been running 2k8 for over two year sin production and I'm very proud of the product we're shipping shortly. Some of you will want to start thinking about what's new in the OS, what has changed, performance, and deployment. Here are some docs/links that will help get you started.

Changes in Functionality from Windows Server 2003 with SP1 to Windows Server 2008

This document describes new features and technologies, which were not available in Windows Server 2003 with Service Pack 1 (SP1), that will help to increase the security of computers running Windows Server 2008, increase productivity, and reduce administrative overhead.
These topics apply to the next release of Windows Server 2008, based on the functionality expected to be included in the Beta releases in 2007. They do not describe all of the changes that are included in Windows Server 2008. Instead, they highlight changes that will potentially have the greatest impact on your use of Windows Server 2008 and provide references to additional information.

Performance Tuning Guidelines for Windows Server 2008

•Performance Tuning for Server Hardware

•Performance Tuning for Networking Subsystem

•Performance Tuning for Storage Subsystem

•Performance Tuning for Web Servers

•Performance Tuning for File Servers

•Performance Tuning for Active Directory Servers

•Performance Tuning for Terminal Server

•Performance Tuning for Terminal Server Gateway

•Performance Tuning for File Server Workload (NetBench)

•Performance Tuning for Network Workload (NTttcp)

•Performance Tuning for Terminal Server Knowledge Worker Workload

•Performance Tuning for SAP Sales and Distribution Two-Tier Workload

Windows Server 2008 Step-by-Step Guides (updated 02/08/2008)

All other inquiries head to the Technical Library for 2k8.

BTW when you install 2k8 and notice that in WinVER or System Properties it shows SP1, don't be alarmed this is by design. Vista SP1 and Win2k8 were developed in parallel, so fixes in the code were included in both versions. This isn't the first time, when we rolled Windows 2003 x64 edition out the door it went as SP1 too...

Technorati Tags: windows 2008

Does my CPU support hardware virtualization (Hyper-V)

Brad Rutkowski — Sat, 26 Jan 2008 08:37:00 GMT

As the talk about hardware virtualization heats up from Microsoft and others you might find yourself wondering if the current hardware you're running supports it. My HP xw9300 workstation doesn't (older AMD Opterons), but it looks like my xw8400 does (Intel Xeon 5150s).

More info on programs you can use (including the one above) found here: http://blogs.msdn.com/volkerw/archive/2007/05/21/hardware-virtualization-check-utility.aspx

Determine Virtualization Readiness in 3 Seconds

Some other tidbits of info:

Hyper-V requires to be running on a 64bit OS, so do'nt install the x86 version of Wink28 if you want to use it.
Hyper-V (Virtual Server) is a server role in Win2k8, you just add it in server manager

Buy the addition with Hyper-V if you want to use it

You can have 64bit VMs, I threw on Win2k3 x64 and Win2k8 x64 on my box.
Your VMs can see more than one logical proc.

At the time of this post the VM had to be Win2k8 as well, downlevel can only see one proc. My x64 Win2k8 VM can see four procs now!

Have to enable Hardware virtualization in the BIOS as well as DEP, the role will still install but when you try to start the VM you'll hit a problem.
To install the additions the VM has to be running WIn2k3 SP2, if not you might find yourself without a NIC until you get the additions installed. You can add a legacy NIC if you find yourself in this situation.
If you just import an old VM VHD from one of your other servers, make sure its an ACPI-compatible one.
If you are looking to install hyper-v on a server running active directory, stop. You won't be able to boot your DC anymore (see below)

If you install the Active Directory Domain Services role and use the Active Directory Domain Services Installation Wizard (dcpromo.exe) to configure that role on the same physical computer on which the Hyper-V role is installed, you will receive a STOP error message 7B on the physical computer when you try to start a virtual machine.

To avoid this issue, do not install Active Directory Domain Services and Hyper-V on the same physical computer.

Microsoft Hyper-V site to find out more info if you like.

Update: Looks like AMD has released a tool help out here: http://blogs.msdn.com/virtual_pc_guy/archive/2008/03/31/amd-releases-hyper-v-check-tool.aspx

Technorati Tags: Hyper-V,Windows 2008,Virtualization