Was looking at an issue today where a whole bunch of our servers were showing clock speeds that didn’t match the max clock speed. These servers were showing up on our exBPA reports and thus landed in my lap to investigate. At first I assumed that something most be wrong with the report and so I logged onto a couple and sure enough they didn’t match the max speed:
PS C:\> get-wmiobject win32_processor | select=object currentclock*,max* | format-table -automatic
CurrentClockSpeed MaxClockSpeed
----------------- -------------
1999 2332
1999 2332
So with a bit of digging on the internet I found a page referencing the Enhanced Intel SpeedStep Technology:
SpeedStep is a trademark for a series of dynamic frequency scaling technologies (including SpeedStep, SpeedStep II, and SpeedStep III) built into some Intel microprocessors that allow the clock speed of the processor to be dynamically changed by software. This allows the processor to meet the instantaneous performance needs of the operation being performed, while minimizing power draw and heat dissipation.
Well that seemed promising, so I found out that you can turn this off in power options in the control panel, setting the minimum processor state to 100%:
PS C:\> gwmi win32_processor | select-object currentclock*,max* | ft -au
CurrentClockSpeed MaxClockSpeed
----------------- -------------
2332 2332
2332 2332
In the end I just turned it back on as this does save power and will dynamically increase to full usage when needed. Another mystery solved…
Technorati Tags:
Windows 2008
Thought I’d doc this for any others who run into this issue. I had to demote/promote a machine this morning and when it finished promoting I found it was missing all the NTDS\* counters in perfmon.
I ran LODCTR /Q and saw that it looked wrong:
C:\Windows\system32>lodctr /q:NTDS
Performance Counter ID Queries [PERFLIB]:
Base Index: 0x00000737 (1847)
Last Counter Text ID: 0x00001792 (6034)
Last Help Text ID: 0x00001793 (6035)
Looking into it further I found it was missing the whole performance key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Performance
Once I found that, I exported the performance registry key from another domain controller and imported to the server missing the values and ran LODCTR /R and then the counters were back where they belonged…
C:\Windows\system32>lodctr /q:NTDS
Performance Counter ID Queries [PERFLIB]:
Base Index: 0x00000737 (1847)
Last Counter Text ID: 0x00001794 (6036)
Last Help Text ID: 0x00001795 (6037)
[NTDS] Performance Counters (Enabled)
DLL Name: %systemroot%\system32\ntdsperf.dll
Open Procedure: OpenNtdsPerformanceData
Collect Procedure: CollectNtdsPerformanceData
Close Procedure: CloseNtdsPerformanceData
First Counter ID: 0x000009DE (2526)
Last Counter ID: 0x000009DE (2526)
First Help ID: 0x000009DF (2527)
Last Help ID: 0x000009DF (2527)
Background:
In an earlier post I talked about some new features for Windows 2008 and Vista. One of those new features that is often overlooked are the data collector sets (DCS). One particular role that leverages data collector sets is active directory. Active directory has put “hooks” into tracing that can really take a lot of the thinking out of the question “why is my domain controller sluggish”. For those of you still running Windows 2003 I go over a similar concept called Server Performance Advisor.
Anyways, you can play around with DCS by typing perfmon and then traversing to the section called Data Collector Sets (shocking). If you have performance issues, go here first as it’s like combining a netmon capture with a kernel trace and then handing you the smoking gun.
Challenge:
In my current role, we have a need to automate things quite a bit and so one of the actions I was looking at solving was collecting diagnostic information when a server is performing poorly. Usually when a high CPU alert comes in, someone would need to logon to the server and go to perfmon and start at DCS collection. More often is the case that by the time someone had been alerted and went to the server the sluggish behavior had subsided (the dreaded “close ticket, no problem found”).
My solution was to try and figure out a way to start a DCS collection remotely at the time of event so that the data was present when an actual human became engaged.
After some hard work, here is the code to do so! You can create your own XML file (your own DCS template) and pass it in, but more than likely you’ll be happy at just kicking off one of the built-in templates (AD/System Perf/System Diags).
Running it via powershell:
First, how to do it on the fly:
## PLA.dll lives under system32 on Vista and 2k8. This will create a powershell com object.
$datacollectorset = new-object -COM Pla.DataCollectorSet
##This is the name of the predefined DCS collector. It's read-only and will always be System\<something>
$name = "System\Active Directory Diagnostics"
##If you make the second param $null it will be the local machine.
$datacollectorset.Query($name,"serverA")
$datacollectorset.start($false)
## Status ReturnCodes: 0=stopped 1=running 2=compiling 3=queued (legacy OS) 4=unknown (usually autologger)
$datacollectorset.status
##When you're ready to stop it call stop.
$datacollectorset.stop($false)
##If you call status here, it will probably be '2' for a while as the server compiles the report.
$datacollectorset.status
And like so, you started and stopped a collection for Active Directory on you’re local computer or a remote server! Like I said though, you can create you’re own templates too. You might want to do this if you want to setup a built-in template to be scheduled to run daily, or perhaps you want to send the data to a network location, run more tasks at completion, etc. If you do want to create a custom template then the code changes a bit:
$datacollectorset = new-object -COM Pla.DataCollectorSet
## If you're making you're own (shows up under user defined).
$xml = get-content C:\custom.xml #You're custom exported XML file.
$datacollectorset.SetXml($xml)
##Commit codes: http://msdn.microsoft.com/en-us/library/aa371873(VS.85).aspx this is add or modify. Can't do this on a system created PLA instances (read only).
$datacollectorset.Commit($DCSPath , $null , 0x0003)
$datacollectorset.Query($DCSPath,$null)
$datacollectorset.start($false)
#Runs...
$datacollectorset.stop($false)
Scripting a solution:
Finally if you wanted to script this you could do something like what I’ve done below. This would collect for a desired interval (in seconds) and then when compilation completed display the path to the report. I wrote this in CTP3, but you can easily take the concepts and backport them. If the destination server is inaccessible, or you don't have permissions, then the script will blow up…
<#
.SYNOPSIS
This will fire up a PLA (Data Collector Set collection on a server and then copy it to the proper debug server
.DESCRIPTION
This is a proof of concept and only acceppts System defined collections. No error handling so I hope you type well.
#>
##Inputs
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)]
<#A system provided report to run like "System\System Performance", System\System Diagnostics, etc. #>
[string]$DCSPath,
[Parameter(Mandatory = $true)]
<# This is how long you want the DCS collection to run in seconds#>
[int32]$time,
[Parameter(Mandatory = $false)]
<#If you don't pass in a server name it will be $null and run on the local system#>
[string]$serverName
)
$datacollectorset = new-object -COM Pla.DataCollectorSet
$datacollectorset.Query($DCSPath,$serverName)
$datacollectorset.start($false)
Start-Sleep $time
$datacollectorset.stop($false)
##Now we'll loop while the report compiles.
$retries = 0
do
{sleep 30; $returnCode = $datacollectorset.Status ; $retries++}
while ($returnCode -eq 2 -and $retries -lt 60)
if ($retries -eq 60)
{
Write-Warning "Compiling has been running on the server for 30 minutes! You'll need to check the following location on the server later for the report:"
Write-Warning $datacollectorset.OutputLocation
break
}
##Compiling has finished, now we can copy the folder to some location
$path = $datacollectorset.OutputLocation
if ($serverName)
{
$path = $path.Replace(":","$")
Write-Host "`nReport complete and can be viewed at \\$serverName\$path\report.html on the server.`n"
}
else
{
Write-Host "`nReport complete and can be viewed at $path\report.html`n"
}
The result:
More info:
PLA reference: http://msdn.microsoft.com/en-us/library/aa372634(VS.85).aspx
Been using this internally for the last couple of weeks and have been digging the changes. Just wanted to put this out here to spread the word.
http://blogs.msdn.com/powershell/archive/2008/12/23/early-christmas-present-from-powershell-team-community-technology-preview-3-ctp3-of-windows-powershell-v2.aspx
Happy Holidays.
Technorati Tags:
powershell
This is more of a proof of concept, but I've used it with success internally. Take it and do with it what you want. Many thx to Brandon who did the "heavy lifting" when I got stuck!
Overview:
Ever run into cluster issues and wanted to see if the driver file versions matched on all the nodes of the cluster to rule out a mismatch on a driver level? Well I did! The basic gist is that you can show all the file versions for each node by just running the script against a node name. If you want to see only the drivers that don’t match then you’d use the pipeline with where-object (?).
Typical output:
When All drivers match:
PS C:\Debuggers> Test-MSCluster.ps1 ServerSQL11 | ?{!$_.IsSame}
Getting Nodes via WMI
Getting the drivers on: ServerSQL11
Getting the file versions for the drivers on: ServerSQL11
Getting the drivers on: ServerSQL12
Getting the file versions for the drivers on: ServerSQL12
PS C:\Debuggers>
One Mismatch:
PS C:\Debuggers> Test-MSCluster.ps1 ServerAX | ?{!$_.IsSame}
Getting Nodes via WMI
Getting the drivers on: ServerAX
Getting the file versions for the drivers on: ServerAX
Getting the drivers on: ServerBX
Getting the file versions for the drivers on: ServerBX
FileName ServerAX ServerBX
-------- ---------- ----------
rmcast.sys 6.0.6001.18000 6.0.6001.18069
Many nodes, many mismatches:
PS C:\Debuggers> Test-MSCluster.ps1 Server-Clus--11 | ?{!$_.IsSame}
Getting Nodes via WMI
Getting the drivers on: Server-Clus--10
Getting the file versions for the drivers on: Server-Clus--10
Getting the drivers on: Server-Clus--11
Getting the file versions for the drivers on: Server-Clus--11
Getting the drivers on: Server-Clus--15
Getting the file versions for the drivers on: Server-Clus--15
Getting the drivers on: Server-Clus--16
Getting the file versions for the drivers on: Server-Clus--16
Getting the drivers on: Server-Clus--13
Getting the file versions for the drivers on: Server-Clus--13
Getting the drivers on: Server-Clus--12
Getting the file versions for the drivers on: Server-Clus--12
FileName : Dbgv.sys
Server-Clus--10 : 4.60
Server-Clus--11 : FileMissing
Server-Clus--15 : FileMissing
Server-Clus--16 : FileMissing
Server-Clus--13 : FileMissing
Server-Clus--12 : FileMissing
IsSame : False
FileName : HpCISSs2.sys
Server-Clus--10 : FileMissing
Server-Clus--11 : FileMissing
Server-Clus--15 : FileMissing
Server-Clus--16 : 6.8.0.64 Build 9 (x86-64)
Server-Clus--13 : 6.8.0.64 Build 9 (x86-64)
Server-Clus--12 : 6.8.0.64 Build 9 (x86-64)
IsSame : False
FileName : USBSTOR.SYS
Server-Clus--10 : FileMissing
Server-Clus--11 : FileMissing
Server-Clus--15 : FileMissing
Server-Clus--16 : FileMissing
Server-Clus--13 : 6.0.6001.18000
Server-Clus--12 : 6.0.6001.18000
IsSame : False
FileName : mrxsmb10.sys
Server-Clus--10 : 6.0.6001.18000
Server-Clus--11 : 6.0.6001.18000
Server-Clus--15 : 6.0.6001.18000
Server-Clus--16 : 6.0.6001.18068
Server-Clus--13 : 6.0.6001.18000
Server-Clus--12 : 6.0.6001.18000
IsSame : False
FileName : nm3.sys
Server-Clus--10 : 03.02.0764.0001
Server-Clus--11 : FileMissing
Server-Clus--15 : FileMissing
Server-Clus--16 : FileMissing
Server-Clus--13 : FileMissing
Server-Clus--12 : FileMissing
IsSame : False
Code:
######################################################################
#Test-MSCluster.ps1
Param($ClusterNode)
# I am using this hashtable to store a unique list of file names.
$Files = @{}
# I am using this array to store my custom objects we create later.
$FileObjects = @()
Write-Host "Getting Nodes via WMI"
$nodes = gwmi -q "Select name from MSCluster_Node" -namespace root\mscluster -computername $ClusterNode -Authentication PacketPrivacy | %{$_.Name}
# Here we process each node and get all the drivers from the node and add it to our $Files HashTable to be processed
foreach ( $node in $nodes )
{
Write-Host "Getting the drivers on:" $node
# Here we are getting a list of the .sys files. Notice I am only getting the names
$filelistFinal = get-childitem "\\$node\admin$\system32\drivers" *.sys | %{$_.name}
Write-Host "Getting the file versions for the drivers on:" $node
foreach($file in $filelistFinal)
{
# foreach file found we add it to the hasttable, but hashtables can only have a key once
# so we need check if the key already exist. I do this because it is possible you could have
# unique drivers per node.
if(!$Files.$file)
{
$Files.Add($file,"added")
}
}
}
# Ok... now we have all our files time to process the hashtable and create our custom objects
foreach($FileName in $Files.Keys)
{
# This is how I create an object for each file
$myFileObj = New-Object System.Object
# This is how we add a property. In this case the FileName property. For these scenarios I chose add-member
# because you can dynamically add properties (i.e. NodeName with value of File version)
$myFileobj | add-Member -MemberType NoteProperty -Name FileName -Value $FileName
# Now we need to add properties for each node.
foreach($node in $nodes)
{
# Making sure the file exist on the node
if(Test-Path \\$node\admin$\system32\drivers\$FileName)
{
# Getting ProductVersion Info to use as the value for the Node Property
$fileInfo = [system.diagnostics.fileversioninfo]::getversioninfo("\\$node\admin$\system32\drivers\$FileName")
$myFileobj | add-Member -MemberType NoteProperty -Name $node -Value $FileInfo.ProductVersion
}
else
{
# File not found using FileMissing as the value for the Node Property
$myFileobj | add-Member -MemberType NoteProperty -Name $node -Value "FileMissing"
}
}
# Outputting Object
$FileObjects += $myFileObj
}
foreach($result in $FileObjects)
{
$isSame = $true
# Getting Server Name from Properties of the custom object
$servers = $result | Get-Member -MemberType Noteproperty | ?{$_.Name -ne "FileName"} | %{$_.Name}
# Checking the value of each server vs the other servers
foreach($server in $servers)
{
foreach($srv in $servers)
{
if($srv -ne $server)
{
# If the the value is different we set $isSame to $false
if($result."$srv" -ne $result."$server"){$isSame = $false}
}
}
}
# add the isSame property to the object
$result | add-Member -MemberType NoteProperty -Name IsSame -value $isSame
# output object
$result
}
######################################################################
***Note: This script is not fast, as it is getting the file versions for every driver (*.sys) on each system, I'd highly suggest not running this over the WAN...
Man it was hard to find info on using set-acl on a registry key! I was looking for a way to set an ACL that once set would be inherited by child keys and values. We needed to give “Local Service” full control on the registry key below and have the subkeys inherit the permission. You might say: “Why not use SUBINACL?”, well due to a bug or by design SUBINACL doesn’t work for WIN7 server core (probably should look into that). Besides, why call an exe when you can do it natively in PS. Anyways here is the code that ended up working. Hope next time someone goes looking for this it’ll be the first hit.
PS C:\> $acl= get-acl -path "hklm:\SOFTWARE\Microsoft\Reliability Analysis"
PS C:\> $inherit = [system.security.accesscontrol.InheritanceFlags]"ContainerInherit, ObjectInherit"
PS C:\> $propagation = [system.security.accesscontrol.PropagationFlags]"None"
PS C:\> $rule=new-object system.security.accesscontrol.registryaccessrule "LOCAL SERVICE","FullControl",$inherit,$propagation,"Allow"
PS C:\> $acl.addaccessrule($rule)
PS C:\> $acl|set-acl
And the output of GET-ACL shows local service now:
PS C:\> get-acl -path "hklm:\SOFTWARE\Microsoft\Reliability Analysis" | fl <—Verifying that it got set.
Path : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Reliability Analysis
Owner : BUILTIN\Administrators
Group : DOMAIN\Domain Users
Access : NT AUTHORITY\LOCAL SERVICE Allow FullControl
BUILTIN\Users Allow ReadKey
BUILTIN\Users Allow -2147483648
BUILTIN\Administrators Allow FullControl
BUILTIN\Administrators Allow 268435456
NT AUTHORITY\SYSTEM Allow FullControl
NT AUTHORITY\SYSTEM Allow 268435456
CREATOR OWNER Allow 268435456
Audit :
Sddl : O:BAG:DUD:AI(A;OICI;KA;;;LS)(A;ID;KR;;;BU)(A;CIIOID;GR;;;BU)(A;ID;KA;;;BA)(A;CIIOID;GA;;;BA)(A;ID;KA;;;SY)(A;CIIOID;GA;;;SY)(A;CIIOID;GA;;;CO)
Over the past few months I have fallen in love with PowerShell. I’ve taken on a new role (starting in mid-October) that will be more focused on automating out administrative tasks via powershell so the focus of this blog might change more towards that subject. I think the crowd that congregates here are IT admins for the most part, so this should remain relevant to your jobs (did this blog have a focus anyways?). If you have a scripting question, drop me a mail and I’ll post about it later.
This is where I started, its a self paced course on the basics. Once you walk through this (2 hours) you’ll start seeing the power: http://download.microsoft.com/download/4/7/1/47104ec6-410d-4492-890b-2a34900c9df2/Workshops-EN.zip
Obligatory powershell jpg:
First, I found the details here.
Second, things can change as this is being done with the CTP for Powershell 2.0
Third, if you don’t know about remoting in 2.0 watch this 5 minute video. Then read this.
Whew.
Backstory:
You might find yourself in a situation where you want to run a batch/vbs/cmd file on a bunch of servers at once. This batch file requires to connect to network locations to gather/put information during run time. The Powershell 2.0 remoting experience out of the box doesn’t allow you to do these “double hops” with the client side credentials. What happens is that when you remote using powershell, you get a set of credentials for use on that machine. When you go off-box, the request hasthe machine credentials. This obviously can cause issues leaving you two solutions:
1) Change the ACLS on the remote share to include the machine credentials
a. Can be done by adding <domain>\domain computers with read access to the shares(s).
b. Create a group that has all the machines required in it and ACL out the share permissions with that group.
2) Use CredSSP so that you get a credential which can do multi-hop.
So what is required to use CredSSP, thus allowing your client-side credentials to “pass-thru” to the server-side and go off box as your creds?
On the client-side:
new-item HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation -force
new-ItemProperty HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation -name AllowFreshCredentials -value 1 -type DWord -force
new-ItemProperty HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation -name ConcatenateDefaults_AllowFresh -value 1 -type DWord -force
new-item HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowFreshCredentials -force
new-ItemProperty HKLM:\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowFreshCredentials -name 1 -value wsman/* -force
winrm s winrm/config/client/auth '@{CredSSP="true"}'
On the server-side:
winrm s winrm/config/service/auth '@{CredSSP="true"}'
Example without credSSP:
PS C:\Debuggers> Invoke-Command -ComputerName server1.domain.com,server2.domain.com -ScriptBlock {c:\debuggers\test_PS.cmd} -Credential reddom\brad
C:\Windows\System32>cd\
C:\>cd debuggers
C:\Debuggers>md test
C:\Debuggers>copy \\serverx\bradshare\Book1.xlsx <-- Can’t make this happen as it goes off-box as the machine account.
Access is denied.
C:\Windows\System32>cd\
C:\>cd debuggers
C:\Debuggers>md test
C:\Debuggers>copy \\serverx\bradshare\Book1.xlsx
Access is denied.
PS C:\Debuggers>
Example with credSSP:
PS C:\Debuggers> Invoke-Command -ComputerName server1.domain.com,server2.domain.com -ScriptBlock {c:\debuggers\test_PS.cmd} -Authentication CredSSP -Credential reddom\brad
//Had to use the FQDN as it does an SPN lookup and hostname fails.
C:\Windows\System32>cd\
C:\>cd debuggers
C:\Debuggers>md test
C:\Debuggers>copy \\serverx\bradshare\Book1.xlsx <-- Now goes off the server-side with my ‘brad’ user account.
1 file(s) copied.
C:\Windows\System32>cd\
C:\>cd debuggers
C:\Debuggers>md test
C:\Debuggers>copy \\serverx\bradrutk$\Book1.xlsx
1 file(s) copied.
PS C:\Debuggers>
Update:
You must have at least the CTP2 verison of WINRM: https://connect.microsoft.com/WSMAN/Downloads
Make sure to run Configure-Wsman.ps1 and WINRM quickconfig too...
Technorati Tags:
powershell
This works for Windows 2003 and Windows 2008. We use it during our reliability study to let the server owners know that they shouldn't reboot their boxes without a good reason. You can use it for whatever you’d like. :)
The two keys to set:
reg add "\\brad-dc-01\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeCaption /t REG_SZ /d "MSIT Reliability Study" /f
reg add "\\brad-dc-01\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeText /t REG_SZ /d "This server is part of the MSIT Windows 7 Reliability Study. The server should not be rebooted. If the server is experiencing a bug, please contact DCOPERATE to triage and they will escalate as needed. If you are rebooting the server for a hotfix, private fix, or other legitimate reason, please document it properly in the shutdown tracker so that the statistics are accurate." /f
Hop to loop it and apply it en masse:
Open CMD with your alt creds and do the following:
C:\Windows\system32>for /f %a in (machines.txt) do (
More? reg add "\\%a\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeCaption /t REG_SZ /d "MSIT Reliability Study" /f
More? reg add "\\%a\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeText /t REG_SZ /d "This server is part of the... (HUGE LONG STRING) ... " /f
More? )
How to turn it off:
C:\Windows\system32>for /f %a in (machines.txt) do (
More? reg add "\\%a\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeCaption /t REG_SZ /d "" /f
More? reg add "\\%a\HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v LegalNoticeText /t REG_SZ /d "" /f
More? )
The result:
Another way of doing this is to set "Interactive logon: Message text for users attempting to logon" in secpol.msc...
Ran into a weird issue where I was getting access denied when trying to query nodes remotely in powershell. The query was working fine against Windows 2003 cluster names and worked locally when ran on a Windows 2008 cluster node, it just didn’t work remotely.
Against 2k3:
PS C:\Debuggers> gwmi -q "Select name from MSCluster_Node" -namespace root\mscluster -computername Server-2k3-01 | Select-Object Name
Name
----
Server-2k3-01
Server-2k3-02
Against 2k8:
PS C:\Debuggers> gwmi -q "Select name from MSCluster_Node" -namespace root\mscluster -computername Server-2k8-01
Get-WmiObject : Access denied
At line:1 char:5
+ gwmi <<<< -q "Select name from MSCluster_Node" -namespace root\mscluster -computername Server-2k8-01
I also tried the query outside of powershell to eliminate that form the equation with the same results and it still failed. So why the difference? Well looking around on the target, I noticed this event in the event log:
Log Name: Application
Source: Microsoft-Windows-WMI
Date: 9/5/2008 10:17:52 AM
Event ID: 5605
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: Server-2k8-01
Description:
Access to the root\mscluster namespace was denied because the namespace is marked with RequiresEncryption but the script or application attempted to connect to this namespace with an authentication level below Pkt_Privacy. Change the authentication level to Pkt_Privacy and run the script or application again.
Doing a little research I ran across this article explaining the event and what needs to happen to run the query properly:
http://technet.microsoft.com/en-us/library/cc727103.aspx In VBScript that means adding: authenticationLevel=pktPrivacy to your query. In Powershell (I’m using 2.0) you just add the authentication switch to get it to work. Now the query works on downlevel as well as 2k8:
PS C:\Debuggers> gwmi -q "Select name from MSCluster_Node" -namespace root\mscluster -computername Server-2k8-01 -Authentication PacketPrivacy | Select-Object Name
Name
----
Server-2k8-01
Server-2k8-02
Server-2k8-03
Server-2k8-04
Server-2k8-05
PostScript:
You can do a whole bunch of cool stuff with powershell check it out! Here’s just a little query to tell me each node and ‘t state:
PS C:\Debuggers> gwmi -q "Select * from MSCluster_Node" -namespace root\mscluster -computername TK5-CLUS-01 -Authentication PacketPrivacy | Select-Object Name,State | Format-Table -au
Name State
---- -----
tk5-clus-01 0
tk5-clus-02 0
tk5-clus-03 0
tk5-clus-04 1
tk5-clus-05 0
tk5-clus-06 0
Had an issue where a server would not allow logon via termian services each time you attempted to logon it would return this:
Soooooooooo, what to do here?
First, we made sure the account existed in the directory since that's why it appeared to be complaining. So I opened LDP and verified it existed, and that all "checked out" with being healthy (stare and compare against a good object).
Second thing we did was crank up netlogon debug logging (nltest dbflag) and see what it showed. It was complaining of a lot of stuff but nothing conclusive unfortunately. So at that point it was time to move to event viewer. The "nice" thing about this issue was that the server was accessible via the network with the same account that was failing to TS so I could do some of the investigation remotely.
One event in particular struck me:
Log Name: System
Source: Microsoft-Windows-Security-Kerberos
Date: 7/31/2008 4:11:24 PM
Event ID: 3
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: BRAD-SRV-01.braddom.bradforest.com
Description:
A Kerberos Error Message was received:
on logon session
Client Time:
Server Time: 23:11:24.0000 7/31/2008 Z
Error Code: 0x7 KDC_ERR_S_PRINCIPAL_UNKNOWN
Extended Error: 0xc0000035 KLIN(0)
Client Realm:
Client Name:
Server Realm: braddom.bradforest.COM
Server Name: host/BRAD-SRV-01.braddom.bradforest.com
Target Name: host/BRAD-SRV-01.braddom.bradforest.com@braddom.bradforest.COM
Error Text:
File: 9
Line: d86
Error Data is in record data.
Using err.exe I resolved the error code and found there was a collision:
C:\localbin>err 0xc0000035
# for hex 0xc0000035 / decimal -1073741771 :
STATUS_OBJECT_NAME_COLLISION ntstatus.h
# Object Name already exists.
# 1 matches found for "0xc0000035"
At this point it's time to look for a collision of "host/BRAD-SRV-01.braddom.bradforest.com" in the forest. The easiest way to do it is use a nice script called querySPN.vbs.
C:\localbin>querySPN.vbs HOST/BRAD-SRV-01.braddom.bradforest.com braddom.bradforest.com
Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.
CN=VL Account,CN=Users,DC=braddom,DC=bradforest,DC=com
Class: user
User Logon: VLSBST
-- host/BRAD-SRV-01.braddom.bradforest.com <----------------------------------------------------------------- Bingo the SPN is registered for two objects!
CN=BRAD-SRV-01,CN=Computers,DC=braddom,DC=bradforest,DC=com
Class: computer
Computer DNS: BRAD-SRV-01.braddom.bradforest.com
-- TERMSRV/BRAD-SRV-01.braddom.bradforest.com
-- TERMSRV/BRAD-SRV-01
-- HOST/BRAD-SRV-01
-- HOST/BRAD-SRV-01.braddom.bradforest.com <-----------------------------------------------------------------
Once we removed the SPN from the user account, logons began to immediately work.
-B
Quick Solution: Check the permissions on the root of C: and ensure that BUILTIN\Users have Read access.
Long Story:
8000FFFF == E_UNEXPECTED, not very helpful…
Had a client where windows update was continually failing with the error code 8000FFFF. When looking in the Windows Update log we’d see errors like this:
WARNING: PTError: 0x80248014
Handler FATAL: CBS called Error with 0x8000ffff, <— Checked the CBS.log file but that didn’t give any clues.
Handler FATAL: Error source is 106.
DnldMgr Error 0x8000ffff occurred while downloading update; notifying dependent calls.
AU # WARNING: Download failed, error = 0x8000FFFF
AU # WARNING: Download failed, error = 0x8000FFFF
AU WARNING: BeginInteractiveInstall failed, error = 0x8024000C
CltUI WARNING: AU directive Interactive Progress is exiting due to error 8024000C
And in the event viewer upon each run we’d see these events:
Log Name: Application
Source: ESENT
Date: 7/2/2008 3:05:16 PM
Event ID: 491
Task Category: General
Level: Error
Keywords: Classic
User: N/A
Computer: XXXX
Description:
Catalog Database (1560) Catalog Database: An attempt to determine the minimum I/O block size for the volume "C:\" containing "C:\Windows\system32\CatRoot2\" failed with system error 5 (0x00000005): "Access is denied. ". The operation will fail with error -1032 (0xfffffbf8).
Log Name: Application
Source: Microsoft-Windows-CAPI2
Date: 7/2/2008 3:05:16 PM
Event ID: 257
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: XXXX
Description:
The Cryptographic Services service failed to initialize the Catalog Database. The ESENT error was: -1032.
After seeing this data I did a stare and compare between my root permissions and his and found that he’d modified the c:\ permissions on his system:
His machine:
c:\temp\xcacls c:
C:\ NT AUTHORITY\SYSTEM:(OI)(CI)F
BUILTIN\Administrators:(OI)(CI)F
Mine:
C:\>xcacls c:\
c:\ BUILTIN\Administrators:F
BUILTIN\Administrators:(OI)(CI)(IO)F
NT AUTHORITY\SYSTEM:F
NT AUTHORITY\SYSTEM:(OI)(CI)(IO)F
BUILTIN\Users:(OI)(CI)R <— This is the key one missing that was causing the headache.
NT AUTHORITY\Authenticated Users:(OI)(CI)(IO)C
NT AUTHORITY\Authenticated Users:(special access:)
FILE_APPEND_DATA
The Cryptographic Services runs under “Network Service” which would require Users to have read access. I added BUILTIN\Users with read access to C and all worked again.
Hopefully this post will guide others with similar issues to the solution quickly.
Ran into an issue this week that was strange. When you TS’d to the box it would just show a blank background and nothing else. If you tried to launch task manager it would just fail silently to the user (actually access denied in the debugger). My user account was in the admin group and the server was completely accessible remotely with administrative perms. It was just when I (or anyone) tried to logon to the server locally or through TS that it was messed up. Another piece of the puzzle was that if you disabled UAC and rebooted the server the issue no longer repro’d.
So what was there with UAC and logging onto this server?
When logging on this event was triggered:
Log Name: Application
Source: Microsoft-Windows-Winlogon
Date: 5/27/2008 5:13:28 PM
Event ID: 4006
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: XXXX
Description:
The Windows logon process has failed to spawn a user application. Application name: . Command line parameters: C:\Windows\system32\userinit.exe.
Turns out that they removed the Account "NT AUTHORITY\INTERACTIVE" from the Users group on the machine. We added that account back into the users group and like magic it worked again. I'm working on getting a KB filed and written for this issue, but until then at least people can find it if they notice this event in the event log.
Reference:
http://technet2.microsoft.com/WindowsVista/en/library/00d04415-2b2f-422c-b70e-b18ff918c2811033.mspx?mfr=true
UAC Architecture
While the Windows Vista logon process externally appears to be the same as the logon process in Windows XP, the internal mechanics have greatly changed. The following illustration details how the logon process for an administrator differs from the logon process for a standard user.
Windows Vista logon process
When an administrator logs on, the user is granted two access tokens: a full administrator access token and a "filtered" standard user access token. By default, when a member of the local Administrators group logs on, the administrative Windows privileges are disabled and elevated user rights are removed, resulting in the standard user access token. The standard user access token is then used to launch the desktop (Explorer.exe).
HatTip to Ben on my Team who actually figured this out after I tried to debug it for 3 days...
This is an FYI post so others on the intertubes can find the answer quickly.
If you get this error:
Log Name: Application
Source: Application Error
Date: 4.11.2008 07:20:41
Event ID: 1000 Task Category: (100)
Level: Error
Keywords: Classic
User: N/A Computer: xxxxxxx
Description: Faulting application SearchIndexer.exe, version 6.0.6000.16386, time stamp 0x4549b667, faulting module mssrch.dll, version 6.0.6000.16386, time stamp 0x4549bd4b, exception code 0xc00000fd, fault offset 0x00003f8f...
Open up Wercon and if it looks like this:
Product
Microsoft Windows Search Indexer
Problem
Stopped working
Date
4/21/2008 8:30 AM
Status
Report Sent
Problem signature
Problem Event Name: APPCRASH
Application Name: SearchIndexer.exe
Application Version: 6.0.6000.16386
Application Timestamp: 4549b667
Fault Module Name: mssrch.dll
Fault Module Version: 6.0.6000.16386
Fault Module Timestamp: 4549bd4b
Exception Code: c00000fd
Exception Offset: 00007c4c
OS Version: 6.0.6000.2.0.0.256.4
Locale ID: 1033
Additional Information 1: f790
Additional Information 2: 174183f92d554d49550d71425f227859
Additional Information 3: efdd
Additional Information 4: 9c7dda392c8f13823238fe93325e6861
Extra information about the problem
Bucket ID: 349776197
Then you might be able to resolve this by:
A) Upgrading to Vista SP1
B) Install Windows Search 4 (which has now released): http://www.microsoft.com/windows/products/winfamily/desktopsearch/choose/windowssearch4.mspx
So i know there are tools out there to do this but figured some would be interested on how to do this real quick with stuff that's already in the OS.
1) Turn off echos to make the out put clean (don’t forget to turn it back on when its done via “echo on”).
2) The set is a sequence of numbers from start to end, by step amount. So (1,1,5) would generate the sequence 1 2 3 4 5 and (5,-1,1) would generate the sequence (5 4 3 2 1). So in this instance 1,1,254 would step to 254.
3) For the ping the –n says send one request instead the default of four. the –w sets the timeout for the echo request to 300 milliseconds, since I knew the subnet was close, so I did not need to wait the full timeout for the packet to return.
Example:
C:\debuggers>echo off
for /L %a in (1,1,254) do ping -n 1 -w 300 20.232.12.%a |findstr /i reply
Reply from 20.232.12.1: bytes=32 time=2ms TTL=245
Reply from 20.232.12.7: bytes=32 time=2ms TTL=55
Reply from 20.232.12.8: bytes=32 time=2ms TTL=53
Reply from 20.232.12.9: bytes=32 time=2ms TTL=55
Reply from 20.232.12.11: bytes=32 time=2ms TTL=53
Reply from 20.232.12.12: bytes=32 time=2ms TTL=55
Reply from 20.232.12.14: bytes=32 time=2ms TTL=55
Reply from 20.232.12.15: bytes=32 time=2ms TTL=53
Reply from 20.232.12.27: bytes=32 time=2ms TTL=53
Reply from 20.232.12.78: bytes=32 time=2ms TTL=53
Reply from 20.232.12.81: bytes=32 time=2ms TTL=55
Reply from 20.232.12.82: bytes=32 time=2ms TTL=53
Reply from 20.232.12.83: bytes=32 time=2ms TTL=53
Reply from 20.232.12.84: bytes=32 time=2ms TTL=53
Reply from 20.232.12.85: bytes=32 time=2ms TTL=55
Reply from 20.232.12.87: bytes=32 time=2ms TTL=53
Reply from 20.232.12.88: bytes=32 time=2ms TTL=53
Reply from 20.232.12.89: bytes=32 time=2ms TTL=53
Reply from 20.232.12.107: bytes=32 time=1ms TTL=53
Reply from 20.232.12.108: bytes=32 time=2ms TTL=53
Reply from 20.232.12.110: bytes=32 time=2ms TTL=53
Reply from 20.232.12.111: bytes=32 time=1ms TTL=55
Reply from 20.232.12.113: bytes=32 time=2ms TTL=55
Reply from 20.232.12.115: bytes=32 time=2ms TTL=55
Reply from 20.232.12.116: bytes=32 time=2ms TTL=53
Reply from 20.232.12.117: bytes=32 time=2ms TTL=55
Reply from 20.232.12.118: bytes=32 time=1ms TTL=55
Reply from 20.232.12.119: bytes=32 time=2ms TTL=53
Reply from 20.232.12.120: bytes=32 time=2ms TTL=53
Reply from 20.232.12.231: bytes=32 time=2ms TTL=53
Reply from 20.232.12.234: bytes=32 time=1ms TTL=55
Reply from 20.232.12.235: bytes=32 time=1ms TTL=55
Reply from 20.232.12.237: bytes=32 time=2ms TTL=55
Reply from 20.232.12.238: bytes=32 time=1ms TTL=55
Reply from 20.232.12.239: bytes=32 time=2ms TTL=53
Reply from 20.232.12.242: bytes=32 time=1ms TTL=55
Reply from 20.232.12.244: bytes=32 time=1ms TTL=55
Reply from 20.232.12.245: bytes=32 time=2ms TTL=53
Reply from 20.232.12.246: bytes=32 time=2ms TTL=53
Reply from 20.232.12.247: bytes=32 time=1ms TTL=55
Reply from 20.232.12.248: bytes=32 time=1ms TTL=55
Reply from 20.232.12.249: bytes=32 time=2ms TTL=53
Reply from 20.232.12.250: bytes=32 time=2ms TTL=55
Update:
And in powershell: 1..254 | % {ping -n 1 -w 300 157.56.144.$_ | findstr /i reply }
