Had an issue where a server would not allow logon via termian services each time you attempted to logon it would return this:
Soooooooooo, what to do here?
First, we made sure the account existed in the directory since that's why it appeared to be complaining. So I opened LDP and verified it existed, and that all "checked out" with being healthy (stare and compare against a good object).
Second thing we did was crank up netlogon debug logging (nltest dbflag) and see what it showed. It was complaining of a lot of stuff but nothing conclusive unfortunately. So at that point it was time to move to event viewer. The "nice" thing about this issue was that the server was accessible via the network with the same account that was failing to TS so I could do some of the investigation remotely.
One event in particular struck me:
Log Name: System
Source: Microsoft-Windows-Security-Kerberos
Date: 7/31/2008 4:11:24 PM
Event ID: 3
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: BRAD-SRV-01.braddom.bradforest.com
Description:
A Kerberos Error Message was received:
on logon session
Client Time:
Server Time: 23:11:24.0000 7/31/2008 Z
Error Code: 0x7 KDC_ERR_S_PRINCIPAL_UNKNOWN
Extended Error: 0xc0000035 KLIN(0)
Client Realm:
Client Name:
Server Realm: braddom.bradforest.COM
Server Name: host/BRAD-SRV-01.braddom.bradforest.com
Target Name: host/BRAD-SRV-01.braddom.bradforest.com@braddom.bradforest.COM
Error Text:
File: 9
Line: d86
Error Data is in record data.
Using err.exe I resolved the error code and found there was a collision:
C:\localbin>err 0xc0000035
# for hex 0xc0000035 / decimal -1073741771 :
STATUS_OBJECT_NAME_COLLISION ntstatus.h
# Object Name already exists.
# 1 matches found for "0xc0000035"
At this point it's time to look for a collision of "host/BRAD-SRV-01.braddom.bradforest.com" in the forest. The easiest way to do it is use a nice script called querySPN.vbs.
C:\localbin>querySPN.vbs HOST/BRAD-SRV-01.braddom.bradforest.com braddom.bradforest.com
Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.
CN=VL Account,CN=Users,DC=braddom,DC=bradforest,DC=com
Class: user
User Logon: VLSBST
-- host/BRAD-SRV-01.braddom.bradforest.com <----------------------------------------------------------------- Bingo the SPN is registered for two objects!
CN=BRAD-SRV-01,CN=Computers,DC=braddom,DC=bradforest,DC=com
Class: computer
Computer DNS: BRAD-SRV-01.braddom.bradforest.com
-- TERMSRV/BRAD-SRV-01.braddom.bradforest.com
-- TERMSRV/BRAD-SRV-01
-- HOST/BRAD-SRV-01
-- HOST/BRAD-SRV-01.braddom.bradforest.com <-----------------------------------------------------------------
Once we removed the SPN from the user account, logons began to immediately work.
-B
Quick Solution: Check the permissions on the root of C: and ensure that BUILTIN\Users have Read access.
Long Story:
8000FFFF == E_UNEXPECTED, not very helpful…
Had a client where windows update was continually failing with the error code 8000FFFF. When looking in the Windows Update log we’d see errors like this:
WARNING: PTError: 0x80248014
Handler FATAL: CBS called Error with 0x8000ffff, <— Checked the CBS.log file but that didn’t give any clues.
Handler FATAL: Error source is 106.
DnldMgr Error 0x8000ffff occurred while downloading update; notifying dependent calls.
AU # WARNING: Download failed, error = 0x8000FFFF
AU # WARNING: Download failed, error = 0x8000FFFF
AU WARNING: BeginInteractiveInstall failed, error = 0x8024000C
CltUI WARNING: AU directive Interactive Progress is exiting due to error 8024000C
And in the event viewer upon each run we’d see these events:
Log Name: Application
Source: ESENT
Date: 7/2/2008 3:05:16 PM
Event ID: 491
Task Category: General
Level: Error
Keywords: Classic
User: N/A
Computer: XXXX
Description:
Catalog Database (1560) Catalog Database: An attempt to determine the minimum I/O block size for the volume "C:\" containing "C:\Windows\system32\CatRoot2\" failed with system error 5 (0x00000005): "Access is denied. ". The operation will fail with error -1032 (0xfffffbf8).
Log Name: Application
Source: Microsoft-Windows-CAPI2
Date: 7/2/2008 3:05:16 PM
Event ID: 257
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: XXXX
Description:
The Cryptographic Services service failed to initialize the Catalog Database. The ESENT error was: -1032.
After seeing this data I did a stare and compare between my root permissions and his and found that he’d modified the c:\ permissions on his system:
His machine:
c:\temp\xcacls c:
C:\ NT AUTHORITY\SYSTEM:(OI)(CI)F
BUILTIN\Administrators:(OI)(CI)F
Mine:
C:\>xcacls c:\
c:\ BUILTIN\Administrators:F
BUILTIN\Administrators:(OI)(CI)(IO)F
NT AUTHORITY\SYSTEM:F
NT AUTHORITY\SYSTEM:(OI)(CI)(IO)F
BUILTIN\Users:(OI)(CI)R <— This is the key one missing that was causing the headache.
NT AUTHORITY\Authenticated Users:(OI)(CI)(IO)C
NT AUTHORITY\Authenticated Users:(special access:)
FILE_APPEND_DATA
The Cryptographic Services runs under “Network Service” which would require Users to have read access. I added BUILTIN\Users with read access to C and all worked again.
Hopefully this post will guide others with similar issues to the solution quickly.
Ran into an issue this week that was strange. When you TS’d to the box it would just show a blank background and nothing else. If you tried to launch task manager it would just fail silently to the user (actually access denied in the debugger). My user account was in the admin group and the server was completely accessible remotely with administrative perms. It was just when I (or anyone) tried to logon to the server locally or through TS that it was messed up. Another piece of the puzzle was that if you disabled UAC and rebooted the server the issue no longer repro’d.
So what was there with UAC and logging onto this server?
When logging on this event was triggered:
Log Name: Application
Source: Microsoft-Windows-Winlogon
Date: 5/27/2008 5:13:28 PM
Event ID: 4006
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: XXXX
Description:
The Windows logon process has failed to spawn a user application. Application name: . Command line parameters: C:\Windows\system32\userinit.exe.
Turns out that they removed the Account "NT AUTHORITY\INTERACTIVE" from the Users group on the machine. We added that account back into the users group and like magic it worked again. I'm working on getting a KB filed and written for this issue, but until then at least people can find it if they notice this event in the event log.
Reference:
http://technet2.microsoft.com/WindowsVista/en/library/00d04415-2b2f-422c-b70e-b18ff918c2811033.mspx?mfr=true
UAC Architecture
While the Windows Vista logon process externally appears to be the same as the logon process in Windows XP, the internal mechanics have greatly changed. The following illustration details how the logon process for an administrator differs from the logon process for a standard user.
Windows Vista logon process
When an administrator logs on, the user is granted two access tokens: a full administrator access token and a "filtered" standard user access token. By default, when a member of the local Administrators group logs on, the administrative Windows privileges are disabled and elevated user rights are removed, resulting in the standard user access token. The standard user access token is then used to launch the desktop (Explorer.exe).
HatTip to Ben on my Team who actually figured this out after I tried to debug it for 3 days...
This is an FYI post so others on the intertubes can find the answer quickly.
If you get this error:
Log Name: Application
Source: Application Error
Date: 4.11.2008 07:20:41
Event ID: 1000 Task Category: (100)
Level: Error
Keywords: Classic
User: N/A Computer: xxxxxxx
Description: Faulting application SearchIndexer.exe, version 6.0.6000.16386, time stamp 0x4549b667, faulting module mssrch.dll, version 6.0.6000.16386, time stamp 0x4549bd4b, exception code 0xc00000fd, fault offset 0x00003f8f...
Open up Wercon and if it looks like this:
Product
Microsoft Windows Search Indexer
Problem
Stopped working
Date
4/21/2008 8:30 AM
Status
Report Sent
Problem signature
Problem Event Name: APPCRASH
Application Name: SearchIndexer.exe
Application Version: 6.0.6000.16386
Application Timestamp: 4549b667
Fault Module Name: mssrch.dll
Fault Module Version: 6.0.6000.16386
Fault Module Timestamp: 4549bd4b
Exception Code: c00000fd
Exception Offset: 00007c4c
OS Version: 6.0.6000.2.0.0.256.4
Locale ID: 1033
Additional Information 1: f790
Additional Information 2: 174183f92d554d49550d71425f227859
Additional Information 3: efdd
Additional Information 4: 9c7dda392c8f13823238fe93325e6861
Extra information about the problem
Bucket ID: 349776197
Then you might be able to resolve this by:
A) Upgrading to Vista SP1
B) Install Windows Search 4 (which has now released): http://www.microsoft.com/windows/products/winfamily/desktopsearch/choose/windowssearch4.mspx
So i know there are tools out there to do this but figured some would be interested on how to do this real quick with stuff that's already in the OS.
1) Turn off echos to make the out put clean (don’t forget to turn it back on when its done via “echo on”).
2) The set is a sequence of numbers from start to end, by step amount. So (1,1,5) would generate the sequence 1 2 3 4 5 and (5,-1,1) would generate the sequence (5 4 3 2 1). So in this instance 1,1,254 would step to 254.
3) For the ping the –n says send one request instead the default of four. the –w sets the timeout for the echo request to 300 milliseconds, since I knew the subnet was close, so I did not need to wait the full timeout for the packet to return.
Example:
C:\debuggers>echo off
for /L %a in (1,1,254) do ping -n 1 -w 300 20.232.12.%a |findstr /i reply
Reply from 20.232.12.1: bytes=32 time=2ms TTL=245
Reply from 20.232.12.7: bytes=32 time=2ms TTL=55
Reply from 20.232.12.8: bytes=32 time=2ms TTL=53
Reply from 20.232.12.9: bytes=32 time=2ms TTL=55
Reply from 20.232.12.11: bytes=32 time=2ms TTL=53
Reply from 20.232.12.12: bytes=32 time=2ms TTL=55
Reply from 20.232.12.14: bytes=32 time=2ms TTL=55
Reply from 20.232.12.15: bytes=32 time=2ms TTL=53
Reply from 20.232.12.27: bytes=32 time=2ms TTL=53
Reply from 20.232.12.78: bytes=32 time=2ms TTL=53
Reply from 20.232.12.81: bytes=32 time=2ms TTL=55
Reply from 20.232.12.82: bytes=32 time=2ms TTL=53
Reply from 20.232.12.83: bytes=32 time=2ms TTL=53
Reply from 20.232.12.84: bytes=32 time=2ms TTL=53
Reply from 20.232.12.85: bytes=32 time=2ms TTL=55
Reply from 20.232.12.87: bytes=32 time=2ms TTL=53
Reply from 20.232.12.88: bytes=32 time=2ms TTL=53
Reply from 20.232.12.89: bytes=32 time=2ms TTL=53
Reply from 20.232.12.107: bytes=32 time=1ms TTL=53
Reply from 20.232.12.108: bytes=32 time=2ms TTL=53
Reply from 20.232.12.110: bytes=32 time=2ms TTL=53
Reply from 20.232.12.111: bytes=32 time=1ms TTL=55
Reply from 20.232.12.113: bytes=32 time=2ms TTL=55
Reply from 20.232.12.115: bytes=32 time=2ms TTL=55
Reply from 20.232.12.116: bytes=32 time=2ms TTL=53
Reply from 20.232.12.117: bytes=32 time=2ms TTL=55
Reply from 20.232.12.118: bytes=32 time=1ms TTL=55
Reply from 20.232.12.119: bytes=32 time=2ms TTL=53
Reply from 20.232.12.120: bytes=32 time=2ms TTL=53
Reply from 20.232.12.231: bytes=32 time=2ms TTL=53
Reply from 20.232.12.234: bytes=32 time=1ms TTL=55
Reply from 20.232.12.235: bytes=32 time=1ms TTL=55
Reply from 20.232.12.237: bytes=32 time=2ms TTL=55
Reply from 20.232.12.238: bytes=32 time=1ms TTL=55
Reply from 20.232.12.239: bytes=32 time=2ms TTL=53
Reply from 20.232.12.242: bytes=32 time=1ms TTL=55
Reply from 20.232.12.244: bytes=32 time=1ms TTL=55
Reply from 20.232.12.245: bytes=32 time=2ms TTL=53
Reply from 20.232.12.246: bytes=32 time=2ms TTL=53
Reply from 20.232.12.247: bytes=32 time=1ms TTL=55
Reply from 20.232.12.248: bytes=32 time=1ms TTL=55
Reply from 20.232.12.249: bytes=32 time=2ms TTL=53
Reply from 20.232.12.250: bytes=32 time=2ms TTL=55
Just saw this being discussed internally and thought that it was quite useful to a lot of you out there so I thought I'd share. The true boolean to grp.GetMembers tells it to recursively get the nested group members too. I tested this out on discussion groups, security groups, with users and computers and works as expected.
http://msdn2.microsoft.com/en-us/library/bb339975.aspx
using System;
using System.Collections.Generic;
using System.Text;
using System.DirectoryServices.AccountManagement;
namespace groupEnum
{
class Program
{
public static string groupName = string.Empty;
public static string domainName = string.Empty;
static void Main(string[] args)
{
groupName = args[0];
domainName = args[1];
PrincipalContext ctx = new PrincipalContext(ContextType.Domain, domainName);
GroupPrincipal grp = GroupPrincipal.FindByIdentity(ctx, IdentityType.Name, groupName);
if (grp != null)
{
foreach (Principal p in grp.GetMembers(true))
{
Console.WriteLine(p.Name); //You can add more attributes, samaccountname, UPN, DN, object type, etc...
}
grp.Dispose();
ctx.Dispose();
}
else
{
Console.WriteLine("\nWe did not find that group in that domain, perhaps the group resides in a different domain?");
}
}
}
}
The Microsoft IPsec Diagnostic Tool is available for Windows Server 2008, for Windows Vista, for Windows Server 2003, and for Windows XP
This should help you out when you have those weird "network" issues going on with some clients where IPSEC is deployed.
Description from KB:
You can use the Microsoft IPsec Diagnostic Tool to check for common network problems on the host computer. When problems are found, the tool suggests appropriate repair commands. The tool also collects IPsec policy information on the computer, and it parses the IPsec logs to determine the reasons for network failures. Additionally, you can use this tool for collecting traces of VPN connections and for collecting information about NAT clients, about Windows Firewall configuration, about Group Policy updates, about Wireless events, and about System events.
This diagnostic report that is generated by this tool is derived from the system logs that are collected by the tool during its analysis phase. Therefore, this report is conclusive. The information in these logs is sufficient to diagnose any network-related issues. For assisted support, you may have to share the logs with network administrators or with Microsoft Support. For more assistance, see the Help feature that is included with the tool.
Technorati Tags:
IPSEC,
Windows
All of these are for kernel mode, these are just commands I use often that don't troubleshoot a particular problem, but are helpful in getting a general picture of the system. If you have a specific issue you're trying to understand, drop a note and I'll see if there is a command to help you out.
Vertarget:
Lists Version information for the machine/dump you're debugging. You can also use "version" to tell you about the debugger bits.
1: kd> vertarget
Windows Kernel Version 6001 (Service Pack 1) MP (4 procs) Free x64
Product: LanManNt, suite: TerminalServer SingleUserTS
Built by: 6001.18000.amd64fre.longhorn_rtm.080118-1840
Kernel base = 0xfffff800`0160c000 PsLoadedModuleList = 0xfffff800`017d1db0
Debug session time: Tue Apr 1 14:29:22.553 2008 (GMT-7)
System Uptime: 0 days 0:03:14.328
!sysinfo
Good utility to check the CPU revs, BIOS revs, etc
1: kd> !sysinfo machineid
Machine ID Information [From Smbios 2.3, DMIVersion 35, Size=3752]
BiosVendor = American Megatrends Inc.
BiosVersion = 080002
BiosReleaseDate = 10/01/2007
SystemManufacturer = Microsoft Corporation
SystemProductName = Virtual Machine
SystemVersion = 5.0
BaseBoardManufacturer = Microsoft Corporation
BaseBoardProduct = Virtual Machine
BaseBoardVersion = 5.0
1: kd> !sysinfo cpuinfo
[CPU Information]
~MHz = REG_DWORD 2660
Component Information = REG_BINARY 0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
Configuration Data = REG_FULL_RESOURCE_DESCRIPTOR ff,ff,ff,ff,ff,ff,ff,ff,0,0,0,0,0,0,0,0
Identifier = REG_SZ Intel64 Family 6 Model 15 Stepping 6
ProcessorNameString = REG_SZ Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
Update Signature = REG_BINARY 0,0,0,0,0,0,0,0
Update Status = REG_DWORD 8
VendorIdentifier = REG_SZ GenuineIntel
MSR8B = REG_QWORD 0
Getting the server name from the dump:
It's quite a bit easier to do internally, but this will get it done too. Good to know you're debugging the right server. :)
1: kd> x srv!SrvComputerName
fffffa60`04024500 srv!SrvComputerName = <no type information>
1: kd> dq fffffa60`04024500
fffffa60`04024500 00000000`00180018 fffff880`04ccd8c0
fffffa60`04024510 00000000`00000000 00000000`00000000
fffffa60`04024520 00000000`00000000 00000000`00000000
fffffa60`04024530 00000000`000c000a fffff880`04a0fc60
fffffa60`04024540 fffffa60`04024540 fffffa60`04024540
fffffa60`04024550 00000000`00060001 fffffa60`04024558
fffffa60`04024560 fffffa60`04024558 00000000`ffffffff
fffffa60`04024570 00000000`00000000 00000000`00000000
1: kd> du fffff880`04ccd8c0
fffff880`04ccd8c0 "BRAD-LHDC-01?"
!running -ti
This will dump the stacks of each thread that is running on each processor
1: kd> !running -ti
System Processors f (affinity mask)
Idle Processors f
All processors idle.
Prcb Current Next
0 fffff80001780680 fffff80001785b80 ................
Child-SP RetAddr Call Site
fffff800`026bb8d0 fffffa60`00a066da nt!KeSetTimer+0x89
fffff800`026bb920 fffffa60`00a06aca NETIO!WfpStartTimerForLeftTime+0x8a
fffff800`026bb970 fffffa60`00a06585 NETIO!WfppLeastRecentlyUsedTimerRoutine+0x1aa
fffff800`026bb9c0 fffffa60`00a067ff NETIO!WfpTimerWheelTimeoutHandler+0x175
fffff800`026bba40 fffff800`016698b3 NETIO!WfpSysTimerNdisCallback+0x4f
fffff800`026bba70 fffff800`0166a238 nt!KiTimerListExpire+0x333
fffff800`026bbca0 fffff800`0166aa9f nt!KiTimerExpiration+0x1d8
fffff800`026bbd10 fffff800`0166bb72 nt!KiRetireDpcList+0x1df
fffff800`026bbd80 fffff800`018395c0 nt!KiIdleLoop+0x62
fffff800`026bbdb0 00000000`fffff800 nt!zzz_AsmCodeRange_End+0x4
1 fffffa60005f3180 fffffa60005fcd40 ................
Child-SP RetAddr Call Site
fffffa60`0171bb08 fffff800`016b03d7 nt!RtlpBreakWithStatusInstruction
fffffa60`0171bb10 fffff800`0165afef nt! ?? ::FNODOBFM::`string'+0x356a
fffffa60`0171bb50 fffffa60`026867a2 nt!KiSecondaryClockInterrupt+0x11f
fffffa60`0171bce8 fffffa60`02685685 intelppm!C1Halt+0x2
fffffa60`0171bcf0 fffff800`0167c7c8 intelppm!C1Idle+0x9
fffffa60`0171bd20 fffff800`0166bb31 nt!PoIdle+0x148
fffffa60`0171bd80 fffff800`018395c0 nt!KiIdleLoop+0x21
fffffa60`0171bdb0 00000000`fffffa60 nt!zzz_AsmCodeRange_End+0x4
!stacks
This is a great utility to check what threads are waiting on for each process. Find out more in the debuggers chm.
1: kd> !stacks 2
Proc.Thread .Thread Ticks ThreadState Blocker
Max cache size is : 1048576 bytes (0x400 KB)
Total memory in cache : 0 bytes (0 KB)
Number of regions cached: 0
0 full reads broken into 0 partial reads
counts: 0 cached/0 uncached, 0.00% cached
bytes : 0 cached/0 uncached, 0.00% cached
** Prototype PTEs are implicitly decoded
[fffffa8000c77950 System]
4.000008 fffffa8000c774c0 ffffe94b GATEWAIT nt!KiSwapContext+0x7f
nt!KiSwapThread+0x2fa
nt!KeWaitForGate+0x22a
nt!MmZeroPageThread+0x162
nt!Phase1Initialization+0xe
nt!PspSystemThreadStartup+0x57
nt!KiStartSystemThread+0x16
4.000010 fffffa8000ca0720 ffffff8c Blocked nt!KiSwapContext+0x7f
nt!KiSwapThread+0x2fa
nt!KeWaitForSingleObject+0x2da
nt!PopIrpWorkerControl+0x22
nt!PspSystemThreadStartup+0x57
nt!KiStartSystemThread+0x16
4.000014 fffffa8000c78bb0 fffffcb0 Blocked nt!KiSwapContext+0x7f
nt!KiSwapThread+0x2fa
nt!KeWaitForSingleObject+0x2da
nt!PopIrpWorker+0x164
nt!PspSystemThreadStartup+0x57
nt!KiStartSystemThread+0x16
<SNIP>
!PCR
Command will show you some useful info from the processor control block. Like the current thread, next, DPQ queues (Can run !dpcs).
1: kd> !pcr
KPCR for Processor 1 at fffffa60005f3000:
Major 1 Minor 1
NtTib.ExceptionList: fffffa60005fd280
NtTib.StackBase: fffffa60005f6cc0
NtTib.StackLimit: 000000000554f578
NtTib.SubSystemTib: fffffa60005f3000
NtTib.Version: 00000000005f3180
NtTib.UserPointer: fffffa60005f37f0
NtTib.SelfTib: 000007fffff8a000
SelfPcr: 0000000000000000
Prcb: fffffa60005f3180
Irql: 0000000000000000
IRR: 0000000000000000
IDR: 0000000000000000
InterruptMode: 0000000000000000
IDT: 0000000000000000
GDT: 0000000000000000
TSS: 0000000000000000
CurrentThread: fffffa60005fcd40
NextThread: 0000000000000000
IdleThread: fffffa60005fcd40
DpcQueue: 0xfffffa800124dc70 0xfffffa6000e7abe0 [Normal] tcpip!TcpPeriodicTimeoutHandler
1: kd>
!LMI <driver>
When I want to find out ifno about a particular driver in the dump, i use "lm n t" to get all of them, but then !lmi to drill into one. I use it quite often to see if I have the private or public symbol loaded
1: kd> !lmi srv.sys
Loaded Module Info: [srv.sys]
Module: srv
Base Address: fffffa6004007000
Image Name: srv.sys
Machine Type: 34404 (X64)
Time Stamp: 47919135 Fri Jan 18 21:57:09 2008
Size: 94000
CheckSum: 70fe5
Characteristics: 22 perf
Debug Data Dirs: Type Size VA Pointer
CODEVIEW 20, 142c8, 136c8 RSDS - GUID: {D3FD3BA3-615D-437E-83B9-D339ED15DEE3}
Age: 2, Pdb: srv.pdb
CLSID 4, 142c4, 136c4 [Data not mapped]
Image Type: MEMORY - Image read successfully from loaded memory.
Symbol Type: PDB - Symbols loaded successfully from symbol server.
C:\Debugger_Public\sym\srv.pdb\D3FD3BA3615D437E83B9D339ED15DEE32\srv.pdb
Load Report: public symbols , not source indexed
C:\Debugger_Public\sym\srv.pdb\D3FD3BA3615D437E83B9D339ED15DEE32\srv.pdb
Just got released yesterday:
X86: http://www.microsoft.com/downloads/details.aspx?FamilyID=9ff6e897-23ce-4a36-b7fc-d52065de9960&DisplayLang=en
X64: http://www.microsoft.com/downloads/details.aspx?FamilyID=d647a60b-63fd-4ac5-9243-bd3c497d2bc5&DisplayLang=en
Overview
Microsoft Remote Server Administration Tools (RSAT) enables IT administrators to remotely manage roles and features in Windows Server 2008 from a computer running Windows Vista with SP1. It includes support for remote management of computers running either a Server Core installation or the full installation option of Windows Server 2008. It provides similar functionality to Windows Server 2003 Administration Tools Pack.
After you install this item, you may have to restart your computer. This update is provided to you and licensed under the Windows Vista License Terms.
Once you install the KB, you need to enable the RSAT tools by doing the following:
1. Click Start, click Control Panel, and then click Programs.
2. In the Programs and Features area, click Turn Windows features on or off.
3. If you are prompted by User Account Control to allow the Windows Features dialog box to open, click Continue.
4. In the Windows Features dialog box, expand Remote Server Administration Tools.
5. Select the remote management tools that you want to install.
6. Click OK.
Other notables:
1) Windows Server® 2008 Network Shell (Netsh) Technical Reference What can you do in Netsh in Win2k8? How do I add IP info? How do I adjust the firewall? How do I connect to a remote server via netsh? You get the point.
2) Active Directory Database Mounting Tool Screencast Great screen cast on how to take a snapshot of your DIT, mount it, and view an offline copy via dsa.msc
3) IIS7 Media Pack Bit Rate Throttling Module For media files, Bit Rate Throttling implements a dynamic per-file throttling capability to provide intelligent progressive downloading.
Those who are just getting into scripting might be wondering how to query info from remote machines using WMI and how to find useful information to query. When I started out trying to learn some of the WMI syntax and gathering info, I started with ScriptoMatic.
I found this tool to be quick and painless for finding out what could be pulled from WMI and how it was done, if you've never played with it, go grab it and check it out.
When you click the "run" button it'll dump out whatever you asked scriptomatic to search for:
==========================================
Computer: ServerA
==========================================
Caption: Domain
ClientSiteName: NA-WA-SITE
CreationClassName: Win32_NTDomain
DcSiteName: NA-WA-SITE
Description: Domain
DnsForestName: microsoft.com
DomainControllerAddress: \\2002:4898:dc5:33:218:feff:fe75:904
DomainControllerAddressType: 1
DomainControllerName: \\DC-DC-35
DomainGuid: {F488EF59-EEEF-11D2-A5DA-00805F9F34DE}
DomainName: Domain
DSDirectoryServiceFlag: True
DSDnsControllerFlag: False
DSDnsDomainFlag: False
DSDnsForestFlag: True
DSGlobalCatalogFlag: True
DSKerberosDistributionCenterFlag: True
DSPrimaryDomainControllerFlag: False
DSTimeServiceFlag: True
DSWritableFlag: True
Name: Domain: Domain
PrimaryOwnerContact:
PrimaryOwnerName:
Roles:
Status: OK
Other site with WMI scripts prepopulated for you:
WMI has a plethora of information that can be gathered locally or remotely from systems so it might be daunting to find out what you want to gather. I stumbled upon this site today and found a ton of stuff that will be useful to admins: WMI Tasks for Scripts and Applications.
Here are the the task categories and descriptions from the page:
Accounts and Domains
Obtain information such as the computer domain or the currently logged-on user. Many domain- or account-related tasks are best performed with ADSI scripts. For examples, see the TechNet ScriptCenter at http://www.microsoft.com/technet.
Computer Hardware
Obtain information about the presence, state, or properties of hardware components. For example, you can determine whether a computer is a desktop or laptop.
Computer Software
Obtain information such as which software is installed by the Windows Installer (MSI) and software versions.
Connecting to the WMI Service
To get data from WMI, either on the local computer or from a remote computer, you must connect to the WMI service by connecting to a specific namespace. In most cases, use either the shorthand moniker connection or the Locator connection.
Dates and Times
Windows XP introduced several WMI classes and a scripting object to parse or convert the CIM datetime format.
Desktop Management
Obtain data from or control remote desktops. For example, you can determine whether or not the screensaver requires a password. WMI also gives you the ability shut down a remote computer.
Disks and File Systems
Obtain information about disk drive hardware state, logical volumes.
Event Logs
Obtain event data from NT Event log files and perform operations like backing up or clearing log files.
Files and Folders
Change file or folder properties through WMI, including creating a share or renaming a file.
Networking
Manage and obtain information about connections and IP or MAC addresses.
Operating Systems
Obtain information about the operating system such as version, whether it is activated, or which hotfixes are installed.
Performance Monitoring
Use the WMI classes that obtain data from performance counters to access and refresh data about computer performance.
Processes
Obtain information such as the account under which a process is running. You can perform actions like creating processes.
Printers and Printing
Manage and obtain data about printers, such as finding or setting the default printer.
Registry
Create and modify registry keys and values.
Scheduled Tasks
Create and get information about scheduled tasks.
Services
Obtain information about services, including dependent or antecedent services.
One last thing:
Scritpomatic does have a twin for ADSI too: ADSI ScriptoMatic.
I've been messing around over the last week making a tool that will frisk a remote machine. It's been a fun project, a couple of items I got hung up on were if the machine was server core and if it was a VM.
I mean who would knowingly TS to a server if they knew it was server core? As for the VM, it's nice to know before-hand so you dont request a debugger to be attached to a virtual server ;).
Well here are some snippets for those two things, hope it helps those trying to do similar queries...
Server Core:
Basically you just need to look at the OperatingSystemSKU value and if it E (hex) or 14 (decimal) then its server core. This and all the other SKU numbers are listed here: http://msdn2.microsoft.com/en-us/library/ms724358.aspx
System.Management.ConnectionOptions objconn = new System.Management.ConnectionOptions();
objconn.Impersonation = System.Management.ImpersonationLevel.Impersonate;
objconn.EnablePrivileges = true;
System.Management.ManagementScope exmangescope = new System.Management.ManagementScope(@"\\" + srvName + @"\root\cimv2", objconn);
System.Management.ObjectQuery objquery = new System.Management.ObjectQuery("SELECT * FROM Win32_OperatingSystem");
System.Management.ManagementObjectSearcher objsearch = new System.Management.ManagementObjectSearcher(exmangescope, objquery);
System.Management.ManagementObjectCollection queryCollection = objsearch.Get();
foreach (System.Management.ManagementObject stringer in queryCollection)
{
serverCoreval = stringer["OperatingSystemSKU"].ToString();
//Console.WriteLine(serverCoreval);
}
Virtual Machine:
If the VM is either Vista/Windows 2008 it's a simple reg query:
if (buildInt >= 6000)
{
sysInfo = RegistryKey.OpenRemoteBaseKey(RegistryHive.LocalMachine, srvName).OpenSubKey(@"SYSTEM\CurrentControlSet\Control\SystemInformation").GetValue("SystemProductName").ToString();
if (sysInfo.Contains("Virtual"))
{
vmCheck = 1;
}
}
If the VM is downlevel then it's a WMI query
else if (buildInt == 3790)
{
System.Management.ConnectionOptions objconn = new System.Management.ConnectionOptions();
objconn.Impersonation = System.Management.ImpersonationLevel.Impersonate;
objconn.EnablePrivileges = true;
System.Management.ManagementScope exmangescope = new System.Management.ManagementScope(@"\\" + srvName + @"\root\cimv2", objconn);
System.Management.ObjectQuery objquery = new System.Management.ObjectQuery("SELECT * FROM Win32_ComputerSystem");
System.Management.ManagementObjectSearcher objsearch = new System.Management.ManagementObjectSearcher(exmangescope, objquery);
System.Management.ManagementObjectCollection queryCollection1 = objsearch.Get();
foreach (System.Management.ManagementObject stringer in queryCollection1)
{
sysInfo = stringer["Model"].ToString();
//System.Console.WriteLine(sysinfo);
}
if (sysInfo.Contains("Virtual"))
{
vmCheck = 1;
}
Just some random stuff as you get ready for 2k8...
Getting the Classic cluster logs:
Clustering in Win2k8 has undergone some major changes (for the better). One of those changes is that the cluster events are now part of the event stream so sifting through the cluster logs is a thing of the past. You might find it easier sometimes though to have the cluster logs in which case you can generate them:
C:\>cluster log /G /Copy:"c:\debuggers"
Generating the cluster log(s) ...
The cluster log has been successfully generated on node 'server-10'...
The cluster log has been successfully generated on node 'server-11'...
The cluster log has been successfully copied from node 'server-11'...
The cluster log has been successfully copied from node 'server-10'...
The cluster log has been successfully generated on node 'server-15'...
The cluster log has been successfully copied from node 'server-15'...
The cluster log has been successfully generated on node 'server-16'...
The cluster log has been successfully copied from node 'server-16'...
The cluster log(s) have been copied to 'c:\debuggers'...
Multiple TS connections to the same server with the same account:
You may notice that in Win2k8 that if you are already logged on via TS to a server and use the same account from a different machine to connect to the server it will take over the session you already have connected instead of creating a new one. This is by default in 2k8. If you/your team use a test account to logon to your servers this could be quite annoying and you might want to set it back to what it was like in 2k3. You can do this by unchecking "Restrict each user to a single session" in tsconfg.msc, which just toggles the fSingleSessionPerUser value to zero under "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server" if you want to do it remotely.
Also, I already mentioned it but you need to use the /admin switch to connect to the console session with 2k8 and Vista SP1. More info from Terminal Services Team Blog.
Getting the system info for investigations:
Have a customer who is having issues? Stop asking questions and have them run msinfo32.exe /nfo c:\test.nfo and send you the test.nfo file. What's in there? Everything of your dreams. No really, it has a plethora of information on the system where its taken, and is quite helpful. If you just want to grab the basics from a server locally/remotely use systeminfo.exe which is under system32.
Setup failed and I do not know why:
For general troubleshooting, check the Setupact.log and Setuperr.log files. Depending on when the installation failed, these files will be located in the $WINDOWS.~BT\Sources\Panther folder or the Windows\Panther folder. In most cases, these folders are located on the partition that Windows Server 2008 is being installed on or the partition that contains the old operating system. However, if Setup failed on an Itanium-based computer, this folder might be located on another drive that has available hard disk space. From here.
I'd also add if you dont find any info in the panther log locations check the cbs.log file under %windir%\Logs\CBS. This has good information for any setup/install failures.
Installing Win2k8 and using it as your desktop:
For the uber-nerds cough *not me* cough: http://blogs.msdn.com/vijaysk/archive/2008/02/11/using-windows-server-2008-as-a-super-desktop-os.aspx
Windows 2008 is fast as hell, and if you got the horses you might think this is a good idea. IF you can live without sidebar! Oh wait, does anyone use that?
Microsoft Assessment and Planning (MAP) released yesterday:
Finally for those of you who want to scan your hardware inventory with zero-touch, the Solution Accelerator for 2k8 went out the door yesterday. If anything you should take the link and check it out.
The Microsoft Assessment and Planning Solution Accelerator performs three key functions - including hardware and device inventory, compatibility analysis, and readiness reporting.
Technorati Tags:
Windows 2008
You've probably heard that netmon3.1 is out, but you might not know that you can easily launch a capture at the command prompt. I find this useful when we're waiting on a repro, we want a capture, but we don’t know when that's going to happen. Sure you could set this up in the GUI too, but who wants to do that when it's as easy as this?
The below will setup a capture on all networks that the system is attached to and wait until I hit ctrl+c (you can see its been a while with no repro). The CHN extension used tells netmon to take multiple captures in a chain (see file syntax). I also put some examples at the bottom so you can see what else you can do. Have fun!
C:\Windows\system32>nmcap /capture /network * /File netmoncap.chn:100M
Netmon Command Line Capture (nmcap) 03.01.0512.0000
Saving info to:
C:\Windows\system32\netmoncap.cap - using chain captures of size 100.00 MB.
ATTENTION: Conversations Enabled: consumes more memory (see Help for details)
Exit by Ctrl+C
Saved Frames: 9232127 Capture Frames: 9438779 (44181 seconds)
Hit Ctrl+C
Cancelled by user
Final Results : Saved Frames: 722 Capture Frames: 722
C:\Program Files\Microsoft Network Monitor 3>dir netmoncap.cap
Volume in drive C has no label.
Volume Serial Number is FCC3-5AF7
Directory of C:\Program Files\Microsoft Network Monitor 3
02/22/2008 09:06 AM 384,748 netmoncap.cap
1 File(s) 384,748 bytes
0 Dir(s) 16,699,654,144 bytes free
Here's the breakdown fo the /File syntax:
/File <Capture File>[:<File Size Limit>]
Name of capture file to save frames to. Extensions are used to determine
the behavior of nmcap.
.cap -- Netmon 2 capture file
.chn -- Series of Netmon 2 capture files: t.cap, t(1).cap, t(2).cap...
<File Size Limit> are optional. It limits the file size of each capture
file generated. Default single capture file size limit is 20M. The
upper bound of the file size limit is 500M. The lower bound of the file
size limit depends on the frame size captured. (Note that the maximal size
of ethernet frames is 1500 Bytes)
The files are circular, so once the size limit is reached, new data will
overwrite older data.
Example Usage: /File t.cap:50M
Some other examples from the NMCAP help:
This example starts capturing network frames that DO NOT contain ARPs, ICMP,
NBtNs and BROWSER frames. If you want to stop capturing, Press Control+C.
nmcap /network * /capture (!ARP AND !ICMP AND !NBTNS AND !BROWSER) /File NoNoise.cap
Starts capturing network frames immediately. All TCP frames that have a source
port or destination port of 80 are saved to the chained capture files named
test.cap, test(1).cap, test(2).cap, ... When the user presses the 'x' key the
program stops.
nmcap /network * /capture tcp.port == 80 /file c:\temp\test.chn:6M /stopwhen /keypress x
This example starts capturing network frames that are TCP Continuations. The
capture filter is searching for String "Continuation in TCP Frame Summary
Description. In order to see the complete list of Netmon Properties that are
filterable,type ".Property" in the Netmon Filter UI.
nmcap /network * /capture contains(.Property.Description, \"Continuation\") /File TCPContinuations.cap
System Page Table Entry (PTE) issues are some of the top support issues for servers that run large server applications and have a relatively large amount of Random Access Memory (RAM). PTEs are structures used to track pages of RAM, similar to the way a telephone number is used to track a telephone to a specific location.
You can now track down those 3f bugchecks using !sysptes using the public symbols. Usually when we hit a server running out of system PTEs, it will just tip over and we don't see an actual bugcheck.
Prior to Vista SP1/Windows 2008 if you tried to run SYSPTES 4 on a server with public symbols you'd get this error message: "Unable to get System PTE individual lock consumer information". Well Windows Dev has fixed the bug. Below is an example of what we would typically see, and then how we'd use !sysptes to narrow down who is consuming the space.
You find this sort of output in !VM:
0: kd> !vm 1
*** Virtual Memory Usage ***
Physical Memory: 999242 ( 3996968 Kb)
Page File: \??\C:\pagefile.sys
Current: 927744 Kb Free Space: 884312 Kb
Minimum: 927744 Kb Maximum: 927744 Kb
Page File: \??\E:\pagefile.sys
Current: 3072000 Kb Free Space: 3024624 Kb
Minimum: 3072000 Kb Maximum: 3072000 Kb
Available Pages: 265887 ( 1063548 Kb)
ResAvail Pages: 933615 ( 3734460 Kb)
Locked IO Pages: 1679 ( 6716 Kb)
Free System PTEs: 500 ( 2000 Kb)
********** Running out of system PTEs **************
******* 416179544 system PTE allocations have failed ******
Free NP PTEs: 1630 ( 6520 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 478 ( 1912 Kb)
Modified PF Pages: 477 ( 1908 Kb)
NonPagedPool Usage: 8814 ( 35256 Kb)
NonPagedPool Max: 32351 ( 129404 Kb)
PagedPool 0 Usage: 10590 ( 42360 Kb)
PagedPool 1 Usage: 994 ( 3976 Kb)
PagedPool 2 Usage: 958 ( 3832 Kb)
PagedPool 3 Usage: 972 ( 3888 Kb)
PagedPool 4 Usage: 931 ( 3724 Kb)
PagedPool Usage: 14445 ( 57780 Kb)
PagedPool Maximum: 54784 ( 219136 Kb)
Shared Commit: 4163 ( 16652 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 10930 ( 43720 Kb)
PagedPool Commit: 14485 ( 57940 Kb)
Driver Commit: 1963 ( 7852 Kb)
Committed pages: 747541 ( 2990164 Kb)
Commit limit: 1952440 ( 7809760 Kb)
So what now? If you don't have the registry value set below, well for all intents and purposes you SOL. So reboot, set the value, and then wait for repro. Usually when we see the issue it comes back pretty quickly as some driver is eating up the space.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
trackptes REG_DWORD 0x1
Once the waiting is over and the system tips over, run !sysptes 4 and it will tell you what is allocating the PTEs and how many per call. The “!SYSPTES 4” command only lists driver PTE allocations. This is because, historically, drivers have made the most use and misuse of system PTEs. Sometimes you'll find one heavy hitter with a huge count (like in the article linked below), or in other instances you might find a certain sequence allocating many times, in either case you now have a clue as to who is using the PTEs and can either investigate that driver via break points, or contact the vendor who is eating up all the PTEs.
Cleaned up a bit for sanity's sake:
2: kd>!sysptes 4
VA MDL PageCount Caller/CallersCaller
f0769080 fce7fb18 2 mrxsmb+0x2bed2/mrxsmb+0x2da71
efbbb8b8 fce0f658 2 rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
f1c17080 fd0eb7a8 2 mrxsmb+0x2bed2/mrxsmb+0x2da71
eff41820 fd41bc70 2 rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
f1d10080 fcd91950 2 mrxsmb+0x2bed2/mrxsmb+0x2da71
f027f108 fd051f88 2 rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
efbf7080 fd7f3e80 2 mrxsmb+0x2bed2/mrxsmb+0x2da71
f1cede10 fce71460 2 rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
ef8a8080 fcedde80 2 mrxsmb+0x2bed2/mrxsmb+0x2da71
ef963730 fc9c2868 2 rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
f0281080 fccc52c0 2 mrxsmb+0x2bed2/mrxsmb+0x2da71
f1dfaff8 fd156650 2 rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
f0141080 fc6e82c0 2 mrxsmb+0x2bed2/mrxsmb+0x2da71
ef8f3508 fd003a30 2 rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
ef637080 fee1dde0 2 mrxsmb+0x2bed2/mrxsmb+0x2da71
eff3fa18 fd304050 2 rdbss!RUserBuffer+0x2f/rdbss!UserBufferForLowIo+0x2b
The actual process of tracking down the PTEs is thoroughly explained here: Detection, Analysis, and Corrective Actions for Low Page Table Entry Issues
So next time your system is acting up, get in there with Live KD and see what's going on!
Performance Analysis of Logs (PAL) tool
Project Description:
Ever have a performance problem, but don't know what performance counters to collect or how to analyze them? The PAL (Performance Analysis of Logs) tool is a new and powerful tool that reads in a performance monitor counter log (any known format) and analyzes it using complex, but known thresholds (provided). The tool generates an HTML based report which graphically charts important performance counters and throws alerts when thresholds are exceeded. The thresholds are originally based on thresholds defined by the Microsoft product teams and members of Microsoft support, but continue to be expanded by this ongoing project. This tool is not a replacement of traditional performance analysis, but it automates the analysis of performance counter logs enough to save you time. This is a VBScript and requires Microsoft LogParser (free download).
My take on the tool:
For those of us out there that don't have to deal with performance data on a daily basis I see a few options to help troubleshoot performance issues on your servers. 1) If your using 2k3 use SPA. 2) If you're running 2k8/Vista use data collection sets. 3) Collect analyze your own perfmon captures.
Now you might want to look into this tool. I found the tool simple to use and it's really a four step process. The web page created for the analysis has a plethora of info and links to the codeplex site for more info. Sweet.
Really in the end it's just a time saver. After collecting performance data on a server you need to analyze that data. This entails opening the log file, adding the counters that you've collected and finding out if any of the counters are above any thresholds (deemed by you). This tool does that analysis for you. It comes out-of-the-box with some predefined thresholds defined as high according to the MSFT consulting/development but those can be adjusted to whatever suits your fancy.
Once you get everything installed its time to do some analysis. It comes with some threshold templates for AD, System Overview, IIS, SQL, Exchange, etc (see pic) You point the app at the performance log you've captured during your perf issue, choose a threshold template to your liking, answer some basic questions, add the form and execute:
Once it completes it generates a webpage with the analysis information you desire. The webpage shows you alerts for activity that it finds suspect and graphs for the different areas of interest. I can't paste all the pics/info in here as it is quite lengthy depending on the interval you provide. But this definitely seems like a tool that could be handy down the road. Looking at the web page it looks really similar to SPA, but with graphs provided via the Office Web Components add-in. For example here is how I could find out LDP was using too much CPU:
First I found the alert which said that something was being excessive and I clicked on the link (sorry for the blurriness):
Then I found LDP consuming the CPU:
Add it to your bag of tricks, hope it helps.
