Brad Rutkowski's Blog : DS

NTDS performance counters missing

Brad Rutkowski — Fri, 20 Mar 2009 00:28:52 GMT

Thought I’d doc this for any others who run into this issue. I had to demote/promote a machine this morning and when it finished promoting I found it was missing all the NTDS\* counters in perfmon.

I ran LODCTR /Q and saw that it looked wrong:

C:\Windows\system32>lodctr /q:NTDS
Performance Counter ID Queries [PERFLIB]:
    Base Index: 0x00000737 (1847)
    Last Counter Text ID: 0x00001792 (6034)
    Last Help Text ID: 0x00001793 (6035)

Looking into it further I found it was missing the whole performance key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Performance

Once I found that, I exported the performance registry key from another domain controller and imported to the server missing the values and ran LODCTR /R and then the counters were back where they belonged…

C:\Windows\system32>lodctr /q:NTDS
Performance Counter ID Queries [PERFLIB]:
    Base Index: 0x00000737 (1847)
    Last Counter Text ID: 0x00001794 (6036)
    Last Help Text ID: 0x00001795 (6037)

[NTDS] Performance Counters (Enabled)
    DLL Name: %systemroot%\system32\ntdsperf.dll
    Open Procedure: OpenNtdsPerformanceData
    Collect Procedure: CollectNtdsPerformanceData
    Close Procedure: CloseNtdsPerformanceData
    First Counter ID: 0x000009DE (2526)
    Last Counter ID: 0x000009DE (2526)
    First Help ID: 0x000009DF (2527)
    Last Help ID: 0x000009DF (2527)

Interacting with Data Collector Sets via Powershell

Brad Rutkowski — Thu, 19 Feb 2009 00:30:12 GMT

Background:

In an earlier post I talked about some new features for Windows 2008 and Vista. One of those new features that is often overlooked are the data collector sets (DCS). One particular role that leverages data collector sets is active directory. Active directory has put “hooks” into tracing that can really take a lot of the thinking out of the question “why is my domain controller sluggish”. For those of you still running Windows 2003 I go over a similar concept called Server Performance Advisor.

Anyways, you can play around with DCS by typing perfmon and then traversing to the section called Data Collector Sets (shocking). If you have performance issues, go here first as it’s like combining a netmon capture with a kernel trace and then handing you the smoking gun.

Challenge:

In my current role, we have a need to automate things quite a bit and so one of the actions I was looking at solving was collecting diagnostic information when a server is performing poorly. Usually when a high CPU alert comes in, someone would need to logon to the server and go to perfmon and start at DCS collection. More often is the case that by the time someone had been alerted and went to the server the sluggish behavior had subsided (the dreaded “close ticket, no problem found”).

My solution was to try and figure out a way to start a DCS collection remotely at the time of event so that the data was present when an actual human became engaged.

After some hard work, here is the code to do so! You can create your own XML file (your own DCS template) and pass it in, but more than likely you’ll be happy at just kicking off one of the built-in templates (AD/System Perf/System Diags).

Running it via powershell:

First, how to do it on the fly:

## PLA.dll lives under system32 on Vista and 2k8.  This will create a powershell com object.
$datacollectorset = new-object -COM Pla.DataCollectorSet
##This is the name of the predefined DCS collector.  It's read-only and will always be System\<something>
$name = "System\Active Directory Diagnostics"
##If you make the second param $null it will be the local machine.
$datacollectorset.Query($name,"serverA") 
$datacollectorset.start($false)
## Status ReturnCodes: 0=stopped 1=running 2=compiling 3=queued (legacy OS) 4=unknown (usually autologger)
$datacollectorset.status
##When you're ready to stop it call stop.
$datacollectorset.stop($false)
##If you call status here, it will probably be '2' for a while as the server compiles the report.
$datacollectorset.status

And like so, you started and stopped a collection for Active Directory on you’re local computer or a remote server! Like I said though, you can create you’re own templates too. You might want to do this if you want to setup a built-in template to be scheduled to run daily, or perhaps you want to send the data to a network location, run more tasks at completion, etc. If you do want to create a custom template then the code changes a bit:

$datacollectorset = new-object -COM Pla.DataCollectorSet
## If you're making you're own (shows up under user defined).  
$xml = get-content C:\custom.xml #You're custom exported XML file.
$datacollectorset.SetXml($xml)
##Commit codes: http://msdn.microsoft.com/en-us/library/aa371873(VS.85).aspx this is add or modify.  Can't do this on a system created PLA instances (read only).
$datacollectorset.Commit($DCSPath , $null , 0x0003)     
$datacollectorset.Query($DCSPath,$null)
$datacollectorset.start($false)
#Runs...
$datacollectorset.stop($false)

Scripting a solution:

Finally if you wanted to script this you could do something like what I’ve done below. This would collect for a desired interval (in seconds) and then when compilation completed display the path to the report. I wrote this in CTP3, but you can easily take the concepts and backport them. If the destination server is inaccessible, or you don't have permissions, then the script will blow up…

<#
    .SYNOPSIS
    This will fire up a PLA (Data Collector Set collection on a server and then copy it to the proper debug server
 
    .DESCRIPTION
    This is a proof of concept and only acceppts System defined collections.  No error handling so I hope you type well.

#>

##Inputs
[CmdletBinding()]
param(
   [Parameter(Mandatory = $true)]
   <#A system provided report to run like "System\System Performance", System\System Diagnostics, etc. #>
   [string]$DCSPath,
   [Parameter(Mandatory = $true)]
   <# This is how long you want the DCS collection to run in seconds#>
   [int32]$time,
   [Parameter(Mandatory = $false)]
   <#If you don't pass in a server name it will be $null and run on the local system#>
   [string]$serverName
    )

    $datacollectorset = new-object -COM Pla.DataCollectorSet  
    $datacollectorset.Query($DCSPath,$serverName)
    $datacollectorset.start($false)
    Start-Sleep $time
    $datacollectorset.stop($false)
    
    ##Now we'll loop while the report compiles.
    $retries = 0
    do 
        {sleep 30; $returnCode = $datacollectorset.Status ; $retries++} 
    while ($returnCode -eq 2 -and $retries -lt 60)
    
    if ($retries -eq 60)
    {
        Write-Warning "Compiling has been running on the server for 30 minutes!  You'll need to check the following location on the server later for the report:"
        Write-Warning $datacollectorset.OutputLocation
        break
    }
    
    ##Compiling has finished, now we can copy the folder to some location
    $path = $datacollectorset.OutputLocation
    if ($serverName)
    {
    $path = $path.Replace(":","$")
    Write-Host "`nReport complete and can be viewed at \\$serverName\$path\report.html on the server.`n" 
    }
    else
    {
    Write-Host "`nReport complete and can be viewed at $path\report.html`n"
    }

The result:

More info:

PLA reference: http://msdn.microsoft.com/en-us/library/aa372634(VS.85).aspx

Technorati Tags: Powershell,Windows 2008,Active Directory,Windows

Domain doesn't know about my computer account? I vouch for my computer, you can trust me...

Brad Rutkowski — Fri, 01 Aug 2008 22:31:11 GMT

Had an issue where a server would not allow logon via termian services each time you attempted to logon it would return this:

Soooooooooo, what to do here?

First, we made sure the account existed in the directory since that's why it appeared to be complaining. So I opened LDP and verified it existed, and that all "checked out" with being healthy (stare and compare against a good object).

Second thing we did was crank up netlogon debug logging (nltest dbflag) and see what it showed. It was complaining of a lot of stuff but nothing conclusive unfortunately. So at that point it was time to move to event viewer. The "nice" thing about this issue was that the server was accessible via the network with the same account that was failing to TS so I could do some of the investigation remotely.

One event in particular struck me:

Log Name:      System
Source:        Microsoft-Windows-Security-Kerberos
Date:          7/31/2008 4:11:24 PM
Event ID:      3
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      BRAD-SRV-01.braddom.bradforest.com
Description:
A Kerberos Error Message was received:
on logon session
Client Time:
Server Time: 23:11:24.0000 7/31/2008 Z
Error Code: 0x7 KDC_ERR_S_PRINCIPAL_UNKNOWN
Extended Error: 0xc0000035 KLIN(0)
Client Realm:
Client Name:
Server Realm: braddom.bradforest.COM
Server Name: host/BRAD-SRV-01.braddom.bradforest.com
Target Name: host/BRAD-SRV-01.braddom.bradforest.com@braddom.bradforest.COM
Error Text:
File: 9
Line: d86
Error Data is in record data.

Using err.exe I resolved the error code and found there was a collision:

C:\localbin>err 0xc0000035
# for hex 0xc0000035 / decimal -1073741771 :
STATUS_OBJECT_NAME_COLLISION ntstatus.h
# Object Name already exists.
# 1 matches found for "0xc0000035"

At this point it's time to look for a collision of "host/BRAD-SRV-01.braddom.bradforest.com" in the forest. The easiest way to do it is use a nice script called querySPN.vbs.

C:\localbin>querySPN.vbs HOST/BRAD-SRV-01.braddom.bradforest.com braddom.bradforest.com
Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.

CN=VL Account,CN=Users,DC=braddom,DC=bradforest,DC=com
Class: user
User Logon: VLSBST
-- host/BRAD-SRV-01.braddom.bradforest.com <----------------------------------------------------------------- Bingo the SPN is registered for two objects!

CN=BRAD-SRV-01,CN=Computers,DC=braddom,DC=bradforest,DC=com
Class: computer
Computer DNS: BRAD-SRV-01.braddom.bradforest.com
-- TERMSRV/BRAD-SRV-01.braddom.bradforest.com
-- TERMSRV/BRAD-SRV-01
-- HOST/BRAD-SRV-01
-- HOST/BRAD-SRV-01.braddom.bradforest.com <-----------------------------------------------------------------

Once we removed the SPN from the user account, logons began to immediately work.

-B

Supported upgrades for domain controllers to Windows 2008 (Melting Pot in CorpNet)

Brad Rutkowski — Thu, 24 Jan 2008 07:00:23 GMT

Currently we are running Win2k3 SP1, R2, SP2, Win2k8 Beta3, RC0, RC1, and RTM Escrow idomain controllers in production... Since we're running some downlevel servers in the environment and I was interested in what is supported to be upgraded to Windows 2008 when we sign off and the the DVDs start getting pressed. Luckily a mail came by on one of our discussion aliases that has the details I (we) desire.

You can have DCs with down-level OS down to Windows 2000 SP4 in the same forest along with WS2008 DCs.
This means you can have forests with a mix of WS2008, WS2003 SP2, WS2003 R2, WS2003 SP1 and Win2K SP4 (please have in mind that this depends on the forest and domain functional levels).
If you have a down-level only forest (i.e. no WS2008) and want to introduce a new WS2008 you will need to run ADPrep (ForestPrep and DomainPrep).
You can run ADPrep having down-level OS down to Win2K SP4, you don’t need to have all of them with WS03 SP2.
However if you are going to in-place upgrade any of the down-level DCs, these have to be at least WS2003 SP1.

Refs:

Upgrading Active Directory Domains to Windows Server 2008 AD DS Domains

What Service Packs can be upgraded to Windows 2008

What do you say you DO here?

Brad Rutkowski — Fri, 21 Dec 2007 22:21:10 GMT

Just noticed the AD jigsaw poster has been updated for 2k8. So next time your boss asks you "What do say you do here?", don't reply with I'm a people person! Do the following:

1) Download one of the jigsaw posters from http://www.microsoft.com/downloads/details.aspx?FamilyID=c2b9e44e-0bbd-47cb-bc09-b3d48be7f867&DisplayLang=en

2) Print out on plotter

3) Give to boss

4) Ask for raise

We used to have one of these hanging outside our office, and as the rotating bosses would come by to see what we did, we'd just point them to the poster. Wonder where that thing is now...

Technorati tags: Active Directory, Windows 2008

Booting into DSRM in Windows 2008

Brad Rutkowski — Mon, 01 Oct 2007 23:40:05 GMT

Since the boot.ini file no longer exists in Windows 2008, the way to boot into directory service repair mode has changed. You can setup the OS to boot to DSRM a couple of ways as shown below. I'd also mention that to due offline defrags and other NTDSUTIL commands against the database you can now just stop NTDS from the services.msc snap-in. So the main reason to get into DSRM now would be to do an auth-restore of an object.

GUI:

Type MSCONFIG in the start men and go the the boot tab.

CMD prompt:

C:\Users\Administrator.BRAD-DC-04>bcdedit /set safeboot dsrepair

The operation completed successfully.

To restart the server normally, type the following command:

C:\Users\Administrator.BRAD-DC-04>bcdedit /deletevalue safeboot

The operation completed successfully.

Webcast coming up: Looking cool in front of your AD peers in Win2k8

Brad Rutkowski — Wed, 19 Sep 2007 21:12:23 GMT

Just got back from vacation and will start getting the posts going again. Just wanted to mention a webcast coming up from AD administration in Windows 2008 that is going to play next Thursday the 27th. If you got some time to kill, I'd make this session for sure.

TechNet Webcast: Extending Windows Server 2008 Active Directory Management with Windows Features (Level 300)

Event Viewer:

Join us to learn about the capabilities in the Windows Vista and Windows Server 2008 operating systems that enable new management scenarios. We explain how system administrators of the Active Directory directory service can use these capabilities to ease their day-to-day life and ensure smoother deployments and monitoring of their infrastructures. In this session, we briefly review the Microsoft WS-Management implementation under Windows and discuss how, as an Active Directory administrator, you can use its new set of features securely. Through practical examples, we demonstrate how the new Windows Remote Management (WinRM), Windows Remote Shell, and event forwarding features can address management needs for Active Directory Server Core deployment and Active Directory monitoring. Attend this presentation to discover how you can accomplish these tasks with a set of features directly available from the operating system!

-Brad

Technorati tags: Windows 2008, Webcast, Technet

Republish printers easily on a print server to Active Directory.

Brad Rutkowski — Mon, 25 Jun 2007 22:16:04 GMT

Printers can get pruned from the directory for many reasons. The way it is supposed to work is if the printer is stale then a DC will remove the print queue object from the directory after trying to contact it 3 times at 8 hour intervals (default). This also means that if a DC can't net view the print server for a 24 hour period it could potentially prune the print queue objects too. This can happen if one of your domain controllers are in a "bad" state where its online but not functioning as expected.

So what can you do once the print queue objects have been removed? Well to easily republish them you can create a simple script like below. You can then save this as a vbs and then use it to republish the printers in the directory.

If WScript.Arguments.Count <> 1 then
strPC = GetPC()
Else
strPC = wscript.arguments(0)

end if

Set objWMIService = GetObject("winmgmts:\\" & strPC & "\root\cimv2")
Set colItems = objWMIService.ExecQuery("Select * from Win32_Printer",,48)
For Each objItem in colItems
    Wscript.Echo "ShareName: " & objItem.ShareName
     objItem.Published = False
      objItem.Put_
      Wscript.Echo "Published: " & objItem.Published
     objItem.Published = True
      objItem.Put_
      Wscript.Echo "Published: " & objItem.Published
Next

function GetPC()
GetPC = InputBox ("What Server would you like republish the printers on?", "Servername")
End function

To work around this you could do a number of things:

1) Fix the network connectivity issue

a) Find out what Dc is not working as expected and resolve.

2) Disable the spooler service on your DCs

a) Could have lots of stale printers to manually clean up if you have printers in flux in your environment

3) Disable pruning via GPO

a) Set the Directory Pruning Interval value to Never via GPO

b) There will be stale printers in the directory and they will need to be manually cleaned up.

4) You can allow the printers to be pruned and set the Check Published State policy for specific (or all) print servers in the domain. This policy causes the spooler on a print server to periodically verify that its published printers exist in Active Directory. By default, the Spooler service verifies the state of published printers only when it is started.

a) Because the widespread use of this policy on many computers in the domain (that are constantly checking the publication status of their PrintQueue objects in Active Directory) can adversely affect network performance. Microsoft recommends that you set this policy only on the main production print servers.

If you want to know more about how printers are published and pruned this is a thorough article on the subject.

Windows Server 2008 Beta3: Can and RODC be a GC?

Brad Rutkowski — Wed, 30 May 2007 20:40:44 GMT

Answer: Yes.

If you're looking to deploy some RODCs during the B3 timeframe then it would be a good idea to read through this first:

Step-by-Step Guide for Read-Only Domain Controller in Windows Server 2008 Beta 3

One of the gotchas before an RODC will advertise as a GC in your domain is that domainprep needs to be run in each domain, regardless if there are Win2k8 DCs in the domain or not:

If the RODC will be a global catalog server, you must also run adprep /domainprep in all domains in the forest, regardless of whether the domain runs a Windows Server 2008 domain controller. When you run adprep /domainprep in all domains, the RODC can replicate global catalog data from all domains in the forest and then advertise as a global catalog server.

If you haven't looked at RODCs for your branch office deployments for the future now is a good time to do so. I think one of the best things coming for Win2k8 is the ability to run RODCs on Server Core, reducing the attack surface and patching requirements and only caching the passwords for the users needed in the branch site instead of all passwords for the domain.

The step-by-step guide I pointed out above has a plethora of info on RODCs, what they can do, and how to set them up, and how to take advantage of their new features. Do yourself a favor and read it/reference it, if your a DS Admin.

Technorati tags: DS, Directory Service, Windows Server 2008, Longhorn

Sync a partition from one DC to another DC when they don't have a direct replication link. And other Repadmin fun...

Brad Rutkowski — Sat, 21 Apr 2007 01:43:50 GMT

REPADMIN /ADD

I've shown you the beauty of REPADMIN /REPLSUM * /BYSRC /BYDEST /SORT:ERROR to easily find out the largest replication deltas in your forest. Lets take a look at a couple of other things you can do with repadmin, that I use quite a bit.

When I was running the DCs, it was not an uncommon occurrence for me to investigate and troubleshoot and issue locally here in Redmond even though the user account that was affected was located on the other side of the world. Why? Because time is money, and I like low latency when I'm TS'ing to a DC, or poking around in the directory over the network.

Well we could wait a couple of hours for the forest to converge, but usually that person waiting for you to fix the error is at a work stoppage until it is fixed, so what can we do?

We can sync two DCs that are not normally replication partners! Disclaimer: Even though I used this technique for over two years in production without issue, I make no guarantees.

First off you have to open the wonderful world of experthelp: repadmin /experthelp. We're looking for the add switch but you can see there is other fun commands in here for removing lingering objects and rehosting partitions.

Here's the command:

repadmin /add dc=dom,dc=forest,dc=test JAPAN-DC-01.dom2.forest.test REDMOND-DC-01.dom.forest.test /readonly

This will create a one-time replication link between the two DCs, sync the partition I've specified, and then delete the replication link. If you're syncing a GC partition, then use the /readonly switch which I've done above. I'm sure you'll find a use for this, if your managing a large enterprise with a lot of DCs.

REPADMIN /SYNCALL

Another one of my favorites: Repadmin /syncall /edjQSA (case sensitive)

What if you wanted to sync every partition on a DC/GC with each of its adjacent partners, no matter if those partners were local or over wan links? Basically you made some changes and now you want to sync them up so that you don't have to wait for replication? If you run "Repadmin /syncall /?" you can see the syntax breakdown of the command.

Before sync, you can see the delta for this Dc and it's partners is ~39 minutes.

C:\>repadmin /replsum BRAD-dc-01
Replication Summary Start Time: 2007-04-20 15:32:07

Beginning data collection for replication summary, this may take awhile:
....

Source DSA largest delta fails/total %% error
SONJA-DC-01 39m:22s 0 / 4 0
SONJA-DC-04 39m:22s 0 / 11 0
BRAD-DC-02 39m:22s 0 / 13 0
BRAD-DC-14 39m:22s 0 / 5 0
BRAD-DC-15 39m:22s 0 / 13 0
BRAD-DC-27 39m:22s 0 / 13 0
BRAD-DC-36 39m:22s 0 / 13 0
BRAD-DC-38 21m:52s 0 / 13 0

Destination DSA largest delta fails/total %% error
BRAD-DC-01 39m:22s 0 / 85 0

Use the command to sync

C:\>repadmin /syncall BRAD-dc-01 /edjQSA
Syncing all NC's held on BRAD-dc-01.
Syncing partition: DC=DomainDnsZones,DC=red,DC=dom,DC=forest,DC=test

Syncing partition: DC=ForestDnsZones,DC=dom,DC=forest,DC=test

Syncing partition: DC=red,DC=dom,DC=forest,DC=test

Syncing partition: CN=Schema,CN=Configuration,DC=dom,DC=forest,DC=test

Syncing partition: CN=Configuration,DC=dom,DC=forest,DC=test

Syncing partition: DC=midway,DC=dom,DC=forest,DC=test

Syncing partition: DC=afr,DC=dom,DC=forest,DC=test

Syncing partition: DC=sp,DC=dom,DC=forest,DC=test

Syncing partition: DC=eu,DC=dom,DC=forest,DC=test

Syncing partition: DC=fur,DC=dom,DC=forest,DC=test

Syncing partition: DC=st,DC=dom,DC=forest,DC=test

Syncing partition: DC=nwd,DC=dom,DC=forest,DC=test

Syncing partition: DC=can,DC=dom,DC=forest,DC=test

Now you can see the delta for all the partners is near zero.

C:\>repadmin /replsum BRAD-dc-01
Replication Summary Start Time: 2007-04-20 15:32:20

Beginning data collection for replication summary, this may take awhile:
....

Source DSA largest delta fails/total %% error
SONJA-DC-01 :03s 0 / 4 0
SONJA-DC-04 :03s 0 / 11 0
BRAD-DC-02 :04s 0 / 13 0
BRAD-DC-14 :04s 0 / 5 0
BRAD-DC-15 :04s 0 / 13 0
BRAD-DC-27 :03s 0 / 13 0
BRAD-DC-36 :04s 0 / 13 0
BRAD-DC-38 :04s 0 / 13 0

Destination DSA largest delta fails/total %% error
BRAD-DC-01 :05s 0 / 85 0

Technorati tags: Active Directory, DS, Windows, Windows 2003, AD

New Dcdiag switches for Windows 2003 SP2

Brad Rutkowski — Mon, 12 Mar 2007 19:17:59 GMT

If you want to see what else is coming your way fro SP2 check out this link.

This is nice, because we all know when you scan your enterprise for the DNS tests in can be a bit verbose....

New options have been added to the Dcdiag.exe Domain Name System (DNS) tests. These new options are /x and /xsl:xslfile.xsl or /xsl:xsltfile.xslt. They generate XML tags when the tests are run with the /test:dns option. You can use this new output mechanism to more easily parse the verbose log that the DNS tests generate.

To direct the XML output file to XMLLog.xml, use the /x option. For example:

dcdiag /test:dns /v /e /x:XMLLog.xml

Note: The /x: option only works with the /test:dns option.

To add the processing instructions that reference the specified style sheet, use the /xsl:xslfile.xsl or /xsl:xsltfile.xslt option. For example:

dcdiag /test:dns /v /e /x:XMLLog.xml; /xsl:xslfile.xsl

dcdiag /test:dns /v /e /x:XMLLog.xml; /xsl:xsltfile.xslt

Note: The /xsl:xslfile.xsl or /xsl:xsltfile.xslt option only works with the /test:dns /x:XMLLog.xml option.

Technorati tags: AD, Directory Service, Windows

Can you have a mix of 32 and 64 bit domain controllers?

Brad Rutkowski — Thu, 08 Feb 2007 20:43:43 GMT

I see this questions come up quite a bit about the interoperability of x86 and x64 domain controllers. Does replication work? Do the tools cross over well? Any gotchas that we should know about? Etc. Well I'm here to tell you that here at Microsoft we've been running a mix of 32 and 64 bit domain controllers since beta Wk23 SP1, and I can report back to you "Don't worry about interoperability of the domain controllers".

Replication works as expected.

Remote management tools connect the same.

When you TS everything feels the exact same.

You wont even know if its x64 or x86 unless you open up task manager and check out how much memory LSASS is using...

When I have engagements with customers, they ask if they should upgrade to x64 in their environment. My response is: that depends. Should you buy x64 compatible hardware? Yes, the cost difference is not that drastically different and this prepares you to install an x64 OS in the future. At the same time, an x86 OS installs and works as expected on the hardware, although you don't get all the nice benefits of running in true x64.

Do you need to spend tons of extra memory so that the DIT can be cached? That depends, and you can read this article for an explanation.

-B

How to frisk a DC when people are complaining of "Authentication Issues".

Brad Rutkowski — Sat, 02 Dec 2006 06:22:00 GMT

At Microsoft we do quite a bit of dogfooding (imagine that) and in doing so we run into issues in the infrastructure and a lot of the time they crop up as "authentication issues". For example, users can't get to a website, a share, e-mail, etc. The symptoms can be varied and the outcome is the same, angry people at your door (sometimes literally).

So in these situations how can you find out if a DC is misbehaving or give the "all clear" for the directory service and tell them to go look elsewhere? Well here are some of the common things I check and do in this situation.

1) Portqry.exe the ports 389 (LDAP), 3268 (GC), 445 (Microsoft-DS), 139 (NetBIOS), and 88 (Kerberos). This should give us an idea if the basics are working as far as the DC listening as it should. Sometimes I through out a nbtstat -A against the IP of the DC for good measure.

2) Use Nltest and Tail.exe in tandem. I like to do this remotely so I never actually have to TS (actually that goes for pretty much everything on this site). I run nltest /server:BRAD-DC-01 /dbflag:2080FFFF. This will turn on the netlogon debug logging. You can turn this on the server where authentication issues are happening as well as the DC its pointed to. Then use tail.exe (part of the resource kit tools) to watch the file in real time. Now you can watch all the stuff go by in the netlogon log, or you can utilize findstr to just look for errors.

C:\Localbin>tail -f \\brad-dc-01\admin$\Debug\Netlogon.log |findstr /i Critical
12/01 18:45:10 [CRITICAL] BRADDOM: NlGetIncomingPassword: server.bradddom.brad.com: cannot LsarQueryTrustedDomainInfoByName 0xc0000034
12/01 18:45:13 [CRITICAL] BRADDOM: NlGetIncomingPassword: server.bradddom.brad.com: cannot LsarQueryTrustedDomainInfoByName 0xc0000034
12/01 18:45:23 [CRITICAL] BRADDOM: NlGetIncomingPassword: Can't NlSamOpenNamedUser for machine3$ 0xc0000064.
12/01 18:45:23 [CRITICAL] BRADDOM: NetrServerAuthenticate: Can't NlGetIncomingPassword for machine3$ 0xc0000064.
12/01 18:45:23 [CRITICAL] Ping from Brad-DC-01 for domain brad-dc-01.BRADDOM.brad.com (null) for (null) on <Local> is invalid since we don't host the named domain.
12/01 18:45:26 [CRITICAL] BRADDOM: NlGetIncomingPassword: Can't NlSamOpenNamedUser for Baller$ 0xc0000064.
12/01 18:45:26 [CRITICAL] BRADDOM: NetrServerAuthenticate: Can't NlGetIncomingPassword for Baller$ 0xc0000064.

We can then use err.exe (which I mentioned in my blog earlier) to look up some error codes of interest...

C:\Debuggers>err 0xc0000064
# for hex 0xc0000064 / decimal -1073741724
STATUS_NO_SUCH_USER ntstatus.h

ERROR_DUP_NAME winerror.h
# You were not connected because a duplicate name exists on
# the network. Go to System in Control Panel to change the
# computer name and try again.
# 2 matches found for "0xc0000034"

3) Net view the server would be a good idea, depending on the error code it might point to different things. For instance if the DC is pinging but not responding to net view, it could be the firewall that you should look into, or perhaps IPSEC.

4) repadmin commands. I use a few of these to get a feel for if the DC is in sync and everything's cool from a replication standpoint.

C:\Localbin>repadmin /replsum brad-dc-01 /bysrc /bydest /sort:error
Replication Summary Start Time: 2006-12-01 18:52:35

Beginning data collection for replication summary, this may take awhile:
....

Source DC largest delta fails/total %% error
brad-dc-15 01d.11h:07m:03s 8 / 13 61 (1256) The remote system is not available. For information about network troubleshooting, see Windows Help. //DC offline
Sonja-DC-04 11h:56m:04s 4 / 11 36 (1256) The remote system is not available. For information about network troubleshooting, see Windows Help. //DC offline
brad-dc-14 42m:39s 0 / 5 0
brad-dc-02 42m:39s 0 / 13 0
brad-dc-03 42m:39s 0 / 13 0
brad-dc-12 50m:51s 0 / 13 0
brad-dc-19 42m:39s 0 / 13 0
brad-dc-25 42m:39s 0 / 13 0
CORP-DC-07 42m:39s 0 / 11 0

Destination DC largest delta fails/total %% error
brad-dc-01 01d.11h:07m:03s 12 / 105 11 (1256) The remote system is not available. For information about network troubleshooting, see Windows Help.

C:\Localbin>repadmin /queue brad-dc-01
Queue contains 0 items.

C:\Localbin>repadmin /showoutcalls brad-dc-01
brad-dc-01 has 1 outgoing DRS RPC calls in progress:

Call type: DRS_CALL_REPLICA_SYNC
Target server: 32577452-1d08-467b-8dd7-2384458f93232._msdcs.bradddom.brad.com //Outgoing to call to sync to this DC, lets see what it is below
Handle info: bound 1 FromCache 1 InCache 1
Client thread id: 1252
Time call started: 2006-12-01 18:51:15
Call timeout: 5 minutes
Call duration: 1 minutes and 42 seconds

C:\Localbin>ping 32577452-1d08-467b-8dd7-2384458f93232._msdcs.bradddom.brad.com

Pinging Sonja-DC-04 [0000:4898:dc05:32:3456:8a45:4588:4323] from 0000:4898:dc05:23:3456:618c:3cc4:1234 with 32 bytes of data:

Reply from 0000:4898:dc05:32:3456:8a45:4588:4323: time<1ms // It's Sonja-DC-04 from the repadmin report above that's offline for whatever reason we should probably look into that server.
Reply from 0000:4898:dc05:32:3456:8a45:4588:4323: time<1ms

5) Check DNS! It seems to come back and bite us once in a while. If SRV records get scavenged or something else is messed up you can see some weird behavior. For instance, you could see a few DC's that are pegged at 100% CPU while the others are not loaded, or clients going to DC's outside of their site.

6) Use SPA. If you have it installed :) This will give you an idea if its network related that is causing load on the D, if that's your problem.

7) nltest /sc_query:domain /server:server. I should have mentioned this first. When you get the call that comes in indicating authentication issues with a particular resource, you should find out what that resource has for its secure channel and then start the frisking!

Now of course there are many other things you ca check, but this will at least give you peace of mind that a DC is healthy. Other tools include: evnetvwr, dcdiag, netdiag, replmon, NTDS diagnostics, Netmon, etc.

So what to do when you don't know what DC is messed up?

Well that's a bit more tricky. Usually the easiest method is to start with the resource that is affected instead of looking for the needle in the haystack. Again though repadmin is QUITE useful with error codes and could give you some clues.

Easiest way to use repadmin to check every DC in the forest?

REPADMIN /REPLSUM * /BYSRC /BYDEST /SORT:ERROR

Give it a shot and see if your AD infrastructure is online and healthy!

As always your comments are welcome...

Technorati tags: AD, Directory Service, Authentication, Domain Controller

Booting a DC into DSRM without touching it locally.

Brad Rutkowski — Fri, 20 Oct 2006 09:02:00 GMT

Short answer:

/safeboot:DSRepair

Long answer:

So you will run into it often when being an AD admin that you need to boot a DC into DSRM (Directory Service Restore Mode), so what is the easiest way of doing this without having someone stand at the console and hit F8 on reboot and select DSRM from the menu?

1) Reset the DSRM admin password, because it would really suck if you got to the logon screen and did not know the password to logon. You'll notice below that NTDSUTIL excepts shorthand...

C:\>ntdsutil
ntdsutil: set dsrm pas
Reset DSRM Administrator Password: res pa on serve brad-dc-01
Please type password for DS Restore Mode Administrator Account: ************
Please confirm new password: ************
Password has been set successfully.

2) Configure the boot.ini on the server, first we have to turn off all those attributes:

attrib -r -h -a -s \\brad-dc-01\c$\boot.ini

3) Now we can adjust the boot.ini remotely, I like to copy the orignal line and put it below my modified line so "just in case" we have our old string handy. All you need to do is add the line /safeboot:DSRepair

[boot loader]
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003 Enterprise x64 Edition" /fastdetect /sos /3GB /usepmtimer /safeboot:DSRepair
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003 Enterprise x64 Edition" /fastdetect /sos /3GB /usepmtimer <-- Original

4) Now use shutdown.exe to reboot the machine.

So you have officially never TS'd to the machine and when it starts pinging again, you'll be in DS repair mode.

Technorati tags: AD, active directory, Directory Service

Dumping out all the DCs in a domain to a txt file

Brad Rutkowski — Tue, 17 Oct 2006 20:47:00 GMT

Short and sweet way of dumping out the DCs to a txt file, in a script:

for /f "skip=1" %%a in ('netdom query dc /domain:YOURDOMAIN') do (if %%a == The (echo.) else echo %%a >> test.txt)

So what's the deal with all the syntax, and how would this be useful? Well if you do simple admin scripting then this is pretty useful, say you want to look at all the DCs in the forest and check to make sure that they all have a certain reg key (sample below), well you could create a simple cmd script in about 5 minutes using the above line and the use reg query to do the dirty work. Hope you can find a use for this, my next post will use this to demonstrate how to do some ghetto time skew monitoring on DCs.

Syntax breaks down like this:

/f is needed because we are using a command to pull the variable %a ('netdom query dc /domain:YOURDOMAIN')

"skip=1" We use this so that we skip the first line of the output from netdom query dc which looks like this:

C:\localbinx64>netdom query dc /domain:braddom
List of domain controllers with accounts in the domain: <-- Skips this line.

BRAD-DC-20
BRAD-DC-22
BRAD-DC-26
BRAD-DC-15
The command completed successfully. <-- Don't want this either see below on how we get around this.

(if %%a == The (echo.) else echo %%a >> test.txt) And what's all this? Well its my way of getting around the last line.

Here is a script that uses this technique and checks the strict replication key, I don't dump the servers to a txt file because hey I dont need to, just save this into a cmd file... Play around to figure our what the findstr does.

@echo off

for /f "skip=1" %%a in ('netdom query dc /domain:Yourdomain') do (
if %%a == The (echo.)
echo %%a
reg query \\%%a\HKLM\system\currentcontrolset\services\ntds\parameters /v "strict replication consistency" |findstr /i strict)

Output looks like:

C:\>strict.cmd
BRAD-DC-20
strict replication consistency REG_DWORD 0x1
BRAD-DC-22
strict replication consistency REG_DWORD 0x1
BRAD-DC-26
strict replication consistency REG_DWORD 0x1
BRAD-DC-05
strict replication consistency REG_DWORD 0x1
BRAD-DC-27
strict replication consistency REG_DWORD 0x1
BRAD-DC-10
strict replication consistency REG_DWORD 0x1
BRAD-DC-11
strict replication consistency REG_DWORD 0x1
BRAD-DC-25
ERROR: The system was unable to find the specified registry key or value.
BRAD-DC-24
ERROR: The system was unable to find the specified registry key or value.
BRAD-DC-35
strict replication consistency REG_DWORD 0x1
BRAD-DC-04
strict replication consistency REG_DWORD 0x1
BRAD-DC-03
strict replication consistency REG_DWORD 0x1
BRAD-DC-23
strict replication consistency REG_DWORD 0x1
BRAD-DC-14
strict replication consistency REG_DWORD 0x1
BRAD-DC-08
strict replication consistency REG_DWORD 0x1
BRAD-DC-18
strict replication consistency REG_DWORD 0x1
BRAD-DC-15
strict replication consistency REG_DWORD 0x1

Technorati tags: Active Directory, Directory Service, AD

IceRocket tags: Active Directory