Active Directory Blog : Network

Split IO and Intermittent “File Not Found” Errors

Tim Springston — Mon, 10 Aug 2009 16:30:00 GMT

There are a whole host of issues that are simply never seen unless you have a large distributed environment. I know that sounds startling but here’s a hypothetical example. Imagine that you are an online retailer and for every identity that you are transacting...(read more)

DCs and Network Address Translation

Tim Springston — Wed, 22 Apr 2009 16:11:00 GMT

A lot of planning goes into the features and capabilities of each Windows release. Over the years I’ve noticed that there is not a great deal of awareness out in the general public for just how much work and labor goes into a new version of Windows. We’ll...(read more)

When Smartcard Logon Doesn't

Tim Springston — Mon, 06 Apr 2009 17:48:00 GMT

Authentication is entering every facet of our lives nowadays. It is common to have multiple passwords: passwords for work, home email, and Internet websites to name a few. It’s easy to have a lot of different passwords, and equally easy to use only one...(read more)

Downgrade "Attack"? A little more info

Tim Springston — Fri, 20 Mar 2009 18:56:00 GMT

I decided that we needed some more detail and to give a walk through scenario on this downgrade attack deal I mentioned a while back in a blog post . As a recap, a customer called in after noticing the events below appearing intermittently but repeatedly-and...(read more)

Tabula Rasa

Tim Springston — Wed, 14 Jan 2009 17:15:00 GMT

I was well and truly stumped a few months ago. I joke that once a year I am flat out wrong, and rarely do I have nothing to say on a subject. The 'once a year I may be flat out wrong' statement may be true simply because after 15 years in the IT industry...(read more)

Fooling the DC Locator

Tim Springston — Fri, 02 Jan 2009 20:25:39 GMT

There are an ever increasing number of scenarios out there in the business world where two different companies, or company divisions, may be using Active Directory for their directory service but may not be setting up an actual trust between the two....(read more)

Scary Sounding Errors

Tim Springston — Fri, 12 Dec 2008 18:44:48 GMT

We have a temporary role in CSS where support folks will help out in supporting prerelease (also known as beta) software. I’ve worked a couple of Windows betas, and it’s a great experience. I mention this since I remember a few...(read more)

Name Hijacked, Bystander DC Hangs

Tim Springston — Mon, 24 Nov 2008 17:00:36 GMT

I learn more about AD and other things every day, which is part of the fun of this job we do-learning about how things work. This story does a good job of lending some understanding to something that can be tough to understand-trust secure channels. This...(read more)

Why! Won't! PAC! Validation! Turn! Off!

Tim Springston — Mon, 29 Sep 2008 16:00:00 GMT

A while back I wrote a blog post regarding PAC (Privilege Attribute Certificate) validation in Microsoft Kerberos. We’ve had enough interest in this lately, particularly around the idea of disabling it, that it seemed like a good idea to post about this...(read more)

Question about AD authentication, Put In Context

Tim Springston — Fri, 07 Mar 2008 17:34:00 GMT

Occasionally I am contacted with specific questions or topics people would like to hear more about. This post is a reply to one of those. Here’s the question: My question is what are the impact when I change the logon workstation property of a user account...(read more)

Server 2008 and Windows Vista: Encryption Better Together

Tim Springston — Fri, 02 Nov 2007 15:00:00 GMT

A while back I did a blog post about some problems that were seen with people testing Windows Vista and then “rolling back” to Windows XP and some problems that could be seen when using the same computer object (also known as account) in AD. If you didn’t get a chance to read it here’s the post. What that scenario highlighted was the added level of encryption, by way of leaving behind a little of the supporting infrastructure in the msds-supported-encryptiontypes attribute value. In this post we’re going to talk about the Big Picture of the new authentication encryption available and a few things to keep in mind....(read more)

All The Logging In The World

Tim Springston — Mon, 08 Oct 2007 19:40:00 GMT

There’s normal troubleshooting and then there’s the stuff you do when the basic troubleshooting doesn’t get things resolved. Normal troubleshooting can be things like selecting “last known good” on a reboot after installing a new driver and having a blue screen. Or perhaps uninstalling and then reinstalling an application, or altering settings for the application or operating system to alleviate a problem. Sometimes we have to dig in and find out more. Many admins out there in the world live that every day. Which is why we add methods to find out more into our products. This post is all about listing all of the data gathering methods that a Directory Services person may ever need to know. Since there are so many it will be difficult to organize well in one uber post but I’m going to put out here for you all anyway, disorganized or not....(read more)

Soliloquy for AD Logon

Tim Springston — Sat, 28 Jul 2007 07:15:00 GMT

Some readers may scoff a little when I talk about how under appreciated the whole Active Directory scheme is. Not schema but scheme. I’m talking about the entire client and server interaction and how they work together to provide all of the distributed services that make up Active Directory and user logon.

That’s a bold statement, you may say. Let me back it up with some general facts.

If you are going to have a secure business environment then you need to have a secure client computer (workstation) for your information worker to use. We have that in Windows Vista and earlier. These operating systems provide a logon mechanism to that workstation which can allow user identity verification by way of a network discussion with a centralized repository that keeps track of users and passwords and things like that. Without a validated logon in this manner no user shell is created, no Explorer will run, no access to resources in the environment can take place in an authenticated user’s context. Without things being done comprehensively in that context guaranteeing security for files, other objects and actions that can be done on that computer and on the network.

Of course that central repository is the Active Directory database. Think of this interaction as a symphony of various types of network traffic, each representing another part of the secure user logon experience our products provide which comes together as a harmonic to give to you the entire Microsoft Directory Services suite. Poetic, ain’t it?

So let’s go over a few of the types of network communication you’ll see when a domain computer starts up and a user logs on.

Finding a domain controller as the computer starts up is an important piece of things. If the domain client computer doesn’t know where to send its requests then nothing else will happen well. So Microsoft client computers, when they are joined to the domain, know that they can query the DNS zone which for that domain. Service, or SRV records, will answer DNS queries from the client computer with the IP address of a server that has LDAP, GC, Time and other services based on the closest site.

User logon (and the computer being authenticated at startup) with authentication of the user in a secure encrypted manner is provided by Kerberos network traffic from workstation to domain controller. You’ll see this in UDP and TCP traffic along port 88. Most parsers filter Kerberos easily (Wireshark.exe for example will identify all such traffic by typing Kerberos in the parsing line) and in a manner that shows not just the ticket requests and responses but also the various transport protocols which negotiate using Kerberos.

User logon with roaming profiles is basically comprised of a short query to a domain controller for location and then a whole lot of SMB traffic as the files are compared with anything that is already present on the client computer and then pulled down across the network. Depending on the size of the profile, and the use of folder redirection and client side caching, that can mean a lot of traffic.

Group Policy processing will first use LDAP queries to assess all of the policies which would apply to the user (or computer earlier on) based on where the user object is located in AD and what GPOs are linked to that object. This is followed by a client side assessment of whether the user has permissions to the GPO, and if it’s got the applicable type of settings (user or computer).

There will later be group policy traffic as the settings which the GPOs are giving are passed to the appropriate client side extensions for processing (scripts, folder redirection, software installation, registry, security, disk quota, EFS, Internet Explorer, and IP Security). These various CSEs, if they are going to process based on whether the GPOs say they should and if they haven’t already, will each use their own types of network traffic based on the remote resources they need to connect to in order to get their job done. Here’s a little not-so-secret: CSEs have their own logging to see what’s happening under the hood.

For Windows Server 2003, XP and 2000 you could see this on the client computer, with time indices, in the userenv log if you had it enabled. This was a pretty comprehensive way to see all that was happening on the client computer and when used in conjunction with a network trace you can really have a good understanding of what is happening on your client computer at computer startup or user logon. Let me confess, ahem, that Windows Vista and Server 2008 have this logging as well but it is not as easy for you, the customer, to gather useful data from it. That’s a topic for another day, but rest assured that Microsoft support is there for you to help glean data from it when needed.

Finally, let me end this diatribe by bringing some relevant links that may be useful to you when it comes to understanding and troubleshooting user logon…

How to enable user environment debug logging in retail builds of Windows

http://support.microsoft.com/kb/221833

Troubleshooting Group Policy Client-Side Extension Behavior

http://support.microsoft.com/kb/216358

And one of my favorite articles that is still largely applicable even in newer operating system releases:

Windows 2000 Startup and Logon Traffic Analysis

http://www.microsoft.com/technet/prodtechnol/windows2000serv/deploy/confeat/w2kstart.mspx

A Day at the SPA

Tim Springston — Mon, 09 Jul 2007 06:38:00 GMT

Ah, there’s nothing like the stop-everything, our-company-has-come-to-a-complete-halt emergency call we sometimes get where the domain controllers have slowed to a figurative crawl. Resulting in nearly all other business likewise emulating a glacier as well owing to logon and application failures and the like.

If you’ve had that happen to one of your domain controllers then you are nodding your head now and feeling some relief that you are reading about it and not experiencing that issue right this moment.

The question for this post is: what do you do when that waking nightmare happens (other than consider where you can hide where your boss can’t find you)?

Well, you use my favorite, and the guest of honor for this post: Server Performance Advisor. Otherwise known as SPA.

Think of SPA as a distilled and concentrated version of the Perfmon data you might review in this scenario. Answers to your questions are boiled down to what you need to know; things that are not relevant to Active Directory performance aren’t gathered, collated or mentioned. SPA may not tell you the cause of the problem in every case, but it will tell you where to look to find that cause.

So I’ve talked about the generalities of SPA, now let’s delve into the specifics. Well, not all of them, but an overview and the highlights that will be most useful to you.

SPA’s AD data collector is comprised of sections called Performance Advice, Active Directory, Application Tables, CPU, Network, Disk, Memory, Tuning Parameters, and General Information.

Before you reach all of the hard data in those sections, though, SPA gives you a summary at the top of the report. It’ll look something like this:

Summary

CPU Usage(%)

Top Process Group	CPU%
lsass.exe	98

Top Activity	CPU%
SamEnumUsersInDom	5

Top Client	CPU%
Rockyroaddc01.icecream.dairy.org	5

Top Disk by IO Rate	IO/sec
0	11

Performance Advice is pretty self explanatory and is one of the big benefits of SPA over other performance data tools. It’s a synopsis of the more common bottlenecks that can be found with an assessment of whether they are a problem in your case. Very helpful. It looks at CPU, Network, Memory and Disk I/O and gives a percentage of overall utilization, it’s judgment on whether the performance seen is idle, normal or a problem and a short detail sentence that may tell more.

The Active Directory portion gives good collated data and some hard numbers on AD specific counters. These are most useful if you already have an understanding of what that domain controllers baseline performance counters are. In other words, what the normal numbers would be for that domain controller based on what role it has and services it provides day to day. Generally speaking, though, SPA is most often used when a sudden problem has occurred, and so at that point establishing a baseline is not what it should be used for.

The good collated data includes a listing of clients with the most CPU usage for LDAP searches. Client names are resolved by FQDN and there is a separate are that gives the result of those searches.

AD has indices for fast searches and those indices can get hammered sometimes. The Application Tables section gives data on how those indices are used. The information this gives to you can be used to refine queries being issued to the database (if they were to traverse too many entries to get you a result for example) if you have an application that is doing that sort of thing, it can suggest that you need to index something new, or that you need to examine and perhaps fix your database using ntdsutil.exe.

The CPU portion gives a good snapshot of the busiest processes running on the server during the data gathering. Typically, this would show LSASS.EXE as being the busiest on a domain controller, but not always-particularly in situations where the domain controller has multiple jobs (file server, application server of some kind perhaps). Generally speaking, having a domain controller be just a domain controller is a good thing.

Note: If Idle has the highest CPU percentage then you may want to make sure you gathered data during the problem actually occurring.

The Network section is one of the most commonly useful ones. Among other things, this summarizes the TCP and UDCP client inbound and outbound traffic by computer. It also tells what processes on the local server were being used in conjunction with that traffic. Good stuff which can give a “smoking gun” for some issues. The remaining data in the Network section is also useful but we have to draw the line somewhere or this becomes less of a blog post and more like training.

The Disk and Memory sections will provide very useful data, more so if you have that baseline for that system to tell you what is out of the normal for it typically.

SPA is a free download from our site, and installs as a new program group. Here’s where you can get it (install does not require a reboot):

http://www.microsoft.com/downloads/details.aspx?familyid=09115420-8c9d-46b9-a9a5-9bffcd237da2&displaylang=en

A few other things to discuss regarding SPA.

· It requires Server 2003 to run.

· As I stated above, when you have a problem is the worst time to establish a baseline

· The duration of the test can be altered depending on your issue. The default time set for it is 300 seconds (5 minutes). Keep in mind that if you gather data a great deal longer than the duration of the problem then you run the risk of averaging out the data and making it not useful for troubleshooting.

· In the same way that there are ADAM performance counters, SPA has an ADAM data collector

· The latest version (above) includes an executable that can kick this off from a command line, and which can be run remotely via PsExec or similar.

· Server 2008 Perfmon will include SPA like data collectors…or so I hear.

· SPA will not necessarily be the only thing you do, but it’s a great starting place to figure out the problem.

See? A day at the SPA can really take the edge off of a stop-everything, our-company-has-come-to-a-complete-halt emergency kind of day. Very relaxing indeed.

A New Twist on Initial Delay in SSL Session Setup

Tim Springston — Wed, 20 Jun 2007 04:50:00 GMT

A while back I posted about troubleshooting a problem where a customer had seen a home grown application was not working as expected. The app was designed to run in a web site which the user would connect to from Internet Explorer on a client. The HTTP connection was secured by SSL.

The problem symptom was a 20 second delay when you first connected to the site before the client would see the web site “draw” after the client received the data. If you closed IE and then reconnected to the web site after that first delayed connection then the subsequent connections would not see a delay.

I went over that issue in detail in this blog post: http://blogs.technet.com/ad/archive/2007/01/03/enough-with-the-delays-already.aspx

Well, we recently saw an issue that, on the face of it, was the exact same thing. Unfortunately, the resolution we ultimately used for the prior case did not resolve the new one. That resolution was to check for permissions insufficient to allow access to folders or files located in the local certificate store(s). Access to those certificates are necessary for SSL session negotiation, even if they are not used for that session.

Our first data gathering method in the last issue was the first thing to do here as well. And a good thing it was we did it.

In the new scenario the IE client connected to an application running in an IIS application pool, which in turn queried Active Directory for some information. So there were at least three computers involved: IE client, IIS server, Active Directory domain controller.

An additional point of data was that the IIS server was a member of an Active Directory domain-in a forest we’ll call brownies.com, and the Active Directory which the application was intended to query was in a different Forest which we’ll call cookies.net.

In a network trace, we saw that the time delay seemed to be occurring during the traffic below:

No. Time Source Destination Protocol Info

65 15:18:48.771242 10.234.32.51 10.234.32.22 DNS Standard query A SugarDC1.chocolate.brownies.com

68 15:18:48.771242 10.234.32.22 10.234.32.51 DNS Standard query response A 10.248.175.4

144 15:18:54.755617 10.234.32.51 10.234.32.22 DNS Standard query SRV _ldap._tcp.OutofSight._sites.luvme.some.cookies.net

150 15:18:55.755617 10.234.32.51 10.248.175.4 DNS Standard query SRV _ldap._tcp.OutofSight._sites.luvme.some.cookies.net

157 15:18:56.755617 10.234.32.51 10.234.32.22 DNS Standard query SRV _ldap._tcp.OutofSight._sites.luvme.some.cookies.net

168 15:18:58.755617 10.234.32.51 10.234.32.22 DNS Standard query SRV _ldap._tcp.OutofSight._sites.luvme.some.cookies.net

169 15:18:58.755617 10.234.32.51 10.248.175.4 DNS Standard query SRV _ldap._tcp.OutofSight._sites.luvme.some.cookies.net

196 15:19:02.755617 10.234.32.51 10.234.32.22 DNS Standard query SRV _ldap._tcp.OutofSight._sites.luvme.some.cookies.net

197 15:19:02.755617 10.234.32.51 10.248.175.4 DNS Standard query SRV _ldap._tcp.OutofSight._sites.luvme.some.cookies.net

207 15:19:04.396242 10.234.32.51 10.234.32.22 DNS Standard query A OreoDC1.chocolate.brownies.com

208 15:19:04.396242 10.234.32.22 10.234.32.51 DNS Standard query response, No such name

209 15:19:04.396242 10.234.32.51 10.234.32.22 DNS Standard query A OreoDC1.brownies.com

210 15:19:04.396242 10.234.32.22 10.234.32.51 DNS Standard query response A 10.248.191.110

297 15:19:09.771242 10.234.32.22 10.234.32.51 DNS Standard query response, Server failure

309 15:19:11.474367 10.248.175.4 10.234.32.51 DNS Standard query response, Server failure

The above is interesting since we can see the duration of this traffic is 22 seconds (other traces reproducing the problem displayed a similar delay). No significant delay was present in the rest of the traffic this scenario generated.

But look at what’s happening in the above network trace excerpt. We start with a query for a local (in the forest the IIS server resides in, a.k.a. brownies.com), then begin querying for an LDAP specific SRV record to locate an LDAP server which could answer LDAP queries for the luvme.some.cookies.net directory. But there is no reply with a server to go to for this. Eventually the traffic is sent to a domain controller in the luvme.some.cookies.net domain after all, but this is the result of an entry in a Host file resident on the IIS server.

Why is this a scenario worth mentioning to you all? So that you can know what to expect in similar scenarios. If the server sending an LDAP query is a member of an Active Directory domain then it will send LDAP SRV queries in order to locate an LDAP server to service the requests it needs to make. The server knows of LDAP SRVs since the DC Locator code on it commonly uses similar behavior to locate a domain controller in its own domain for LDAP requests.

And if it cannot locate a domain controller in this method, it will fall back to whatever it has locally cached for that domain name despite the fact that the server in question is not verifiably an LDAP server.

How did we remove this delay? By making sure DNS could resolve the SRV query. This could be done any number of ways-stub zone, secondary zone-but we chose to add a forwarder on the DNS server which the IIS server was configured to look to for primary DNS.

A final conclusion we could make with this issue was to never assume that the same symptom is the result of the same cause. Let the facts bear things out, as we did in the network traces we gathered.

Until next time folks, take care out there!