Brian Puhl's Weblog : Technical Stuff - AD

Identity and Access Webcast Series

bpuhl — Tue, 31 Oct 2006 08:09:00 GMT

Here's some info on some upcoming webcasts... This first series is for the "Technical Decision Makers", but I'll post the "IT Pro" series when they get announced.

-Brian

--------------

Microsoft offers a broad range of technologies and products to enable a customer’s identity and access infrastructure. This web-cast and virtual lab series is designed to educate Technical Decision Makers (TDMs), and IT Professionals about Microsoft’s IDA solution areas centered around the following products:

Windows Rights Management Services (RMS)
Active Directory Federation Services (ADFS)
Microsoft Identity Integration Server MIIS)
Certificate Lifecycle Manger (CLM)
Active Directory (AD)

These webcasts are structured under different categories. The categories take attendees from Product/Solutions Overview, what the product is and how it can help the customer’s infrastructure, to Deployment, and through the different categories to, “What is New for the Future”.

Our kickoff webcast by Peter Houston, and Product/Solution Overview webcasts are for the Technical Decision Makers, while the following webcasts categories will be for IT Professionals.

Join our webcast series to help plan for the future, deploy new solutions, manage and optimize your existing IT infrastructure

As Technical Decisions Makers you should attend (a) our kickoff webcast IDA Vision and Strategy, and (b) Product Overview webcasts segment, to see how our IDA products can be improve cost, increase protection for your IT infrastructure Then encourage your IT Professionals to attend our following webcasts on deeper IT content.

We will be announcing more upcoming webcasts for IT Professionals very soon.

First IDA Webcasts:

(a) IDA Vision Webcast

Title: Microsoft Identity and Access (IDA) Vision and Strategy

Description: Identity and access in connected systems has gone beyond a technical concern and become a top business issue as organizations look to reduce security risk, decrease operational costs, satisfy regulatory requirements, and deepen their electronic relationships with customers and partners. In this session, learn about Microsoft's vision for identity and access technology, including the evolution of Active Directory (AD), Microsoft Identity Integration Server (MIIS), 'CardSpace', and Certificate Lifecycle Manager (CLM). You will also gain insight into Microsoft's vision for IDA in the future.

Presenter: Peter Houston

Date/Time: 11/10/2006, 10:00Am - 11:00PM Pacific Time

Click here to Register: http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032315361&Culture=en-US

(b) Product Overview Webcasts:

Title: Information Protection with Windows Rights Management Services (RMS)

Description: Protecting confidential information and intellectual property, such as e-mail and documents, is critical to the success of many organizations…

Presenter: Tim Upton

Date/Time: 11/16/2006, 1:00 PM – 2:00PM Pacific Time

Click here to Register: http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313768&Culture=en-US

Title: Introduction to Microsoft Certificate Lifecycle Manager

Description: Join this webcast to learn about the new Microsoft Certificate Lifecycle Manager (CLM)…

Presenter: Amesh Mansukhani

Date/Time: 11/20/2006, 1:00 PM – 2:00PM Pacific Time

Click here to Register: http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313484&Culture=en-US

Title: Web Single Sign-On and Identity Federation with Active Directory Federation Services

Description: As organizations extend their information technology (IT) infrastructures to provide partners with access to Web-based applications, they face difficult administrative and security challenges…

Presenter: Howard Ting

Date/Time: 11/27/2006, 11:00 AM – 12:00PM Pacific Time

Click here to Register: http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313783&Culture=en-US

Title: Identity Life-Cycle Management with Microsoft Identity Integration Server 2003

Description: Join this webcast to see how Microsoft Identity Integration Server (MIIS) 2003 enables the automation of identity life-cycle management in the enterprise…

Presenter: Lori Craw

Date/Time: 11/29/2006, 11:00 AM – 12:00PM Pacific Time

Click here to Register: http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313486&Culture=en-US

Regards,

x64 Domain Controllers

bpuhl — Wed, 13 Sep 2006 09:54:00 GMT

Had an e-mail thread with Joe recently, which also resulted in this blog entry. He's a consultant for another big tech company, and was working with a customer that was migrating a lot of non-domain joined machines to AD as well as deploying other AD aware applications. The net result though, is that he was in the unenviable position of having no performance baseline to go off of, and a bunch of customers asking how many 64-bit domain controllers they needed to buy. And therein lies the problem, there just aren't that many 64-bit DC's deployed out there (yet), so if you're starting from scratch, where do you start?

Well, to make a long story short (too late), a few e-mail back and forth later and I fired off some of the stats that we use internally here at Microsoft. In the spirit of copy/paste, here's the mail I sent (slightly edited to protect the innocent), if you don't have anything else to go on or just want some general reference...then you can use this.

REMEMBER - "IT DEPENDS" and "YOUR MILEAGE WILL VARY"

________________________________________
From: Brian Puhl [mailto:Brian.Puhl@microsoft.com]
Sent: Wednesday, September 06, 2006 6:11 PM
To: Joe
Subject: RE: Ping...

Well, like you said, “it depends” and “your mileage WILL vary.”

It’s tough, because we don’t plan based on numbers of users, workstations, or anything like that… We base capacity on performance trends, which I realize is ultimately where you’re trying to get <customer> to… So instead, here are some details from our Redmond domain. These are live numbers, which you can use to approximate. Remember that MS is probably a higher utilization environment than <customer>, so you can use these to build a deployment plan with the expectation that you could end up slightly over capacity.

Domain Details:
   99%+ of the users are in a single AD site, so assume that this is all for a single site.
   49K user accounts (includes service accounts, etc…)
   160K computer accounts
   17 DC’s for authentication load, app’s – everything but exchange
   5 DC’s in a separate dedicated Exchange site, shielded from auth load

Typical auth DC spec
   HP DL585
   4 x 2.2GHz AMD64
   16GB RAM (12 GB dit file)
   2 or 4 spindles (0+1) for OS and logs
   6 spindles (0+1) for dit, backup, and sysvol

Typical load profile (randomly picked a DC and pulled open perfmon while I’m typing this mail) – see note below
   Ave CPU – 55%
   Ave Disk Queue – 0.1

   Server Sessions – 585
   NTLM Auths – 215
   Kerb Auths – 92
   DS Client Binds/Sec – 44

   Gigabit NIC card
   NIC Output Queue – 0

Major thing to note about the perf data – We’ve got 3 DC’s offline at the moment due to dogfooding, so this perf load would be with 14 DC’s online. Our target utilization is 20-40% sustained peak CPU.

Also, based on our experience, we’re rarely NIC bound. When we see overloaded DC’s, they typically tend to be disk bound or processor bound. Even when we had x86 with 4GB of RAM, the memory pressure just translated into disk queues, so when you’re spec’ing out your servers I would be least concerned about the connectivity. You probably also noticed in the whitepaper that x64 doesn’t give you a whole lot of benefit in a pure auth environment. These operations tend to be disk bound even in a 64-bit OS.

I think you’re hoping for a “5000-10000 user” type answer, but even if I gave you a completely wild guess, It would probably do more harm than good in your conversations with the customer.

Does this give you a better idea? Are there other details that would help you make a better guess?

The whitepaper that I referred to is the Active Directory 64-bit Performance Comparison paper, located here.

Useful repadmin switch

bpuhl — Sat, 22 Jul 2006 11:03:00 GMT

Repadmin is the "swiss army knife" of AD tools - But the following can be one of those "big red buttons" that you keep in your back pocket and hopefully never need. But sometimes it's just useful to slow things down until you figure out what's going on...

repadmin /options * +disable_outbound_repl

repadmin.exe - the tool
/options - Did you even know this was there? If not, try repadmin.exe /experthelp
* - means run against all DC's
+disable_outbound_repl - Self explanatory? I hope so...

Please don't go running commands willy-nilly in your production environment - play with repadmin.exe (or any other tool) in a lab or against some virtual DC's so you can learn what it's doing and how to use it...then sit back and let the tools do all the work for you.

AD Training

bpuhl — Sat, 22 Jul 2006 10:06:00 GMT

hmmm....ok, so here's an interesting problem: I'm a Microsoft employee. My blog is hosted on Technet.com. And I'm pretty sure that there is a policy somewhere, which I'm unaware of, that addresses blog posts about 3rd party companies... But I've never really been one for following too many rules anyways, so here you go:

I wrote a post back in May about changes to our organizational structure for supporting AD internally at Microsoft. While I still think the re-org was a great thing to happen within IT, and we're making big progress on many things that had been stalled in the past (ADFS, smartcards, selective auth forest, etc...) - one thing that I noticed were all of the new faces who were going to be managing the DC's. Now, most people in the org have AD experience, but let's face it, there's a big difference between reviewing schema extensions and doing delegations; versus troubleshooting replication or a server on which lsass.exe is taking 90% of the CPU. Both can be difficult, but they are seperate skills. So, to make a long story short, (too late), I fired off an e-mail to Dean at MSETechnology to see if he could help us out with some training. Many people who have been around AD for a while know Dean (or at least "of" him), whether it's the random references in Joe's blog, his answers on ActiveDir.org, or from NetPro's Directory Experts Conference.

Anyways, after a bit of back-and-forth figuring out the logistical details, Dean came on-site here in Redmond and has spent the last week giving what can only be described as the most entertaining, in-depth training on AD that I've ever seen. Topics ranging from replication and topology, to sid history/filtering, to the most...ummm...."descriptive"...segment on the FILE replication service which I've ever sat through, I would have to say that if you're looking for some 300-400 level AD information (as opposed to someone standing up reading a book to you), then this was the class you want to be in.

There's no comparison to the quality of the content, but two things stood out most...and note, that I didn't even sit through the entire week, but was coming and going at random:

While there was definitely structure and order to the content, there was never hesitation to go off on wild tangents which would ultimately enhance the topic being discussed. Most impressive are the impromptu labs, which went something like: "That's a great question...why don't we log into the VPC and try running xyz command and see what happens...ok, well since that didn't work, let's figure out why and then see what we should do." Having taught classes before, I can say that it takes an ENORMOUS amount of confidence in your knowledge to make up labs on the fly.
Professionalism - Yes, a couple of you just looked and said "what? that's not the Dean I know!" Well, actually it is and you know it, but it's fun to play. Most mornings and some afternoons we sat down to go over the class progress and to make sure we were hitting the right topics. There wasn't ever any hesitation to change things "on the fly" (again, very difficult for 'structured' instructors) and the open dialog was exactly what we needed to make sure that everyone was getting the most out of the class. He truly cared about making sure we got the most of the time spent.

So if you're looking to bring in some custom (in-depth, not MOC based) training, and wondering what other people have done, then MSE Technology is worth a look.

AD and DC Builds, tweaks, configurations... The Registry

bpuhl — Fri, 07 Jul 2006 01:54:00 GMT

The first installment, what our hardware looks like, may have been useful...but I know that's not really the juicy gossip that everyone is looking for...so here's a quick and follow-up with the registry tweaks that we set internally...

Strict Replication is enabled on Windows Server 2003 - For Windows 2000 there is the "Correct Missing Objects" key which has similar (though reversed) funcationality. Basically, this stops a DC from replicating lingering objects
HKLM\system\currentcontrolset\services\NTDS\parameters" /v "strict replication consistency" /t REG_DWORD /d 0x1

The Exchange team requires this for RPC/HTTPS functionality
HKLM\system\currentcontrolset\services\NTDS\parameters" /v "NSPI interface protocol sequences" /t REG_MULTI_SZ /d "ncacn_http:6004"

Causes an event to be logged after each online defrag task. The event includes file statistics about the DIT including whitespace. We run a seperate task to harvest these events for database file maintenance.
HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics /v "6 Garbage Collection" /t REG_DWORD /d 1

Set to 5 causes an event to be logged for "expensive" and "inefficient" queries. Extremely useful during troubleshooting isolated load issues.
HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics /v "15 Field Engineering" /t REG_DWORD /d 5

The following keys enable the database perfmon counters (note that these are just the reg keys, you have to enable the counters themselves as well using "Lodctr.exe Esentprf.ini")
HKLM\system\currentcontrolset\Services\ESENT\Performance /v "Open" /t REG_SZ /d "OpenPerformanceData"
HKLM\system\currentcontrolset\Services\ESENT\Performance /v "Collect" /t REG_SZ /d "CollectPerformanceData"
HKLM\system\currentcontrolset\Services\ESENT\Performance /v "Close" /t REG_SZ /d "ClosePerformanceData"
HKLM\system\currentcontrolset\Services\ESENT\Performance /v "Library" /t REG_SZ /d "%systemroot%\system32\esentprf.dll"
HKLM\system\currentcontrolset\Services\ESENT\Performance /v "Squeaky Lobster" /t REG_DWORD /d 1

Just what it sounds like. Causes DFS to use site costed referrals.
HKLM\System\Currentcontrolset\Services\DFS\Parameters" /v "SiteCostedReferrals" /t REG_DWORD /d 1

Last but not least, on some of the servers we set LdapSrvPriority and LdapSrvWeight. These are used for load balancing and isolation, but are not consistent across all of our servers. Older/slower hardware gets lower weight, and special case servers that we want to shield from general traffic get higher priorities. Check here for more info on these keys: http://support.microsoft.com/?id=306602

AD and DC Builds, tweaks, configurations... (1)

bpuhl — Fri, 07 Jul 2006 00:50:00 GMT

I received a mail from a blog reader (Jim) who asked:

"Can you provide any insight regarding and tweaks or configuration settings you guys use on your DC builds?"

Sure, I'm happy to do this, so here I am typing happily along, and realized that there is a lot more configuration/tweaking/settings that we use than I should reasonably put into a single blog entry. Instead this will be the first of multiple entries...

So, let's start at the very beginning (it's a very good place to start)... With our standard hardware platforms. All MS IT domain controllers are based on either our "large" or "small" SKU...internally, we call these are DC-E (enterprise) or DC-F (field) platforms.

The DC-E specs are:

DL585
2 x 1.8GHz AMD Opteron (64-bit) dual core processors
16GB RAM
172GB total storage

Internal Array Controller - 2 x 72GB - RAID 1

50GB OS partition
18.8GB partition for Log files (L: Drive)

Array Controller 1 - External Storage - 6 x 36GB - RAID 0+1

103.2GB partition for DIT, SYSVOL, Backups (M: Drive)

The DC-F specs are:

DL385
1 x 2.2GHz AMD Opteron (64-bit) dual core processor
8GB RAM
137GB total storage

Internal Array Controller
Disk 0 - RAID 1 - 2 x 72GB

50GB OS partition
18.8GB partition for Log files (L: Drive)

Disk 1 - RAID 0 + 1 - 4 x 36GB

68.8GB partition for DIT, SYSVOL, Backups (M: Drive)

All of our DC's run x64 OS's...well...unless we have some dogfood requirement for 32-bit OS runtime (which we periodically do)...but for all intents and purposes, let's just pretend because we really WANT to run all 64-bit OS's.

Somewhere previously I mentioned that our average DIT size is 10-11GB on disk. The DC-E with 16GB of RAM let's us cache the entire database with room for growth, the DC-F with only 8GB of RAM is usually deployed where we need services, but don't have the load so caching is less of an issue. In that case, the DC-F is significantly cheaper for us.

Interesting SSID and Reusing Attributes

bpuhl — Sat, 03 Jun 2006 07:19:00 GMT

I bought a new truck a few months ago, and right on schedule (as the salesman promised), as I was coming due for my first oil change, I got a card for my first one free at the dealership. Never being one to turn down a free deal, I dropped in the other day, handed over my keys, and directed to the lobby where the offered, "Free cofee, pastries, and wireless internet." This is Redmond after all.

So sitting down with my laptop, coffee cup, and a donut, I fired up my laptop to synch my e-mail. Viewing the available wireless networks, I almost laughed out loud when I read the SSID:

LJ Chev Cust Net - Ask Cashier for Key

99% of you probably just looked at that and went, "Umm, yeah...DUH!" But both of the people in the office that I mentioned this to said, "Whoa, neat..." which by my statistical methodology made this blog-worthy.

So what does this have to do with AD? It reminds me of a common question that goes through the AD discussion alias at work. The details change, but the gist of it is always something akin to:

"I've got a customer/application/user that is asking whether there are any applications (MS or 3rd party) that use the drink attribute, they are creating a custom password reset portal, and need someplace to store the answer to the secret-question."

Three people just fell out of their chairs laughing, two of you did one of those "I can't believe it" head smacks, and there's one guy who just made a note to himself that he needs to update his portal application to use a different attribute. THAT'S the guy that I want to talk to.

Reusing attributes is bad, each and every one has an intended purpose...well...ok, there are those pesky extensionAttributes, but let's not get picky... If you're creating an application that needs to store data in AD, then go ahead and get an OID branch, and create one... there's a ton of documentation out there on how to do this.

Someone is going to say something about being so cavalier with the schema. Yes, I understand that it's a one-way operation, and I always advocate doing appropriate testing before mucking with your production schema, but I've always been mildly disappointed at the level of FUD that Microsoft created surrounding schema extensions. Caution and due diligence should be taken with everything you do in Active Directory.

How MS IT Does ADFS Value Card now available...

bpuhl — Mon, 12 Dec 2005 19:32:00 GMT

The Microsoft ADFS value card is now available for download......(read more)

Time, time, everyone wants time...

bpuhl — Fri, 09 Dec 2005 07:31:00 GMT

In a previous post about managing FSMO roles, I asked a question about who remembers to configure the new server as an authoritative time source when transferring the PDC FSMO role. The reason I ask this, is because when you look at managing the FSMO roles from an operational perspective, everything is automated or dynamic EXCEPT for the time service configuration.

In MS IT, we have 6 production forests, most of which have at least 2 domains and the main forest has an empty root with 8 child domains. Time service configuration both between forests, and within the forests is critical, and operationally we have had multiple instances where a root domain PDC has ended up on a misconfigured server.

For this reason, a couple of years ago we went through and configured all root domain DC's to be authoritative time servers. This way, no matter which server you transferred the PDC to, they were always "in synch" with each other. This works well for us, because our largest root domain only has 5 DC's in it, and most only have 3 DC's. Manually configuring these servers is a "fix and forget" operation. But what would we do if we were in a single domain environment, with a couple of hundred DC's?

To answer this, I look at how we manage the PDC role in our largest child domain, Redmond. The Redmond PDC is always under the highest load of any DC in that domain, so we take some extra steps to shield it from general auth traffic. Because we know that when we're dogfooding, the PDC is also the most likely candidate to be offline for debug, we also take an extra DC and pre-configure it the same way...the net result is that we have one really busy PDC, and a not-so-busy stand-by PDC. This may be an expensive way of managing failures, but then again we don't expect anyone else to dogfood the way we do either...so for us it's worth the extra cost.

How could our Redmond PDC configuration map to the time service configuration in your single domain forest with a lot of DC's? Simple, just take some subset of your servers...and identify those as the ones which you could transfer your PDC role to if necessary. Pre-configure them as authoritative time servers, then go sit back and relax. When the call comes in the middle of the night and you need to transfer the PDC role to another box, you won't go back to bed with that itchy feeling that you forgot something...instead you'll just relax and get a good nights sleep.

(I also wanted to note that Joe called this out as well in the comments of the previous post. If you haven't checked out his FREE tools (http://joeware.net) then you're probably working too hard...)

What to do with FSMO roles...

bpuhl — Thu, 08 Dec 2005 04:45:00 GMT

We recently hired a new engineer to a team which manages some of the internal MS environments... We were discussing FSMO role placement and he sent me mail (snippet below slightly edited) which I thought was interesting...

The reason why we separated the roles at my last company was due to the FSMO role seizure process. You are correct, although the server is still a single point of failure, we can mitigate this single point of failure by placing the forest roles on one box and the domain roles on another. In the event that we unexpectedly lose a DC that is either a forest or domain FSMO role holder, the process of seizing the roles is minimized (less roles to seize). Also, it had been our experience that the forest roles aren't really used that often. You are correct, FSMO roles are still a single point of failure, however, unless we really need to perform any forest related “stuff”, the single point of failure (from a forest FSMO perspective) is a non-issue. This is not the case with the domain FSMO roles, specifically the PDCE. At my last company, we felt that due to the PDCE functions it was necessary to place the domain FSMO roles on a separate box...

I wanted to share this, because it reminded me of a FSMO related interview question which I've used in some variation or another:

Suppose you're paged in the middle of the night and told that one of the 150 domain controllers in your single domain forest crashed. You're first thought is likely "So what, I'll deal with it in the morning." but then you remember it's the one holding all 5 FSMO roles. If you could only pick one FSMO role to sieze, which one would let you go back to sleep without worrying about the next day?

There are many people that I've asked this question to...the large majority of who answered, "The Schema Master, because without the schema the AD can't function."... Hopefully they aren't reading this blog from their whichever other job they landed...

So back to the the whole FSMO single-point of failure and redundancy thing...

I figured there were 2 possible reasons that they arrived at the idea that seperating FSMO roles based on forest/domain division was logical:

There was some sort of fault tolerance between FSMO roles which could be preserved in a failure
There was some urgency (specifically user impact) to getting a role holder back online immediately should a failure occur

The first reason is obviously false. FSMO is the "Flexible Single Master Operations" with the emphasis on "single"...the whole point of these roles was that even though Active Directory is a distributed system, there were just some things that could only be done in one place at a time. So let's just take the generally accepted knowledge that each FSMO role provides specific functionality which only exists in that role.

The second reason takes a bit more thought, but what really happens when a FSMO role holder fails? Looking at each role, what the impact of it being offline is, and the urgency:

Schema Master – Schema updates are not available – These are generally planned changes, and the first step when doing a schema change is normally something like "make sure your environment is healthy". There isn't any urgency if the schema master fails, having it offline is largely irrelevant until you want to make a schema change.

Domain Naming Master – No new domains or application partitions can be added – This sort of falls into the same "healthy environment" bucket as the schema master. I don't know of anyone who has just randomly decided to add a new domain to a forest without much thought or planning...of course, then again, I don't know all that many people either... You might wonder why I mentioned app partitions there as well...personal experience. When we upgraded the first DC to a beta Server 2003 OS which included the code to create the DNS application partitions, we couldn't figure why they weren't instantiated...until we realized that the server hosting the DNM was offline (being upgraded) at the same time. Sure enough, it came online and there they were... But I've never said we were perfect here...

Infrastructure Master – No cross domain updates, can't run any domain preps – Domain preps are planned (again)...But no cross-domain updates. Hmmm...that could be important if you have a multi-domain environment with a lot of changes occurring...but wait...the IM tasks are throttled to run over a 2 day period (by default), so how much urgency does that really imply? I guess you'd have to call it as you see it in your environment but it's probably not 3am urgency...for my buddy the new engineer, he's only working in single domain forests anyways, so urgency = zero.

RID Master – New RID pools unable to be issued to DC's – This gets a bit more complicated, but let me see if I can make it easy. Every DC is initially issued 500 RID's. When it gets down to 50% (250) it requests a second pool of RID's from the RID master. So when the RID master goes offline, every DC has anywhere between 250 and 750 RIDs available (depending on whether it's hit 50% and received the new pool). So the urgency question is how long will it take your environment to exhaust the RIDs on a given DC? My guess is that in most environments, this isn't that urgent either. Oh yeah, and don't forget that if you do seize the RID master during a failure...that's an automatic flatten and rebuild of the server, you can't bring it back online.

PDC – Time, logins, pw changes, trusts – So we made it to the bottom of the list, and by this point you've figured that the PDC has to be the most urgent FSMO role holder to get back online...the rest of them can be offline for varying amounts of time with no impact at all...so what about this one? Yes, you should get the PDC back online whenever you can but it's not even something that I'd jump out of bed to do...let's call it the "first thing in the morning" list. Time synch's are important, but w32time does a pretty good job and nothings going to diverge between today and tomorrow enough to impact you...users may see funky behavior if they changed their password, but replication will probably have completed before they call the help desk so nothing to worry about, and trust go back to that whole "healthy forest" thing again... The biggest impact we see internally at Microsoft from the PDC being offline are all of the applications which were written in NT4.0 timeframe that are biased towards it. Now that's something to consider.

So when it really comes down to it, is there any benefit to seperating the forest and domain roles onto seperate servers? Probably not...is there any harm in it? Nahh...let's just chalk it up to "operational preference" since the guys who are watching this stuff day to day need to be comfortable with the way the environments are configured.

Pop Quiz Time:

Raise your hand, if when your phone rings in the middle of the night and you get that call...you transfer the PDC role and go back to sleep...

...

...

now keep your hand in the air if you reconfigured the server that you transferred the role to, to also be authoritative for time? I think I found a topic for my next blog...

If I don't see you before then, Merry Christmas, Happy Holidays...or like that commercial says, Merry Chrismahanakwanzaka.

Bulk Password Resets

bpuhl — Wed, 02 Nov 2005 08:26:00 GMT

When a user resets their password, what happens? What about if ALL your users reset their password? Can your infrastructure handle it? Are there "special" changes that you'd want to make? More importantly, this is probably such an edge case that it's not on your top 150 list of things to do? Well, just for kicks, let's look at bulk password resets.

Consider the following scenario:

For whatever reason floated their boat that day, your security team wakes you up in the middle of the night with a mandate that all 20,000 users in your domain need to be forced to change their passwords when the log in the next morning. Your first thought, is probably something about finding a script that can expire the passwords on all of these accounts, but at some point it's likely going to occur to you that the majority of these 20,000 users are all going to come walking into their offices around the same time (8am, 9am, whenever...) and try to log in. Can your DC's handle it?

It turns out that this was a scenario that we wanted to test (without the security boat floating) in January 2003 while we were dogfooding Windows Server 2003. Recently the topic came up again in a discussion, so I dusted out the original report that was written to refresh my memory of what we saw and what we'll do if we ever need to do a mass password reset.

To start with, first you have to assess the true impact. If you're users are spread out across many time zones, then "next morning" is really a relative thing and you should start to consider if any of users are really in the middle of their day. Those are the folks that are going to call help desk on you. In our case, we intentionally reset the passwords of our users here in Redmond, this ensured that nearly everyone was going to come in around the same time (10am'ish) and reset.

Before we get to far into this though, let's look at password reset behavior. When a user logs on and changes their password, there is an extra step that occurs. The NetLogon service of the DC making the password change forwards the new password to the PDC for the domain. Generally this is a good thing, because if the user attempts to authenticate using their new password against a DC that hasn't replicated it in yet, the "failed" password is checked against the PDC for the domain and the user will succeed. Both of these behaviors, the password update and the additional checks, can present a problem during a bulk reset because the volume of changes being pushed from many DC's to one PDC can melt your PDC into slag on the data center floor...and that's always messy to clean up. So the solution is to hide your PDC from the sudden flood of traffic that's going to come from the DC's.

To solve this problem, there is a registry key called AvoidPdcOnWan that needs to be set on every DC (FOR loop with "reg add" is your friend here). Setting this key tells every DC to let normal AD replication take care of getting the password to the PDC. Also, if presented with a bad password, the DC won't check with the PDC but instead will just fail the authentication. The second step is to move the PDC into it's own AD site linked with a site link. In our case, all of the DC's for these 20K users were in a single AD site, which included the PDC. So we created a new site called PDC and linked it to our main site with the minimum replication interval. This effectively put the PDC "on the WAN" to all other DC's.

The net result of these changes is that the bulk of your DC's will do the work AND they'll play nice with the PDC for the domain (always a good thing). Unfortunately there is the behavior change that the DC's won't check a "bad" password with the PDC, so you wouldn't want to leave this config in place all the time as that's a "bad user experience" which normally translates into help desk calls.

Hopefully you never have to reset the passwords for a large bulk of users, but if you do, then setting this reg key, creating a new site, and moving the PDC into it will at least save you from the secondary thrash of watching your PDC slowly fall onto it's side and beg for mercy.

How Does MSIT Do...DC Placement?

bpuhl — Wed, 02 Nov 2005 07:47:00 GMT

In the past couple of months, I've been asked at least 3 or 4 times how MS IT determines where on their network to place domain controllers. The questions are usually coming from larger, enterprise type customers and usually sound something like this:

How does MS IT determine the number of users or workstations resident on a given subnet (which let's you know how many are in the site, based on site/subnet associations)
Based on #1, how does MS IT use this info to determine how many DC's are needed to service the site?
How do you determine over time that you need to add or remove DC's?

The short answers to these are:

We don't
See #1
Based on performance

But short answers really miss the whole point, what they are really looking for is how we do DC placement. To start with, we review our DC placement twice yearly, and our capacity planning (performance reviews) 4 times yearly. Based on experience, we should actually do both more often, however it's just not practical. For example, in a single 6 month period we had over a dozen sites in South America with WAN links to North Carolina, change to use Redmond as their hub... (this is where you might ask why the network guys aren't talking to the AD guys?...good question...)

So during our DC placement reviews, we're looking at the following network criteria:

WAN availability - Greater than 99.5% uptime between the end user and their nearest DC

Max Average Latency - Less than 500ms...this is a loose target, based on feedback from our users in the regions. This tends to change with our environment, but the feedback we get is that if a user enters their username/pw, goes to get coffee, comes back and it's still "Apply Policies"...they'll call help desk and let us know.

Max 95th % utilization - Less than 90% - Typically this isn't an issue, although some of the sites in Africa and the Caribbean come close...

With the network topology understood, we then consider some other factors:

Site Classification - This is one of our general categories that tells us whether the building has a secure physical location for a DC, and what the primary type of users are in the site (ie. Sales, PSS, Dev, etc...)

Critical Applications - There was a time, when this category was called "Exchange", however our Exchange team has done a massive amount of consolidation and we've uncovered some other applications which are business critical

With all of this information in hand, we throw it together in a pot, sprinkle a little sweat on the keyboard and a dash of Excel, and come up with any places where we either need to add or remove DC's to support our users.

This is normally the part where people say..."Yeah, but what about the number of users in a site? Doesn't that matter?" Not for us, we don't actually count the number of users in a site when determining DC placement. There are two great examples, one is a call center in Texas that has several thousand users, redundant links, high bandwidth, low latency, etc... No local DC. By contrast though, Microsoft Game Studios has a development center in the same area with 50 developers who are using applications that have extremely sensitive authentication requirements, and their business has incredibly aggressive deadlines such that they couldn't tolerate ANY possible network outage impact. They get a DC. Does the number of users matter to us? No, but how the users leverage the DC's does matter when it comes to determining where to deploy the servers.

How Does Microsoft IT Do...

bpuhl — Wed, 02 Nov 2005 07:21:00 GMT

Engineers in Microsoft IT spend an unusually large amount of time talking to customers answering questions which start with:

How Does Microsoft IT Do...<fill in the blank here>?

I'm going to try to start to post some of the more common questions and answers in this blog to "share the wealth", but before I get to the first of these HDMSITD posts, I wanted to put in my "default disclaimer" and a little background information to help ease some confusion...so here it goes:

Default Disclaimer:

Odds are extremely high, that your environment does not look like Microsoft's. In fact, I could guarantee it. We do a lot to our internal deployments that most people would consider reckless, all for the sake of "dogfooding" (testing things on ourselves first). So while I'm babbling on and on about how we do things internally, you should look for the "golden nuggets" that you find interesting. Please don't take what we do wholesale and try it out against your production environment.

Ok, that much being said...I'm going to ramble on a bit about our environment so everyone will have some context of where I'm coming from, again to help find the nuggets that may be useful:

We have multiple production, pre-production, and test forests for various business purposes, most of them are 1 or 2 domain forests but our largest contains an empty root and 8 children geographically dispersed...But wait...doesn't MS say not to use an empty root now? Yes...we do...and given the chance to start all over again, we'd probably have one big happy domain...but I digress...

Our main forest has about 100K user accounts, and 300K machine accounts, which represent MS employees in 400+ sites worldwide...our AD database is ~10GB on Windows Server 2003 (~18GB in Windows 2000) and about half of our ~200 DC's are 64-bit, with plans to be fully 64-bit by next summer.

One of the nicer things about our environment is that we generally don't have problems with bandwidth, and this is one of the places where we diverge from many of our customers.

So now you've got some background on where I'm coming from and what our internal environment looks like, which should help put some of our HDMSITD... questions in context.