Ask Premier Field Engineering (PFE) Platforms

So You Want to Be a Active Directory Master, eh?

dgreg - MSFT — Mon, 21 May 2012 06:00:00 GMT

Back in February 2012, I was lucky enough to take part in the Windows 2008 R2 Directory Services Masters class and I promised that I would blog about my experience. Consequently, this will probably turn into another series as I wouldn’t do it any justice by only writing one entry about it.

Introduction

For those unfamiliar with our Microsoft Certified Master’s program, think of it like the Cisco CCIE of the Microsoft world. Microsoft was looking for a way to distinguish the breadth of knowledge and experience of select Microsoft engineers beyond the MCSE and hatched a program about 5 years ago originally called the Ranger Program. It was first started for Exchange engineers and due to overwhelming demand branched out to encompass Active Directory, SQL, OCS/Lync, and Sharepoint. I originally heard about this “Ranger” accreditation through an Exchange engineer friend of mine. I heard it was a grueling three-week long class that would test your deepest technical abilities and the strength of your spirit. I immediately knew I had to do it. :) I told my wife that I eventually wanted to be a Ranger, and she honestly thought I was changing careers to become a Forest Ranger, made sure to tell her friends about it, and occasionally made jokes about it. Here is more information about the program:

http://www.microsoft.com/learning/en/us/certification/master.aspx#tab1

I contacted my manager and told her about my desire to get into the program and was told that there was a two year waiting list. I added my name to the list and waited almost 3 years and even then, it took the recommendation of another accredited Master to get my name into the conversation. Nonetheless, I was now a candidate for the class. This didn’t mean I would get in but I was one hurdle down, many more to go.

Once the excitement wore off, I then read the introduction email and quickly become discouraged as though I was applying for a new job or something. To quickly give you some background on my experience, I’ve been working in IT for over 12 years ranging from web development to teaching MCSE classes to now being a PFE at Microsoft. And with 8 years now in PFE and having delivered almost 200 ADRAP’s, I’ve felt like I’ve seen it all! But even after all of this, I worried whether it would be enough to successfully get through this class?

Prerequisites

The prequisites for the Active Directory Masters class are:

Five or more years of hands-on experience with Windows Active Directory: installing, designing, configuring, and troubleshooting
Thorough understanding of Windows Active Directory design and architecture
300-level understanding of site component topology, forest operations and topology, the Active Directory distributed file system, file replication services, security, client interactions, and Group Policy
Basic understanding of Active Directory Certificate Services, Rights Management Services, Active Directory Federated Services, and ADAM/Active Directory Lightweight Directory Services
Functional skills in basic protocol analysis, Hyper-V, scripting, PKI, and IP addressing and routing
Ability to speak, understand, and write fluent English

And then one of the following certifications:

Microsoft Certified Systems Engineer (MCSE) on Windows Server 2003
Or
Microsoft Certified Systems Engineer (MCSE) on Microsoft Windows 2000 Server

And one of the following exams:

Exam 70-219 or Exam 70-297
Or
Microsoft Certified IT Professional (MCITP): Enterprise Administrator

Once I had met these prerequisites, I then had to complete the following:

Complete the brief application.
Upload your resume or curriculum vitae (CV).
Submit supporting documents including two write-ups on projects that I had been a part of that demonstrated my breadth and knowledge of Active Directory and Microsoft Technologies.
If they can’t verify my experience, I will then be asked to go through a 30 to 60 minute interview.
Register and then pay in full for the program.

It took me a few weeks to pull it all together but I submitted my application and all my supporting documents and waited patiently. Later that week, I got the email that I had gotten in.

The Basics

The MCM class consists of two straight weeks of training in Redmond, WA. During those two weeks, you’ll get only 1 day off although you’ll probably be studying during all your free time. When it starts, it will be 8-10 hours a day Monday through Friday. On Saturday, you’ll have a 3 hour written exam testing you on topics from the previous week. Sunday is the one day off. Then Monday-Friday, classes again are 8-10 hours a day. On that next Saturday, you’ll have another 3 hour exam and the very next day, which is Sunday, you’ll have a very long, grueling 9 hour lab exam. It boils down to about 90 hours of class time, 6 hours of written exam time, and 9 hours of lab exam time. Add this to all the study time and it makes for a very long, exhausting two weeks.

The class covers each of the following topics in depth:

Active Directory Internals
Domain Name Resolutions (DNS)
Client-Side Interactions
AD Site Topology and Replication
RODC
Authentication
Lightweight Directory Services (LDS)
Group Policy
AD Disaster Recovery
PKI
FRS
DFS including DFS+N and DFS+R

Now remember, this class is not for someone that wants to learn about these topics. I really can’t stress this enough but this class is for those that have extensive experience and knowledge on these topics and want to take it to the next level. If you’re not intimately familiar with each of the above topics nor have the desire to learn the internals to each of the above topics, you probably won’t pass this class. I’m not trying to scare but you can’t just read some online brain dump and then pass this class. I’m convinced that successfully getting through this class takes experience + desire + hard work, like most good things in life :)

Preparation

As I began preparing for the MCM, I wasn’t sure exactly how to prepare because I didn’t really know what it would entail. Should I go back and read the Microsoft Resource Kits, Windows Internals, or review every ADRAP I had ever done? In between work, travel, and family, how would I have time? As the MCM approached, I thought back to my college days and all those late nights before those big final exams. I would stay up all night cramming, walk into the classroom like a zombie, and walk out with a C+. But this wasn’t college anymore; this wasn’t a topic I had been studying for only 4 months. This was my career…Something I had been passionate about and worked on every week for almost 14 years; a culmination of my professional career. I decided that if this wasn’t enough, perhaps it just wasn’t meant to be and if this wasn’t enough, I was dying to know the Microsoft studs who wrote this class. Even though I wasn’t sure how to prepare for the class, over the course of the month before the MCM, I was passively going through various scenarios and/or topics in my head to help fill in any gaps.

The best advice I can give for preparation besides studying and knowing the above topics inside and out is to know all the differences and functionality availability based on OS version, domain functional level, and forest functional levels. Also, be familiar with Active Directory troubleshooting to the extent that you’re comfortable with all the built-in AD tools, support tools, and resource kits tools… For example, do you why and what repadmin, klist, certutil, or dfsutil are used for? I don’t think that knowing these tools will necessarily help you get through the class but if you’re tool chest doesn’t comfortably include these, you’re probably not where you need to be for this class.

Over the course of this new series of mine, I’m continue to share my experience of going through the MCM class, the challenges, and mental breakdowns as we slowly start to unfold the mysteries of Active Directory. Stay Tuned!

Roaming AD Clients, with an Updated Script

dsymalla — Mon, 14 May 2012 11:08:00 GMT

Three months ago I posted some information on AD Sites, Subnets and Roaming Clients. The heart of the blog was a PowerShell script that collected and collated netlogon.log files across all Domain Controllers in the forest to report a list of hostnames and IP addresses that have authenticated from IP addresses with no corresponding subnet defined in Active Directory. I call these roaming clients, because they randomly seek out Domain Controllers, with no sense of closeness.

In the past three months, I’ve fielded some good questions from customers about roaming clients and the PowerShell scripts. Below are some of the questions, my responses and an updated script.

Question:

Do I even need to define any subnets in Active Directory?

Answer:

There is one very specific scenario where you don’t need to define any subnets – If you have exactly one site defined in Active Directory. I’ve got some customers who have a simple Active Directory (for an extranet, for example). It has two Domain Controllers (for redundancy) in a single site (usually Default-First-SiteName). In this case you don’t need to define any subnets. All IPs will be associated with that site, and DCs will report no roaming clients in their netlogon.log files. Beware, though. If you define one subnet, you need to define all subnets.

Question:

Getting subnet information from my network team is harder than catching a greased pig. Can I just deploy a “Catch All” subnet to deal with roaming clients?

Answer:

There is a compelling case for using the “Catch All” subnet. To summarize, a “Catch All” subnet is a subnet with a broader scope, which encompasses most/all of your specific subnets. For example, if you have numerous 10.10.x/24 subnets associated with various sites, you could configure a 10.10.0.0/16 subnet and associate it with a major Hub site. So if you forget one of the 10.10.x/24 subnets, the clients will automatically be “caught” by the 10.10.0.0/16 subnet and gravitate to the hub site. So instead of roaming, these forgotten clients will gravitate to the hub site.

While there’s nothing wrong with a “Catch All”, I’m not a big fan. First, it hides the problem of subnet definitions. By associating roaming clients with the hub site, you will never see them logged in the netlogon.log file, so you will never be able to properly fix them. Second, while AD doesn’t have a problem with overlapping subnet definitions, some AD-Aware applications like SCCM don’t like them. The SCCM client in a specific site, which is also covered by a “Catch All” may not be able to determine whether it belongs to the specific site or to the “Catch All” site. AD will always use the more-specific subnet definition when determining which site to which it belongs.

Question:

Since you don’t recommend a “Catch All” and there is no practical way I can keep subnet definitions up-to-date at all times, is there anything else I can do for roaming clients?

Answer:

If your AD environment spans multiple sites, and roaming clients feel the pain of not finding a “close” DC, you can/should use our Branch Office Recommendations. Specifically, you should configure “remote” domain controllers to NOT register generic SRV records. Generic SRV records are used by roaming clients (or non-AD aware clients) to discover DCs. If you configure your remote DCs to NOT register these records, they will only register site-specific SRV records. Thus, only clients in the remote site will find remote DCs. Roaming clients will not find remote DCs, and they will gravitate to hub DCs. Chapter 4 in the Branch Office Guide describes this in more detail, while KB 306602 contains a summary. The beauty of this configuration is that it addresses a number of scenarios where clients might roam, including undefined subnets, in-site DCs being unavailable, or non-site aware applications that look to DNS to discover DCs.

Question:

I’ve tried using your script to report roaming clients. However, after I add subnets to AD and re-run your script, it still reports the clients as roaming. What am I missing?

Answer:

You’re not missing anything. The script isn’t intelligent enough to distinguish between old events and new events. So it will report clients as roaming as long as there are entries in the netlogon.log, regardless of the date. The new script (FindRoamingClientsv2.ps1) is now date aware. You simply run the script and pass it the number of days in the past you would like to consider. For example, the following will only consider events in Netlogon.log from the past 5 days:

.\FindRoamingClientsV2.ps1 5

Note that you can run the script without the # of days parameter. In that case, it will go back 7 days.

Question:

Are there any other improvements to the script that you’d like to mention?

Answer:

The script now report progress better. So in large environments, you can see which DC you are currently collecting logs from. It also includes the number of the current DC and the total number of DCs, so you have an idea of how much longer it will run. I’ve run the script in environments of 200-500 DCs, across multiple continents. In those cases, it took from 2-6 hours to run.

I hope you enjoy the new script, and it helps you stay on top of AD subnet definitions.

Doug Symalla

How to Track the Who, What, When and Where of Active Directory Attribute Changes – Part II (The Case of the Mysteriously Modified UPN)

Rick Bergman — Mon, 07 May 2012 14:00:00 GMT

Hello, Ray Zabilla and Rick Bergman again. As promised in our previous post on this topic we will go into the details of how we created the script, the challenges we had during testing and what final code looks like. We are even so generous that we are going to share the scripts with you too. :)

How to Track the Who, What, When and Where of Active Directory Attribute Changes – Part I (The Case of the Mysteriously Modified UPN)

Quick Review – The story you’re about to hear is true and the names have been changed to protect the innocent…

Some unknown process, running on some unknown computer, at some unknown time was changing the UPN on the Active Directory user accounts.

Since Contoso is running Windows Server 2003 R2 X64 Domain Controllers, we recommended they search the Security event log for Event ID 642 which indicates a successful “User Account Change”. The Event ID includes information that identifies the attribute which was changed and the “calling account” initiating the change. This means that each domain controller will have to be scanned for the Event ID 642, because you never know on which writable DC the change is going to be made.

Contoso uses an enterprise auditing and collection system so the logical thing to do was to use the tool to search for the 642 Event ID versus searching each DC independently. Contoso IT made an inquiry to their security auditing team to give us all event ID 642 from all DC’s in the environment from their enterprise collection system and we would search through them. This effort turned out to be unsuccessful since for some reason the archived logs did not contain all the data and they were only able to provide part of the data needed, which of course did not contain any of the specific UPN change events we were hoping to find. We discussed Microsoft’s SCOM ACS tool, System Center Operations Manager Audit Collection Systems as a solution, but the customer declined because that was not a strategic direction for them.

The Solution – Version 1

It was stated in the previous post that the domain controllers had a security event log size of 180 MB, this means that it took less than 15 minutes for the event log to wrap. The security event log needed to be increased on each DC in order to buy additional time to see if it would be possible to capture the Event ID 642. It was good that Contoso was running Windows Server2003 x64, because the 64-bit OS would be able to handle larger event log sizes.

Change the Security Event File Size

The first thing the Contoso IT team did was increase the Security Event log size to 2GB. We had researched and found there had been a few incidents reported on Server 2003 x64 domain controllers when the security log files were set to the maximum 4GB. We recommend Contoso set the log file size at 2GB which should give us enough data to capture the 642 event but be well below the max size. After increasing the log file size some quick analysis found they now had 3 – 3.5 hours before the security event log wrapped. Our thoughts were this should give us enough time to find Event ID 642.

There is one odd situation that occurred on only one of the Domain Controllers that had its Security Event Log size changed by the GPO. The properties showed that it was properly set at 2GB and the size of the on the disk was at 2 GB, but there were not as many entries in the event log as the others. When changing the size of the Event Viewer Logs, best practice is to use the “clear log” button to allow the event log to properly resize.

Get Event ID 642 from the Domain Controllers

The information provided by the REPADMIN /showobjmeta meant we should only have to search the Security log on the domain controller where the user objects UPN was changed to find the Event ID 642. REPADMIN /showobjmeta gives the precise time when the changed occurred and on which domain controller, allowing our search to specific and limited. Once the Event ID 642 was found in the appropriate security event log we would know the AD account that made the change and could identify 4 of the 5 key variables (who, where, when, what), which would hopefully provide enough information to lead us to the process making the change.

Figure 1 – REPADMIN /showobjmeta output

In the example shown above, Figure 1 - REPADMIN /showobjmeta output, you can see the change that was made to test5455 on DC02NA which we got from the metadata. It also shows the other key piece of information, the AD account initiating the change, which in this case is Administrator.

Armed with this knowledge, we started down the path of creating PowerShell scripts to identify user accounts where the UPN that had been set to an incorrect value and create a “Bad UPN” report/log file with the associated replication metadata. We quickly realized it is going to take an enormous amount of time to do this for all users, we needed subset of users to focus on and that would be quicker.

Contoso IT team came up with 1600 users that they would watch for changes to the UPN and we would use that for the input file for the script. Great, how do we script this in powershell? We looked at the output from REPADMIN /showobjmeta, and quickly surmised that it would take a lot of work to parse the output. How can we do all this in powershell?

We decided to use the PowerShell equivalent of the REPADMIN /showobjmeta, the GetReplicationMetadata method of the System.DirectoryServices.ActiveDirectory.DirectoryContext object for the ease of handling the data. We opened up our favorite search engine, www.bing.com, to start looking for examples and we found one really good example.

AD Replication Metadata (when did that change?)

http://bsonposh.com/archives/253

We used the sample from above website along with some minor changes to retrieve the AD Users object meta data and get the OriginatingServer and LastOrginatingServer values. See figure 2.

Figure 2 – PowerShell REPADMIN /ShowObjMeta example

These values along with the UPN would be written out to a log file, “BadUPN”, so we could use them for searching for Event ID 642. From the “Bad UPN” list, we got the list of domain controllers we needed to get the security event log from.

The Solution – Version 2

Reality set in at this point, it made no sense to repeatedly query the same DCs for the Event ID 642, when we only needed to get the data once. We changed the script so only a filtered list of the DCs would be queried versus all DCs.

Filtered Server List approach

We already had the script looping through the each user identifying their metadata. Part of the information being collected was the name of the server. Using PowerShell, when looking at the server data, if the servers had been removed from the domain, for any reason, the server name returned a Null value. Remember that the server names in AD are actually GUIDs and not the FQDN name. This meant we needed to handle the Null value to ensure it wasn’t included in the array and ensure there weren’t any duplicates server names in the array. FYI, handling the Null value in the array was not obvious to us right way and took a while to figure out why things weren’t working as expected in the script. In Table 1 below, you can see what the Repadmin /showobjmeta output looks like when the DC is missing. The server name looks like “cd5d12e9-ad2e-4e44-a785-f6757f209d4e” when it is missing. When it is there it looks like “Default-First-Site\DC01”.

We checked if the returned Server Name was a Null, and if it was then it was skipped and not put in the Array. If the Server name was a not a Null, then check to see if the array contained this server name. If the array did not contain the Server Name, then add it to the array; otherwise continue on without adding the server name to the array. See Figure 4 for the example code.

Loc.USN Originating DSA Org.USN Org.Time/Date Ver Attribute

======= =============== ========= ============= === =========

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 objectClass

1133651 Default-First-Site\DC01 1133651 2011-08-11 12:04:10 1 cn

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 sn

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 st

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 title

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 postalCode

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 physicalDeliveryOfficeName

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 givenName

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 instanceType

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 whenCreated

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 displayName

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 department

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 company

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 391356011 2011-06-30 10:54:29 3 homeMTA

118496097 Default-First-Site\DC02 65408863 2011-10-26 20:41:02 3 proxyAddresses

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 streetAddress

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 nTSecurityDescriptor

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 mDBUseDefault-First-Sites

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 extensionAttribute9

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 mailNickname

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 employeeType

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 name

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255842 2010-05-11 02:07:58 3 userAccountControl

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255839 2010-05-11 02:07:58 1 codePage

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255839 2010-05-11 02:07:58 1 countryCode

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 employeeID

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255840 2010-05-11 02:07:58 2 unicodePwd

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255840 2010-05-11 02:07:58 2 ntPwdHistory

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255843 2010-05-11 02:07:58 3 pwdLastSet

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255839 2010-05-11 02:07:58 1 primaryGroupID

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255841 2010-05-11 02:07:58 1 supplementalCredentials

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 objectSid

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255839 2010-05-11 02:07:58 1 accountExpires

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255840 2010-05-11 02:07:58 2 lmPwdHistory

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 sAMAccountName

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 division

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 sAMAccountType

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 385295851 2010-05-11 02:08:51 1 legacyExchangeDN

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 userPrincipalName

392184935 Default-First-Site\DC01 392184935 2012-01-23 09:13:47 1 lockoutTime

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 objectCategory

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 msNPAllowDialin

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 385295851 2010-05-11 02:08:51 1 textEncodedORAddress

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 385295851 2010-05-11 02:08:51 1 mail

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 departmentNumber

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 middleName

41048248 Default-First-Site\DC01 41048248 2011-09-07 10:18:35 4 msExchPoliciesIncluded

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 391356011 2011-06-30 10:54:29 3 msExchHomeServerName

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 385296006 2010-05-11 02:09:51 2 msExchALObjectVersion

1133651 cd5d12e9-ad2e-4e44-a785-f6757f209d4e 127255838 2010-05-11 02:07:58 1 msExchHideFromAddressLists

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 385295851 2010-05-11 02:08:51 1 msExchMailboxSecurityDescriptor

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 385295851 2010-05-11 02:08:51 1 msExchUserAccountControl

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 385295851 2010-05-11 02:08:51 1 msExchMailboxGuid

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 610906234 2011-03-02 07:42:52 1 msRTCSIP-InternetAccessEnabled

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 610906237 2011-03-02 07:42:52 1 msRTCSIP-PrimaryUserAddress

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 610906233 2011-03-02 07:42:52 1 msRTCSIP-UserEnabled

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 610906238 2011-03-02 07:42:52 1 msRTCSIP-PrimaryHomeServer

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 610906236 2011-03-02 07:42:52 1 msRTCSIP-OptionFlags

1133651 5c66f360-c067-4e66-959a-d11bba47e42c 610906235 2011-03-02 07:42:52 1 msRTCSIP-FederationEnabled

Table 1 - REPADMIN /showobjmeta output Missing DC

Figure 4 – Filtering DC for the Array

Let’s Test the Script

We were pretty confident that the script would work quite well and it would be something that could be run frequently, like once an hour or two. After the first test taking over 3 hours to complete on just one DC, we were back to rethinking how to improve the performance of collecting Event ID 642 from the list of DCs.

The Solution – Version 3.0

After seeing how long it took to return results we did some more thinking, research and testing on how to improve the performance of collecting the event log entries. While discussing the scenario, it sure made sense to use a multithreaded approach, so we could collect from each DC at the same time. Let’s rephrase the previous sentence into tech speak, “It would be really cool if we could do multithreading with PowerShell.”

Researching PowerShell and Multi-Threading

We fired our favorite search engine, www.bing.com, and starting looking ways to collect from all the DC in the array list the script created. The following sites were extremely helpful in getting us started using PowerShell V2 Start-Job command let.

Ryan's PowerShell Blog

Multi-Threading in PowerShell V2

http://ryan.witschger.net/?p=22

TechNet Library

Start-Job

http://technet.microsoft.com/en-us/library/dd347692.aspx

PowerShell Multi-Threading using Start-Jobs

PowerShell 2.0 has the ability to a form of multithreading and it is called “Jobs.” This would give us the ability to use a parallel versus a serial approach for gathering the events from the event logs. Great, let’s try using “Jobs” to see if it will speed up the security event log collections for the list of DCs. It took a while to figure out what was the proper syntax to use when collecting the security event log from multiple domain controllers. While working through the syntax, we learned that we needed to use the “-ScriptBlock” parameter to get what we wanted to do working correctly. One of the syntax tricks to the Start-Job cmdlet is where the “{“ is placed. Normally for readability of code, the “{“ is place on the next line, but that doesn’t work correctly with Start-Job. The curly bracket needed to go on the same line and after your last parameter for everything to work correctly. The last interesting tidbit of information was figuring out how to use variables in the Job. After reviewing samples and reading forums, we determined that global variables do not work and we needed to pass them. The way to do that is with “-ArgumentList” parameter.

In figure 4 – Start-Job Section, we are showing the working code we came up with to collect the security events logs from multiple Domain Controllers. There is a more to this part of the script than what we have talked about to this point.

Figure 4 – Start-Job Code Section

We tested the script and collected data to prove less time was spent collecting the events from the event logs. The data was collected with start and stop time for each DC we collected event logs from by writing to a log file. It discovered when reviewing the log files that some DC were taking longer than others. When we tested we quickly found out that a couple of the DCs were the bottleneck when it came to collecting the security event logs. The only reason we came up with to explain this was that those slower DCs were averaging more security events/second that the rest.

Event Filtering Performance

We checked to make sure we were using the fastest methods possible for collecting events from the security log. We ended up testing WMI and Get-Eventlog from PowerShell. We found that if we did anymore than just filtering for a single Event ID using “Where” clauses, it took longer to get the data. The reason it takes longer is that it reads the entire event log first and then filters it before out dumping the information to file.

Through multiple tests we determined that using the simple filter for retrieving Event ID 642 only and placing that data in a log file worked the fastest. One other interesting observation we need to share, is the amount of audit data being logged into the Event Logs and how that affects the performance of retrieving the event logs. Especially on Windows Server 2003, the more entries being written the longer it took to retrieve the events from the security log.

Testing & Performance Thoughts

We ran this version through some testing and determined that it did work and actually helped us catch one account that was making the change to the UPN’s. The testing also showed us that we could not run the full version of the script every hour or two like we were thinking. The reality was it needed to be run in the full version right after we knew when the changes were being made to the UPN’s.

We also changed the script to allow it to gather the UPN changes only, but not go out and pull from the DC’s security event logs. The script can be started in in either mode, by using a command line parameter when launching the script. If the script launches without any command line parameters, then it only collects the changed UPN values in a log file. If the ‘Full’ command line parameter is used then the security event log scan is done too.

Command Line Parameters Examples

O365UPNCheckV4.ps1 – will log only UPN update information and NOT gather the Security event logs

O365UPNCheckV4.ps1 ‘Full’ – will log UPN update information and gather the Security event log

Analysis of the Event logs

A Log file is generated from each DC containing the Event ID 642 that has been collected. The log could be quite large and would take a long time to manually review them. To speed that process up, Ray developed another script that will do the analysis of the each of the log files looking for the Event ID 642’s that had the UPN values changed. If a 642 that had its UPN changed, it outputs to a .csv file listing the originating domain controller, date/time, and the new value of the of the UPN attribute. This gives us one place to look for the “who, when and where” the UPN’s changed.

Parallel Tasks

The Contoso IT team was working in parallel to find out what they could about what could be making changes to AD. From the work they had already completed, once they knew the AD account making the change, they were able to identify the offending process in about 15 minutes and get the migration back on track.

PostScript

We hope you found the additional detail we included of how we approached solving the problem, the challenges we went through and the script development process we used. The scripts are attached the blog post and we hope you find those helpful too.

We hoped you enjoyed this post.

Ray and Rick

Attached Sample Scripts

O365UPNCheckV4.txt

ScanForUPN.txt

DHCP, Dynamic DNS, and DCs: How about Some PowerShell to Spice Up a Mind-Numbing Topic?

dsymalla — Mon, 30 Apr 2012 17:54:00 GMT

If the title of this blog hasn’t already put you off, you’re probably interested in the interaction between Microsoft DNS and DHCP services. Specifically, you should understand how Microsoft DHCP servers can be configured to dynamically register A and PTR records in DNS on behalf of their clients.

The default behavior of a Microsoft DHCP server is to only perform dynamic DNS registration on behalf of a client, if the client requests. The default behavior of a relatively modern Microsoft client (XP or higher) is to perform the dynamic registration of their A-record themselves, and to allow the DHCP server to perform the dynamic registration of their PTR-record.

Things become slightly more complex when you use secure-only dynamic DNS, and I hope you do you use secure-only dynamic DNS. Unfortunately, secure-only is not the default configuration for a DNS zone, so you should verify. If DCs run DHCP and perform dynamic DNS registration on behalf of their clients, the potential problem is that DCs are over-privileged with respect to secure dynamic DNS. DCs could, theoretically, hijack any DNS record on-behalf of their clients. Thus, you should use alternate credentials for dynamic DNS registration on the DHCP server.

If you’re really into this stuff, fellow PFE Karam Masri has written a nice, deep in the weeds blog about how DHCP and secure dynamic DNS registrations work (or don’t always work) on Domain Controllers.

At the end of the day, there is some very simple guidance for running DHCP on Domain Controllers – configure DHCP with alternate credentials for dynamic DNS registration. How? Simply use the DHCP management tool, open the DHCP server properties (or IPv4 properties in Windows 2008 R2), then follow three simple steps.

Or if you’re a command-line admin, you can use netsh:

Netsh.exe dhcp server \\servername set dnscredentials username domainname password

Where servername is the name of the DHCP server, username is the name of the user account, domainame is the domain where the user account resides and password is the password associated with the account.

If you just want to see the credentials already configured for a dhcp server:

Netsh.exe dhcp server \\servername show dnscredentials

Some Basic Best Practices

If you’re running dynamic DNS, be sure your zones are allowing “Secure Only” dynamic updates. If you are allowing non-secure and secure updates, alternate credentials are irrelevant and you’ve got bigger security concerns.
When you provision a domain account for alternate credentials, DO NOT grant the account any special privileges. You’ve already got too many service accounts that are over-privileged (and don’t think I don’t know it). Don’t add to the problem.
Even if you’re running DHCP on member servers (not Domain Controllers), you may want to consider alternate credentials for dynamic DNS registration. This makes for a nice transition when you’ve got to replace your existing DHCP server with a new one. If you use the same account on the old and new DHCP servers, ownership of the DNS records will not have to change.

Didn’t You Mention Something About PowerShell?

The real point of this blog was to help you check the alternate credentials for Dynamic DNS, across all of your domain controllers. Manually checking credentials on more than 2 DCs can be a real pain. In fact, I often run into customers who have dozens, if not hundreds of Domain Controllers running DHCP.

Enter the PowerShell Script

If you’ve seen some of my other scripts, some of the code in this new script may look familiar. I basically do the following:

1. Discover DCs (the old fashion way - without using the AD cmdlets, because they require a 2008 R2 DC):

##################################
Function EnumerateDCs
{
     $arrServers =@()
     $rootdse=new-object directoryservices.directoryentry("LDAP://rootdse")
     $Configpath=$rootdse.configurationNamingContext
     $adsientry=new-object directoryservices.directoryentry(LDAP://cn=Sites,$Configpath)
     $adsisearcher=new-object directoryservices.directorysearcher($adsientry)
     $adsisearcher.pagesize=1000
     $adsisearcher.searchscope="subtree"
     $strfilter="(ObjectClass=Server)"
     $adsisearcher.filter=$strfilter
     $colAttributeList = "cn","dNSHostName","ServerReference","distinguishedname"
     Foreach ($c in $colAttributeList)
     {
          [void]$adsiSearcher.PropertiesToLoad.Add($c)
     }
     $objServers=$adsisearcher.findall()
     ForEach ($objServer in $objServers)
     {
          $serverDN = $objServer.properties.item("distinguishedname")
          $ntdsDN = "CN=NTDS Settings,$serverDN"
          if ([adsi]::Exists(LDAP://$ntdsDN))
          {
               $serverdNSHostname = $objServer.properties.item("dNSHostname")
               $arrServers += "$serverDNSHostname"
          }
          $serverdNSHostname=""
     }
     $arrServers
}
##################################

2. Walk through the DCs and use WMI to discover whether or not they are running DHCP:

##################################
Function isRunningDHCP
{
     Param($computer)
     $DHCP = "FALSE"
     $Query = "SELECT Name, Status FROM Win32_Service WHERE (Name = 'DHCPServer') AND (State = 'Running')"
     Try
     {
          $DHCPRunning = Get-WmiObject -Query $Query -ComputerName $Computer -EA Stop
          If ($DHCPRunning){$DHCP = "TRUE"}
     }
     Catch {$DHCP = "FALSE"}
     Finally {$DHCP}
}|
###################################

3. Use Netsh and some string manipulation to determine if DHCP is using alternate credentials for DDNS:

###################################
Function GetAltCreds
{
     Param($computer)
     $AltCreds = $Null
     $Query = Netsh dhcp server "\\$computer" show dnscredentials
     $username = $Query[2].substring(14)
     $domain = $Query[3].substring(14)
     If ($username.length -eq 0){$AltCreds = "NULL"}
     Else {$AltCreds = "$domain\$username"}
     $AltCreds
}
##################################

4. Put it all together and report back the findings. Both on-screen and logged into a CSV file.

Now You Try

Simply download and run the attached script (DHCPDNSCreds.ps1). It requires administrative privileges against your DCs. It will discover all DCs in your forest, and report which are running DHCP and what alternate credentials (if any) are being used. Simply analyze the output (either on-screen, or in the DHCPDynamicDNS.csv file. Look for DCs where Running DHCP=TRUE and AltCreds is blank.

I hope you enjoy the script, and come back for more PowerShell goodness in the future.

Doug Symalla

Who Moved the AD Cheese?

Michael Hildebrand - MSFT — Sun, 22 Apr 2012 16:51:00 GMT

Sometimes, we Microsoft engineers get called into a 'forensics' type situation to help a customer try to answer the "W" questions - where someone (WHO?) did something (WHAT?) at some point (WHEN?) in Active Directory (AD) or some other aspect of a Windows infrastructure. Usually, if we get the call, the change had a big (sometimes catastrophic) negative impact on the company's business or operations.

Depending on how auditing was setup before the event, we may or may not be able to help answer those questions.

This post provides details on how I set up a Windows Server 2008 R2 Active Directory environment for effective auditing of certain AD changes. The changes I chose to audit for this post are a direct result of customer incidents and trying to answer those "W" questions. Some of the incidents resulted in massive outages such as an OU deletion executed via script that was mis-coded and deleted a root-level OU and all contents resulting in thousands of User Accounts getting deleted. Of course, there are additional items that can be audited such as:

Creating and/or deleting objects - User Accounts, Site Links, Sites, etc
Editing/deleting files and folders
Users logging in and/or logging out of the Domain – this one can be tricky to pin down
FSMO Role transfers, Directory Service Restore Mode password changes
Domain/Forest Functional Level changes

This is a post focused on AD object auditing, so I'm not going to cover user logon/log off, file server access or other types of auditing (so little time; so many audit options).

IMPORTANT NOTES AND DISCLAIMERS:

The event details and auditing settings in this post are specific to Windows Server 2008 R2 and are not applicable and/or different in a Windows 2000 or 2003 AD.
!!**WARNING**!! Improper auditing can, among other things:
- SWAMP your DCs and other servers – as with anything, vet this information out as I did - in a lab – proceed with caution.
- SWAMP your Security Event Logs on your DCs or other servers, over writing critical data required for audit compliance – proceed with caution.
- SWAMP your alerting and/or audit collection system and cause Alert storms – proceed with caution.
Auditing is not a 'black or white' technology in Windows and there isn't always a clear answer to the "W" questions, even with auditing enabled.
I chose not to care about failure to make changes – I only cared about successfully making changes, so I enabled SUCCESS audits but not FAILURE audits. This can help to reduce auditing noise. Some would say this reduces visibility into potential denial of service (DOS) attacks.
Turn on auditing at the proper levels. This, too, can help to reduce auditing noise:
- The AD object level –
  - On a partition (i.e. the configuration or domain partition object)
  - On a certain OU(s) or sub OU(s)
  - Other AD objects, such as a certain group or service account
- The AD object inclusion level –
  - This and every object below - "This object and all descendant objects"
  - Only instances of the specific object type = "All descendant OUs"
Consider not enabling auditing for TEST/DEV objects/OUs/ PC/servers, etc
- This can help to reduce auditing noise due to frequent changes in lab environments.
This post only brushes the surface of Auditing in Active Directory and is by no means 'all there is.'

Auditing in AD has come a long way since Windows 2000 but many customers haven't taken or had the time to set it up so that valuable data can be derived from the infrastructure. As a result, often, those WHO/WHAT/WHEN questions for sensitive and/or unexpected/unplanned changes or deletions cannot be answered with empirical evidence.

OK, let's do some stuff!!

Several things need to be addressed before AD auditing can be fruitful.

AD events occur on Domain Controllers; hence, we need to enable Advanced Audit Policy settings on the DCs. In my lab, I set these options in the Default Domain Controllers GPO:

Here's the relevant output of AUDITPOL /get /category: * from the DC:

Here's the setting which forces the newer granular Audit settings to prevent potential conflicts with legacy Audit settings. See the links for a further discussion of this setting: http://technet.microsoft.com/en-us/library/dd408940(v=WS.10).aspx

http://technet.microsoft.com/en-us/library/dd772710(v=WS.10).aspx

Here's one of the audit events for enabling "Success" on Directory Service Changes above (this is audited/logged by default).

Here's a screenshot of the Default Domain Controllers GPO in my lab after my changes:
We need to create one or more System Access Control List entries (SACLs) for what we want to audit.

IMPORTANT – if you enable the above Audit Policy settings but don't also create SACLs, you won't get any audit events from those Audit Policies. I've seen this unpleasant 'surprise' with customers, too.
This is what I set in my lab for this post - adjust to meet your environment's needs/specifics:
- Open AD Users and Computers MMC (DSA.MSC)
- Right-click the Domain or the target AD Object > click Properties

Click the Security tab > Click the Advanced button

Click the Auditing tab
Click Add to begin adding SACL entries

SACL Entries

Activity to Audit - Create and Delete Organizational Units (OUs)

EVERYONE > CREATE Organizational Unit objects > DOMAIN and all descendent objects

EVERYONE > DELETE > Descendant Organizational Unit objects

Activity to Audit - Create and Delete Computer Accounts (including a Move)

EVERYONE > CREATE Computer objects > DOMAIN and all descendent objects

EVERYONE > DELETE > Descendent Computer objects

Activity to Audit - Create and Delete Group Policy Objects (GPOs)

EVERYONE > CREATE Group Policy Container objects > DOMAIN and all descendent objects

EVERYONE > DELETE > Descendent Group Policy Container objects

Activity to Audit - Link and unlink GPOs to OUs

EVERYONE > WRITE GPLink > DOMAIN and all descendent objects

Activity to Audit – Edit Group Memberships and/or Delete Groups

EVERYONE > Write all properties + Delete > Descendant Group objects

SUMMARY SACL LIST TABLE

SACL list for the Domain - After the changes to my lab

You need to consider the increased amount of data gathered as it relates to the size of the Security Event Log on your DCs. You may need to increase the size of the Security Event Log so the data doesn't roll through the Log before you even know you need it.
Some customers enable auditing for Everyone at the Domain level, with all descendent objects, capturing Success and Failure events on everything and think they're all set. However, when they go into the Security Event Log "looking for answers," they're stunned to realize their Security Event Logs wrap in less than 2 hours and the event data they need from the "WHAT THE &*$#@ ?!?" this morning is no longer available.
- I'm stating the obvious here, but this depends on numerous variables such as:
  - How many objects are in the environment
    - An AD with 5 users will produce a lower volume of Audit data than an AD with 50,000 users. Realize that the default Event Log sizes are the same in both cases, though.
  - How many items are being auditing
    - Auditing 4 OUs for deletions will produce a lower volume of Audit data than auditing every attribute on every object in AD for success and failure.

Scenario – Linking a GPO to an OU

If someone links a GPO to an OU, it could produce dramatic results on the contents of that OU, including systems or users falling out of audit compliance. We want to be able to determine the who/what/when for the change.

Event ID 5136 – A directory service object was modified.
This example lists the OU DN path and the linked-GPO's GUID
1. What OU had the link added?
  1. Object "DN:OU=SERVICE ACCOUNTS,OU=-PRODUCTION OU….,DC=LAB"
2. What GPO was linked to the OU?
  1. Attribute:
    1. LDAP Display Name "gPLink"
      1. Value: <GUID of the GPO>
  1. How is this differentiated from removing a GPO Link from an OU?
    1. Operation Type: Value Added

NOTE: In the screenshots I've included, the relevant information to help answer the "W" questions is called out via the red boxes. I labeled this first screenshot with Who/What/When text and arrows, too, but for clarity, on the rest of the screenshots, I only used the red boxes.

You can use a DSQUERY one-liner to derive the Display Name from the GPO GUID, then use GPMC to review the new GPO, if needed.

dsquery * "cn={E83C3E6F-2864-46CD-B6C1-C29CE4D04A88},cn=policies,cn=system,DC=DOMAIN,DC=LAB" -scope base -attr displayname

SCENARIO – Deleting a GPO Link from an OU

If someone unlinks a GPO from an OU, it could produce drastic results on the contents of that OU, including systems or users falling out of audit compliance. We want to be able to determine the who/what/when for the change.

Event ID 5136 – A directory service object was modified.
This example event lists the OU DN path and the un-linked-GPO's GUID
1. What OU had the link deleted?
  1. Object "DN:OU=SERVICE ACCOUNTS,OU=-PRODUCTION OU ….DC=LAB"
2. What GPO was unlinked from the OU?
  1. Attributes to review:
    1. LDAP Display Name "gPLink"
      1. Value: <GUID of the GPO>
    2. Use the same DSQUERY one-liner from the prior example.
3. How is this differentiated from linking a GPO to an OU event?
  1. Operation Type: Value Deleted

SCENARIO – Deleting a GPO

If someone deletes a GPO from AD, it could produce drastic results on the contents of the Site, Domain or OU(s) to which it is linked, including systems or users falling out of audit compliance. We want to be able to determine the who/what/when for the change.

Event ID 5141 – A directory service object was deleted.
This sample event lists the DN path (which is also the GUID) for the GPO that was deleted
1. I could not correlate an Event that listed the Name of the GPO that was deleted and the DSQUERY command from before won't work because the object is gone.
2. I checked in the Deleted Objects Container to see if I could get the Name of the GPO from there but that attribute (along with most others) is cleared upon deletion – no luck.
3. However, I was able to look in my nightly GPO Backups (you do backup your GPOs, right?) and found the GUID for the deleted GPO and got the Name from the GPOReport file that is generated during the GPO backup job.

SCENARIO – Delete an OU

If someone deletes an OU (and everything in it) can produce drastic results. We want to be able to determine who made the change.

Event ID 5141 – A directory service object was deleted
This example event lists the DN path of the OU deleted.

SCENARIO – Moved a computer account from one OU to another OU

This can produce drastic results on the system moved (i.e. a critical application server), including systems or users falling out of audit compliance. We want to be able to determine the who/what/when for the change.

Event ID 5139 – A directory service object was moved.
1. This sample event lists the old and new DN path for the Computer Account.
2. How do we know it was a Computer Account move?
  1. Object > Class: computer

SCENARIO – An OU was moved (possibly drag-n-dropped on accident?)

Moving an OU (and its contents) can produce drastic results on the systems/users in the OU(s). This includes systems or users falling out of audit compliance. We want to be able to determine the who/what/when for the change.

Event ID 5139 – A directory service object was moved.
1. This sample event lists the old and new DN path for the OU.
2. How do we know it was an OU move?
  1. Object > Class: organizationalUnit

SCENARIO – Editing a Group's membership

If someone edits membership to sensitive Groups in AD (such as Domain Admins, Enterprise Admins or others), it could produce drastic results, including systems or users falling out of audit compliance. We want to be able to determine the who/what/when for the change.

Event ID 5136 – A directory service object was modified.
This sample event lists who was added or removed from the group

Group Member Added
- Operation > Type > Value Added
Group Member Removed
- Operation > Type > Value Deleted

SCENARIO – Deleting a Group

If someone deletes a critical Group in AD, it could produce drastic results, including systems or users falling out of audit compliance. We want to be able to determine the who/what/when for the change.

Event ID 5141 – A directory service object was deleted.
This sample event lists the DN path of the group deleted.

BONUS AD Auditing nuggets

If you've stuck with me this long, you must enjoy this stuff as much as I do! So, just between us, here are a few bonus Events for AD environment 'awareness'

Domain Functional Level changed (two events)

Directory Services Event Log Entry

Security Event Log Entry

Forest Functional Level Changed (two events)

Directory Services Event Log Entry

Security Event Log Entry

Directory Services Restore Mode DC Boot-up

RID FSMO Role Transfer

From the prior FSMO Role DC – Directory Service Event Log – notice the "User"

From the new FSMO target DC – Directory Service Event Log – notice the "User"

Domain Naming Master FSMO Transfer

PDCE FSMO Transfer

Infrastructure Master FSMO Transfer

Schema FSMO Transfer

Links:

Eric Fitzgerald's Blog - http://blogs.msdn.com/b/ericfitz/

Minimizing audit noise -http://blogs.msdn.com/b/ericfitz/archive/2005/01/11/350848.aspx
Auditing GPOs - http://blogs.msdn.com/b/ericfitz/archive/2005/08/04/447951.aspx
Audit changes to Audit Policy - http://blogs.msdn.com/b/ericfitz/archive/2010/07/16/auditing-changes-to-audit-policy.aspx
Auditing impact on system performance - http://blogs.msdn.com/b/ericfitz/archive/2009/08/10/auditing-system-impact-on-performance.aspx

TechNet

Advanced Auditing - http://technet.microsoft.com/en-us/library/cc731607(v=WS.10).aspx

AD Object Auditing - http://technet.microsoft.com/en-us/library/cc773209(v=WS.10).aspx

Advanced Security Auditing - http://technet.microsoft.com/en-us/library/dd408940(v=WS.10).aspx

Block accidental deletion of OUs – http://technet.microsoft.com/en-us/library/ee617237.aspx

Look into the "ProtectedFromAccidentalDeletion" switch

Advanced Auditing FAQ - http://technet.microsoft.com/en-us/library/ff182311(WS.10).aspx

Compares legacy and new Audit Policies and how they interaction

Related Ask PFE blog post – Part One of a two-part series on a real-world forensics scenario –

http://blogs.technet.com/b/askpfeplat/archive/2012/03/05/how-to-track-the-who-what-when-and-where-of-active-directory-attribute-changes-part-i-the-case-of-the-mysteriously-modified-upn.aspx

Go-Dos

Get into a lab and run through some of this – a small lab running one DC VM is all you need.
Once you're comfortable with the inputs/outputs in your lab, collaborate with your IT peers/teams and consider rolling out some changes to your Production environment.
Consider combining this information with Event Forwarding/Subscriptions for small-scale environments or a true Audit/Alerting/Monitoring solution such as Ops Manager to achieve near real-time Alerting delivered to a Console and/or a monitored mailbox and ….

KNOW what's happening in your AD!

Additional Screenshot - showing actual User ID

How a DFS Namespace change ruined my morning…

Jake Mowrer [MSFT] — Mon, 16 Apr 2012 10:00:00 GMT

Hello! Jake Mowrer here to share an “experience” with you all that will hopefully help you avoid running into a nightmare the next time you add a DFS Namespace (DFS-N) server. I find that DFS-N in customer environments works well, so well that they require very little maintenance. This tends to lead to admins not wanting to touch the environment unless they really have to, which in turn leads to “We don’t look directly into the eyes of that system, it just works, not sure how and we’re not sure who manages it.”

Ultimately, you will have to add systems to host DFS-N whether due to an OS upgrade, a new AD site rolling out, or just need to scale out a bit. This is what my client and I were doing one late Tuesday night which seemed to go well, until the calls started pouring into the helpdesk the next morning.

We were adding a Windows Server 2008 R2 server into the domain based DFS-N which was entirely Windows Server 2003. The reason we were making this change was that a site was complaining about slow DFS Namespace enumeration. Upon further investigation, we found this site had to hop over two sites to get to the nearest DFS-N. So easy enough, we figured we would add a DFS-N server in a closer site to improve the enumeration time. Here were the steps we used:

1) From a Windows 7 client with the DFS File System RSAT tools installed or a Windows XP machine with the Windows XP Support Tools installed, we ran a DFSUTIL /PKTINFO to confirm what server the client was using for the DFS-N prior to the change. Here was the relevant output:

2) Using the DFS console on the newly installed Windows Server 2008 R2 machine, we added the namespace to the new DFS-N server:

c. We clicked Edit Settings to make sure All Users had read to the share.

d. Clicked OK and it added successfully.

3) We then did a DFSUTIL /PKTFLUSH on the same client, accessed the namespace, then ran a DFSUTIL /PKTINFO, and everything looked fine:

4) The results showed it came up in 2 seconds vs. 20 seconds, so we were happy.

We all went to sleep.

Winter is coming, and so are the helpdesk tickets.

I woke up to my phone ringing and it was the morning shift admin indicating there was a DFS namespace issue. The symptom was that Windows XP clients were receiving this error when accessing a subfolder in the DFS Namespace:

“This file does not have a program associated with it for performing this action. Create an association in the Folder Options control panel.”

“What the heck kind of error is that when trying to open a folder?!?!” I thought to myself.

Here are the things we checked:

1) Ensured the server name was entered correctly in the DFS console, it was.

2) Ran another pktflush on the client, didn’t resolve.

3) Ensure the share existed on the serveralotcloser machine, it did.

By this time it was getting about an hour into the outage, so we were running out of time to find root cause, eventually we needed to back out the change.

Before we did, I had the admin capture a network trace using Network Monitor 3.4 (shameless plug) and repro the issue. We removed the new server from the namespace and the problem was resolved. But how are we going to get these servers added with this issue lingering out there?

I took some time to wake up, eat a Cliff bar, and drive into their office. I looked over the network trace and here is what caught my eye:

That looks like a permissions issue, but I checked the share permissions, so it had to be NTFS!

Researching cases I found one case where my friend David Everett from our DS support team used iCACLS to look at the permissions.

I ran the iCACLS command against the existing servers in the DFS-N (ie. The working servers) and here is what they looked like:

BUILTIN\Administrators:(OI)(CI)(F)

NT AUTHORITY\SYSTEM:(OI)(CI)(F)

BUILTIN\Administrators:(F)

CREATOR OWNER:(OI)(CI)(IO)(F)

BUILTIN\Users:(OI)(CI)(RX)

BUILTIN\Users:(CI)(AD)

BUILTIN\Users:(CI)(WD)

I saw his comment in the case that if the permissions were inherited they wouldn’t replicate.

We checked the c:\dfsroots directory on the server that we removed (it remained there even after removing the server from the DFS-N configuration) and here is what showed up:

BUILTIN\Administrators:(OI)(CI)(F)

NT AUTHORITY\SYSTEM:(OI)(CI)(F)

BUILTIN\Administrators:(F)

CREATOR OWNER:(OI)(CI)(IO)(F)

Again all inherited but notice that the BUILTIN\Users group is not listed. Where were these permissions coming from, we never set them when adding the server to the DFS-N root? Why was BUILTIN\Users missing?

They were coming down from the root of the C: drive, meaning the customer had changed the default permissions on the C: drive in their build process. This filtered down to the c:\dfsroots directory thus “softly” denying “Users” access to the folder. By default on Windows Server 2008 R2, BUILTIN\Users (or computername\Users if you are not on a DC) should have Read/Execute/List Folders on the C: drive. To harden security, the customer removed this ACE on their Windows Server 2008 R2 build.

To fix this on the server we planned on adding to the DFS-N root, we added the BUILTIN\Users group and gave it Read/Execute to the C:\dfsroots directory. We could have added it back to the root of the C: drive but I didn’t want to make that big of a change. Once we added the server back into the DFS-N root we tested using the same method above and it worked as expected. Problem resolved!

To wrap things up:

I hear all the time that hardening a server also adds complexity, this is one prime example.

I asked myself, “So what should we have done different to make the change less impactful?” Here’s my “should have” list:

1) We should have tested with an account that had equal privileges to one that was actually used in the field to access the namespace. I thought we were, but I never asked the customer to confirm, shame on me.

2) We should have checked the NTFS permissions for c:\dfsroots on the new DFS-N root servers after making the change.

I hope this write up helps add one more thing to your checklist when adding a server to your DFS Namespace. Have a great week!

Jake Mowrer

A few things you should know about raising the DFL (and/or) FFL to Windows Server 2008 R2

Greg Jaworski [MSFT] — Mon, 09 Apr 2012 13:00:00 GMT

Hello Greg Jaworski here again to briefly talk about two issues when raising the domain functional level (and/or) the forest functional level to Windows Server 2008 R2. While we have loads of documentation on this and numerous blogs there are a few issues that customers have hit that are a little harder to find.

The first one was first documented (to my knowledge) by Brian Puhl who is a Microsoft employee, but this was not blogged on one of our sites. The link to that blog is below (it is external so the usual warnings apply). I have provided some details below.

http://imav8n.wordpress.com/2007/12/19/replication-version-number-for-your-krbtgt-account-password/

So when you raise the domain functional level to Windows Server 2008 or Windows Server 2008 R2 from Windows Server 2003 or gasp Windows 2000 the krbtgt password will be changed. Some TechNet articles have stated that the krbtgt password is periodically changed but that is not true. There is obvious concern that this password does not change, but this password is very complex and this account is also disabled by default. So back to the topic at hand this password change should not cause issues since we remember the previous password. I have not seen any issues with Windows systems, but I have seen issues with Unix/Linux systems that use 3^rd party AD integration software. In that case simply recycling the daemon fixed the issue since this caused the application to retrieve new Kerberos tickets. This is one of those “it should not break anything” but it should be documented as part of raising the DFL to Windows Server 2008 so that you can be prepared if the unexpected does happen.

The second one is related to the .NET framework prior to version 4.0. Versions of .NET prior to .NET 4.0 do not support the DomainMode enumeration function against a Windows Server 2008 R2 domain or forest. Now not being a developer I have no idea what that function does (well I could guess :) ), but if you have .NET applications that use Active Directory you will want to test and make sure these work, and apply this hotfix if needed. (You did test….right…right)

2260240 FIX: "The requested mode is invalid" error message when you run a managed application that uses the .NET Framework 3.5 SP1 or an earlier version to access a Windows Server 2008 R2 domain or forest

http://support.microsoft.com/default.aspx?scid=kb;en-US;2260240

Resources

What is the Impact of Upgrading the Domain or Forest Functional Level?

http://blogs.technet.com/b/askds/archive/2011/06/14/what-is-the-impact-of-upgrading-the-domain-or-forest-functional-level.aspx

Understanding Active Directory Domain Services (AD DS) Functional Levels

http://technet.microsoft.com/en-us/library/understanding-active-directory-functional-levels(WS.10).aspx

How to raise Active Directory domain and forest functional levels

http://support.microsoft.com/default.aspx?scid=kb;EN-US;322692

FIX: "The requested mode is invalid" error message when you run a managed application that uses the .NET Framework 3.5 SP1 or an earlier version to access a Windows Server 2008 R2 domain or forest

http://support.microsoft.com/kb/2260240

Replication Version Number for your KrbTGT account password?

http://imav8n.wordpress.com/2007/12/19/replication-version-number-for-your-krbtgt-account-password/

W2K3 to W2K8 and W2K8R2 Active Directory Upgrade Considerations

http://blogs.technet.com/b/glennl/archive/2009/08/21/w2k3-to-w2k8-active-directory-upgrade-considerations.aspx

Upgrade Domain Controllers: Microsoft Support Quick Start for Adding Windows Server 2008 or Windows Server 2008 R2 Domain Controllers to Existing Domains

http://technet.microsoft.com/en-us/library/ee522994(WS.10).aspx

Greg Jaworski

The 411 on the KDC 11 Events

dgreg - MSFT — Thu, 29 Mar 2012 05:25:00 GMT

Disclaimer: For brevity and to get some key points across, quite a bit of detail about Kerberos has been purposely ommitted from this blog entry. I'm certain that those that are experiencing any of the problems below won't mind :)

As a Premier Field Engineer, I visit new customers every week and every customer, and I mean every customer, has the KDC 11 events in their system event logs. Consequently, I have to explain to customers what this means and how to clean it up. But rather than just saying, "Look, these accounts have a duplicate SPN's and use setspn or adsiedit to clean them up", I like giving the back story about how duplicate SPN's break authentication and what would happen if the KDC issued Kerberos tickets for resources with duplicate SPN's. So, here's the dialogue of my weekly explanation of Kerberos and duplicate SPN's. By the end, most customers have that light bulb moment and “Get It”. At the end of this, you should be able to do the following:

1.) Fully understand what duplicate SPN's are and the Kerberos Event ID 11.

2.) Understand why duplicate SPN’s break Kerberos authentication.

3.) How to find all duplicates across the entire forest with one simple command.

4.) Be able to determine which account the SPN should be set on.

5.) Understand the Kerberos Event ID 4: "KRB_AP_ERR_MODIFIED".

What is this thing called a SPN?

It stands for service principal name and in most cases, it is the name of a resource that a client or application is connecting to. Every computer account has an attribute named ServicePrincipalName, which is usually populated. Users also have this attribute but it is normally blank unless some application is running under this user account.

Fileserver.contoso.com’s ServicePrincipalName attribute:

Let’s apply this to the real world and do some role playing. Imagine you are a phone operator and are responsible for issuing Kerberos tickets so I can talk to various people. But instead of Active Directory, let’s replace the directory with a phone book but it’s a super-secret phone book where the only two parties that know the phone numbers are the phone operator and the respective owner of each phone number. I realize no one uses phone books anymore but play along with me. :)

Ok, so there is this person I need to talk to named ‘John Smith’ and before I talk to John Smith, I need to acquire a Kerberos ticket to talk to him. I call you, the phone operator, and say that I need a Kerberos ticket for John Smith. In this instance, ‘John Smith’ would be the SPN. What’s the first thing you do with that SPN? Well, you’d look in your phone book for the name John Smith, right? Now, let’s say that when you find him, you take his super-secret phone number and encrypt a Kerberos ticket with it. Now when I talk to John, I provide him with this Kerberos ticket. Since he knows his own phone number, he can decrypt the Kerberos ticket, which proves it was issued by the phone operator, a trusted authority, and then we continue our conversation. But what would you do if you found multiple people in the phone book named ‘John Smith’. You’d probably ask for more information like address to further narrow it down, right?

Well, Kerberos and the domain controllers aren’t this smart. They take the name given and look for all accounts with an identical SPN attribute based on string match. If it finds two accounts with the same SPN attribute, it immediately throws a KDC 11 event, saying a duplicate SPN was found and replies to the client with KDC_ERR_C_PRINCIPAL_UNKNOWN. The client says "Cool, I'll just go ahead and use NTLM instead". But what would happen if you just randomly picked one of the John Smith’s, and encrypted a Kerberos ticket with his corresponding phone number?

Well, if you picked another John Smith than the one I needed to talk to, when I provided the Kerberos ticket over to my John Smith, he wouldn’t be able to decrypt it and Kerberos would fail. When this happens, my John Smith would throw a ‘KRB_AP_ERR_MODIFIED’ error because he can’t properly decrypt the Kerberos ticket. If you get the role play we just went through, you're well on your way to understanding Kerberos and duplicate SPN’s. Congratulations! :)

Key Takeaway: The SPN attribute is the sole means by which the DC determines which account password to encrypt the Kerberos ticket with. Consequently, SPN attributes across all accounts must absolutely be 100% unique across the entire AD Forest, without fail. We'll be discussing in Part 2, why this is, and how you can have duplicates across two different domains.

Now, If we apply this example to Active Directory and how we authenticate to resources, it would be almost identical:

Client connecting to file server

In this example, the user just connects to a share on the fileserver by going to \\fileserver.contoso.com. Upon doing so, the client will then go to the domain controller and request a Kerberos ticket for Host/fileserver.contoso.com. The domain controller uses this SPN provided by the client to determine which account password to use to encrypt the Kerberos ticket with. Domain controllers don't issue SQL queries but for all you SQL geeks out there, the query to find the account would be similar to the following:

Select * from Active Directory where ServicePrincipalName = Host/fileserver.contoso.com

So in this scenario, can the user decrypt the Kerberos ticket intended for the fileserver?

A: Nope, because it’s encrypted with fileserver’s password.

Can the file server decrypt the Kerberos ticket?

A: Yes, because the ticket is encrypted with fileserver’s password.

Key Takeaway: This above scenario worked because the Host/Fileserver.contoso.com SPN was only set on the same AD account that we submitted the Kerberos ticket to. Hence, the fileserver could decrypt the ticket and then determine whether you have access to the file share. The above scenario is how it should work.

Now when it comes to Kerberos and SPN's, always ask yourself the following questions:

1.) Who has to decrypt this Kerberos ticket?

2.) So that it can properly decrypt the ticket, what account must the SPN only be set on to make this work?

From now on, use these questions because the scenarios will start to get a little tricky. The following are the most common scenarios that cause duplicate SPN’s. By going through it this way, you should be able to directly apply this knowledge to most scenarios you encounter.

Scenario #1: Improper process for removing old accounts

Your file server crashed one day and had to be rebuilt. You renamed the old computer account in AD to fileserver_old, rebuilt the server, and then joined it back to the domain with the same name it used to have - fileserver.contoso.com. While their computer names may be different, their SPN attributes are still the same.

When the domain controller searches AD for the HOST/fileserver.contoso.com SPN provided by the client, it finds two computer accounts. At this point, the DC throws a KDC 11 event to the system log and replies to the client with KDC_ERR_C_PRINCIPAL_UNKNOWN. The client then attempts to use NTLM to the fileserver, which will probably work and they will gain access.

1.) Who has to decrypt this Kerberos ticket?

A: When connecting to this File server, the file server has to decrypt the Kerberos ticket.

2.) So that FileServer can properly decrypt the ticket, what account must the Host/Fileserver.contoso.com SPN attribute only be set on to make this work?

A: This SPN must only be set on File Server’s computer account in AD.

3.) Similar to the ‘John Smith’ scenario above, what would happen if the domain controller just randomly picked one of the two computer accounts to encrypt the Kerberos ticket with?

A: The user would submit this Kerberos ticket over to the fileserver and the fileserver may not be able to decrypt the ticket because it could be encrypted with fileserver_old’s password. Remember, the domain controller would never do this because instead it logs a KDC 11 event and replies to the client with KDC_ERR_C_PRINCIPAL_UNKNOWN at which time the client will attempt to use NTLM over the file server. But thinking about this way really highlights the importance of why SPN’s must only be unqiue to each account.

Resolution

Iron out your process for removing old computer accounts in a timely manner. If you’re already having this issue, either delete the fileserver_old account if it is no longer valid or remove the HOST/fileserver.contoso.com SPN from the fileserver_old computer account using setspn or adsiedit. Once you think you fixed the problem by either of the above solutions, from a client, try to connect to \\fileserver.contoso.com again and then run ‘klist tickets’ from the command line. If Kerberos is now working, you’ll see that you acquired a Kerberos ticket for HOST/fileserver.contoso.com. Since computer objects have a SPN for both NetBIOS and FQDN, we would need to remove both from the fileserver_old computer account:

setspn -D Host/Fileserver fileserver_old

setspn -D Host/Fileserver.contoso.com fileserver_old

Scenario #2: Changing accounts that the SQL service runs under

The SQL Admin is asked to install a new SQL server. During the install, he hits Next, Next, Next. Accordingly, SQL is now running under the local system context, which is the computer account. Consequently, the MSSQLSVC/SQLServer.contoso.com SPN’s were added to the computer account. Many months later, company policy dictates that SQL must run under a service account. The same admin configures SQL to now run under the SQLAdmin service account. The SPN’s are then added to this SQLAdmin account. We now have duplicate SPN’s again.

Remember those questions I had you take note of earlier. Let’s go through them again for this scenario.

1.) Who has to decrypt this Kerberos ticket?

A: When connecting to this SQL server, the account that SQL is running under has to decrypt the Kerberos ticket, NOT the SQL computer account. So, in this case, it would be SQLAdmin.

2.) So that SQLAdmin can properly decrypt the ticket, what account must the MSSQLSVC/SQLServer.contoso.com SPN attribute only be set on to make this work?

A: This SQL SPN must only be set only on the SQLAdmin service account.

Resolution

Use setspn or adsiedit and remove the MSSQLSVC/SQLServer.contoso.com SPN from the SQLServer.contoso.com computer account while leaving it on the SQLAdmin account.

setspn -D MSSQLSVC/SQLServer.contoso.com SQLServer

Scenario #3: Changing accounts that an IIS application runs under

This one is almost identical to the SQL one. The IIS admin is asked to install a new IIS application called http://intranet.contoso.com. During the install, he creates a new application pool within IIS and selects the local system account to run it under, which is the web server computer account. He then registers the HTTP/intranet.contoso.com SPN on the web server computer account. Later that year, he’s asked to change the web application to run under a service account. He switches the application pool to run under this new service account called WebAdmin. He also adds the HTTP/intranet.contoso.com SPN to this new service account.

Once again, remember those questions I had you take note of earlier. Let’s go through them for this scenario.

1.) Who has to decrypt this Kerberos ticket?

A: When connecting to this web application, the account that the web application is running under has to decrypt the Kerberos ticket. This would now be WebAdmin.

2.) So that WebAdmin can properly decrypt the ticket, what account must the HTTP/intranet.contoso.com SPN only be set on to make this work?

A: The SPN must only be set on the WebAdmin service account.

Resolution

Use setspn or adsiedit and remove the HTTP/intranet.contoso.com SPN from the webserver.contoso.com computer account while leaving it on the WebAdmin service account.

setspn -D HTTP/intranet.contoso.com webserver

Scenario #4: The SPN is set on the wrong account

I left this scenario for last because while it doesn’t include any duplicate SPN’s, it can be a by-product of improper SPN cleanup and will cause authentication issues and once again highlights the importanace that the SPN must be configured on the right account. In any of the above scenarios, what if the admin was troubleshooting the Kerberos authentication issue and removed the SPN attribute from the wrong account, what would happen? Let’s go back to the original file server scenario and play it out. In this scenario, you discovered that you have duplicate SPN's and remove the HOST/fileserver.contoso.com SPN from fileserver's computer account while leaving this SPN on the JoeAdmin user account.

1.) Who has to decrypt this Kerberos ticket?

A: The file server computer.

2.) Will the file server be able to decrypt the Kerberos ticket?

A: No, when this happens, the file server will throw an Event ID 4: KDC_AP_ERR_MODIFIED to its system log because it is unable to decrypt the ticket because it’s encrypted with JoeAdmin’s password.

3.) In this scenario, will the client fail over to NTLM and successfully authenticate to the fileserver?

A: Actually, no. The client will only fail over to NTLM if it is unable to obtain a Kerberos ticket for the resource in question. Since the client was able to get a Kerberos ticket, although one that doesn't work, the client will continually try Kerberos and it will fail until someone fixes the problem and the client purges and refreshes its Kerberos tickets. This is important to remember.

Resolution

Use setspn or adsiedit and put the HOST/fileserver.contoso.com SPN back on the fileserver.contoso.com computer account and remove this SPN from the JoeAdmin user account.

Remove them from JoeAdmin

setspn -D HOST/fileserver JoeAdmin

setspn -D HOST/fileserver.contoso.com JoeAdmin

Add them back to the fileserver computer account:

setspn -A HOST/fileserver fileserver

setspn -A HOST/fileserver.contoso.com fileserver

Key Takeaway: This is not the only scenario that will cause this error but when the SPN is set on the wrong account other than the one we are submitting the Kerberos ticket to, authentication will continually fail, the user will not be able to authenticate to the resource, and KDC Event ID 4 will be logged to the system log on the destination file server.

How To Find Your Duplicate SPN’s:

Up until now, we’ve focused on troubleshooting duplicate SPN’s based on some given scenarios you might be having because you’ve seen KDC Event ID 4 or 11. What about any duplicate SPN’s you have in AD that just aren’t causing problems yet but may one day. How do we find those without spending many hours writing some PowerShell magic? Thankfully, the new version of setspn that is present on 2008 R2 will find all duplicates across the entire FOREST. Addtionally, this command only takes about a minute to run. Now, I’ve seen where a 2008 R2 server will have two versions of setspn installed and if you just go to the command prompt and run setspn, this version doesn’t support the necessary switches. Consequently, make sure to use the one located in %systemroot%\system32. The following is some example output:

C:\windows\system32\Setspn –x –f

Host/fileserver.contoso.com

CN=Fileserver,CN=Computers,DC=Contoso,DC=Com

CN=FileServer_old,CN=Computers,DC=Contoso,DC=Com

MSSQLSVC/SQLServer.contoso.com:1433

CN=SQLSERVER,CN=Computers,DC=Contoso,DC=Com

CN=SQLAdmin,CN=Users,DC=Contoso,DC=Com

HTTP/intranet.contoso.com

CN=WebServer,CN=Computers,DC=Contoso,DC=Com

CN=WebAdmin,CN=Users,DC=Contoso,DC=Com

The first line is the duplicate SPN and the next two lines are the two accounts that have that SPN set. If you run this command in your environment, most of the duplicates you have will probably match one of the given scenarios above. Now, if you have multiple domains and run this command, at the very bottom of the setspn output, you will see the reported SPN as having a duplicate:

kadmin/changepw

CN=krbtgt,CN=Users,DC=Contoso,DC=com

CN=krbtgt,CN=Users,DC=ChildDomain,DC=Contoso,DC=com

Warning: These are by design. Do not mess with these SPN's or these user accounts

Nonetheless, you now you have the knowledge to deal with these and clean them up.

This Kerberos blog will more than likely become a mini-series. In part two, I’ll talk about duplicate SPN's between different domains/forests, and clustered applications and how to identify and resolve those. Part 3 will be inter-domain and inter-forest Kerberos authentication and Part 4 will be double-hop Kerberos authentication and constrained delegation. Please leave some feedback. Thanks.

-David Gregory

Base-Build Bullet-point List-o-rama

Michael Hildebrand - MSFT — Wed, 21 Mar 2012 21:37:00 GMT

Alot more goes into a "well managed" base OS build design beyond booting from the OS media and then "Next > Next > Finish." The content of this post is the outcome of many fruitful whiteboard sessions around Windows base-OS builds. Some of this applies to physical servers only, some to virtual only but most is applicable to both. Many customer's build processes were designed/built circa Windows Server 2003 and XP or earlier. Alot has changed since then.

Have a look and see if some of these points don't get you fired up about expanding and/or improving your own base OS build system/processes.

Rule #1: Document everything.

Consider creating a SharePoint site for build information/documentation

How-To Docs
Standards
Version info
Boot disk images (if needed)
Contact info
Training/PPT
Shortcuts to boot images or other paths

Specifics

Standardize on hdwr mfg/models/components (to minimize the variety)

Consider a series of ‘hardware templates’ for VMs (low util, standard util, high util)
Consider a series of specs for physcial servers - standard util and high util
RAM
CPU
Local storage
USB
Optical
Standardize on a label/ID process for phys servers
- Front/rear panel label stickers w/ server name (minimum)

Create an “Advisory Board” for the build to get input from various elements across the business and IT
- Ensures a ‘common’ build is developed (where/if possible)
- Ensures consistency across the business (where/if possible)
- Get the network team there for buy-off on the network suitability for the build traffic, DHCP/non-DHCP segments, unicast, multicast, etc
- Talk to the desktop team – they likely have a build mechanism(s) in place and you may be able to integrate with or build off of what they have
Standardize on hdwr/firmwr/sw/ROM/driver versions and update frequency, testing,
Use Policy to set/reinforce settings along the 90/10 rule whenever possible
- Define Local Group Policy settings aligned with corporate policy
- Define AD GPOs to reinforce settings aligned with corporate policy
- Use Exception OUs/GPOs for the exceptions
- Have a solution for getting Local GPO standard settings applied to non-Domain joined systems
  - DMZ
  - LocalGPO tool in MSFT Security Compliance Manager Toolkit
Use a flexible process to create the builds so they can easily be maintained and modified going forward

Consider scripted build vs image-based build (WIM-based or block-based)

WDS
SCCM w/ OSD
Manual
VM templates - don't forget SYSPREP!

Service Pack, patch, driver updates and other changes should be easily added to the ‘base build’
Design/document a policy to update the build at certain time intervals/milestones
- 1x, 2x per year
- Every Service Pack
- ?
Consider off-line builds as well as network connected processes
Consider DMZ builds/rebuilds
Consider remote/branch office builds/rebuilds
Consider partnering w/ hdwr vendor to deploy the build prior to ship/delivery for large-scale roll-outs/refreshes
- Consider security ramifications of doing so
Base the process around defaults/common tools – don’t overly customize a system
- Leads to a single point of failure and a possible bottle neck as the current enviro is reverse-engineered by someone
Consider if DHCP is a requirement of the build process or if static NIC entries can be made
Develop a numbering/tracking system for build versioning
- Service Pack levels
- OS platform (x86/x64)
- OS version (Standard/Enterprise/Datacenter – 2003/2008/R2)
- Core or full GUI
Consider the workloads/roles

Are most/common reqs met?
- AD/SQL/EXG/IIS/TS/etc

Logical/physical drive setup

Logs/DB spindles
SAN
- Space capacity?
- HBA slot(s) avail?
- iSCSI NIC slots/ports avail?

Standardize on the various high-level elements of the build
- Drive config
- Network config
  - Will there be NIC teaming required?
    - Network/switch port capacity for teaming?
    - Drivers/versions/firmware on NIX
    - Supportability statement reminder
    - Fault toler? Load balance? Auto?
    - Speed/duplex settings?
  - What slot will the NIX go in?
  - Consider additional slot use/capacity planning
    - Controller(s)
    - HBA(s)
    - Additional NIX (i.e. VM host server)
  - Naming of the NIX – be consistent and helpful (slot/port/speed//etc)
    - Consider naming them so that based on the name, they can be ‘found’ in the OS on the server – ‘which is which’?
  - Decide to use hdwr vendor drivers or in-box MSFT drivers
  - IPv6 – enabled/disabled/supportability
  - NetBIOS over TCP/IP – enabled/disabled?
  - DNS suffix list?
  - NIC setting standards
    - WINS? Multiple entries – unless it is a WINS server (in which case, it points to itself only)
    - DNS? Multiple entries
- Go through ALL BIOS settings and understand them
  - Consider the various settings/values
  - Define and document the standard
    - See if the hdwr vendor has a way to automate/replicate setting all servers to the spec (HP SmartStart Scripting Tools)
- Go through ALL Controller settings and understand them
  - Consider the various settings/values
  - Define and document the standard
    - See if the hdwr vendor has a way to automate/replicate setting all servers to the spec (HP SmartStart Scripting Tools)
Consider server naming standards and flexibility (vs strict adherence) to be entered during the build process
Consider domain-joins and computer account (pre)creation in AD
- This also includes OU location within AD to ensure proper OUs are applied and security policy is applied as expected/required/desired
- Consider rebuilds, too, and/or existing computer accounts needing to be ‘touched’ prior to (re)deploying a build
Consider time zone settings being configured as part of the build process, if desired
Consider a ‘post build’ script or manual checklist that will verify/validate items
- SCCM/DCM/other inventory tools?
Consider logging during the build to ease troubleshooting what can become a VERY complex collection of tasks
Consider how complex (and light-touch) you want to design vs how simple (and more-touch)
Consider asset tracking systems/updates as part of a build process if needed
Control access to the build images to help control sprawl and casual/undocumented changes
Consider change mgmt. as required
- No builds during office hours due to network impact
- Isolated/insulated/dedicated segment for builds
- Is a change request needed to build a server?
- Ensure there is tracking within the build system to answer the common questions - possibly a custom reg key(s)?
  - Who built this system?
  - When was it built?
  - What version of build/components?
Consider ‘thick’ or ‘thin’ (build type - not to be confused with fixed-size vs dynamically expanding VHDs)
- Thin = starting with just what’s on the OS install media
- Thick = complete, fully-loaded end point system
  - 3^rd party agents
  - All settings
  - On-going mgmt. of both options
Consider aspects of the OS

Strongly consider the default settings of current OS versions

W2k8/R2 are secure out of the box
Supportability
- Will some custom setting revert when a Service Pack is applied?
- Will some patch install make assumptions that aren’t valid on a highly-customized build?
- Third party tools/agents/etc
  - Many are developed using the base OS default settings
Auditing design - base OS builds and the build system itself

What do we want audited and what do we need to answer the questions

Who/What/When/Where

This is likely bigger than a base build component, but it is part of a base build

Local Policy Settings
- Security Policy
- Other settings
Power Management
- Often an interaction between this and the hdwr vendor/BIOS settings, drivers, firmware
Pagefile details
- How big?
- Separate spindle?
Desktop layout and “Folder” or view options, BG Info?
System failure behavior
- Full dump
- Mini-dump
- Kernel dump
- Dump file location
  - Lots of RAM might mean huge pagefile
    - Consider disk space requirements
- Auto-reboot
- Over-write existing file?
Windows Firewall Profiles, Network Location Profiles
Backup – either in-box or 3^rd party add-on
User Account Control settings
- During the build – preventing some 3^rd party drivers/utils to install?
- After the build – define the design of UAC, set via Local GPO; reinforce/manage via AD GPO
Remote Desktop – enabled?
Services state
- Auto/Manual/Disable
- Supportability
WinRM?
Powershell code signing reqs?
Activation/KMS
IPv6?
- Enabled/disabled
- Supportability reminder
3rd party or add-on Agents
- Backup
- Monitoring
- Management
Application ‘platform’ elements
- App install location/path/folder(s)
- Permissions req’d for app run/install?
  - NTFS
  - Registry perms
- Data drive size/letter
- Local policy settings and any app impacts?
  - Advanced User Rights
  - Security Settings
Permissions model
- Do we stray from defaults on the System drive?
- Data drive?
Customer Improvement Experience Program settings
Error reporting settings (‘Do you want to send this to the Internet?’)
Recovery Tools or hidden partitions
- WINRE
Mgmt of local Admin account and password

Enabled? Disabled?
On-going pwd mgmt. of some type of local Admin-level account

Consider aspects of 3^rd party agents, tools, add-ons
- Where to install?
  - C:\Program Files?
  - A/V exclusion mgmt.
    - What files/folders are we real-time scanning?
  - Security Policy settings req’d by any local service accounts?
    - Log on as a batch job
    - Act as part of the OS
    - Etc
  - Additional pre-reqs for these tools
    - .Net versions
    - WebDav
    - Java
Management of the builds

Updating driver packs, etc
Updating Service Packs
Updating 3^rd party agents
Firmware/ROM updates
DR – can you restore the box with your normal methods after one or more of these updates have been applied?

Provide training to staff on how to build/rebuild
- Avoid one-offs or manual builds getting into Production
- Ensure process/procedures/standards followed
Consider DR of the build system(s)
- Can you restore a base-built server?
- What if a key DB gets corrupt?
- What if a config change error is made? How do we get back to the previous setting(s)
- The simpler, likely, the better – from many standpoints incl DR
- Change tracking

Well, I've pretty thoroughly tested out the "bullet point" feature of the blog editor.

Ping us back in the comments and let us know what works/doesn't work in your experiences.

Cheers!

Hilde

The Journey of a Thousand VMs Begins with a Few Steps

Michael Hildebrand - MSFT — Sun, 18 Mar 2012 00:42:00 GMT

As PFEs, one of our major roles and responsibilities is to relay best practices and “lessons from the field.” In this post, I’ll present a method – not the only method – of how to progress through a Hyper-V infrastructure design. This is a high-level post and the content should not be considered “enough” to arrive at a suitable end-result design, but hopefully, this helps the reader along the virtualization path and will stimulate some thoughts and discussions.

NOTE: the focus here is on server virtualization only and does not include aspects for desktop/VDI or application virtualization (such as RemoteFX or App-V). Nor does this discussion address all aspects of a Private Cloud solution (more on Private Cloud details can be found here: http://www.microsoft.com/en-us/server-cloud/private-cloud/overview.aspx)

High Level Design Steps for a Hyper-V Deployment

Benchmark your dev/test/prod server fleet and establish your candidates for virtualization

We offer a free toolkit that can inventory an environment and produce very detailed reports and information to help with this (the MAP is a very useful tool beyond just virtualization efforts, too)

http://technet.microsoft.com/en-us/solutionaccelerators/dd537570

You may have your own tool(s) or may already have an established list of virtualization candidates

Determine availability requirements of the applications/workloads/VMs

Do the service levels of the applications/workloads allow for routine maintenance of the system?

A departmental application that is typically used during business hours only

Are there requirements for the app to sustain high levels of availability?

Mission-critical line-of-business application that is used 24x7

Consider the deployment location/environment where the VM guests will be hosted

Branch office – often a single-node host deployed on fault-tolerant server hardware
HA Branch Office – often a two-node Failover Cluster deployed on fault-tolerant server hardware
Centralized Data Center – often one or more multi-node Failover Cluster ‘farms’

Determine the desired VM Guest ‘hardware’ profile(s) – vProc, vRAM, VHD(s), vNIC(s)

One idea is to create typical use-case profiles such that the number of VMs per physical host can be easily predicted/budgeted

Low Utilization VM – 1 proc; 768 MB RAM; 20 GB C:\
Standard Utilization VM – 1 proc; 1024 GB RAM; 40 GB C:\
High Utilization VM – 2 proc; 2048 GB RAM; 60 GB C:\

Another idea is to spec each VM based on detailed measurements/requirements for each particular workload.

This can provide more optimal use of physical host server resources but can be more difficult to accomplish due to variations of server workloads and additional time to benchmark/perfmon each application

Application XYZ measured out for 1GB RAM and two Procs
Application ABC measured out at 768 MB RAM and one Proc

SCOM/SCVMM and Dynamic Memory features can help facilitate this effort more easily

Determine the number of planned VM Guests and consider future capacity needs
Determine the OS for the VM Hosts

Microsoft Hyper-V Server 2008 R2 SP1

Free download
Command-line only interface
Hyper-V Role only
http://www.microsoft.com/en-us/server-cloud/hyper-v-server/default.aspx

Microsoft Windows Server 2008 R2 SP1 CORE install + Hyper-V Role

Full feature for-cost OS
Command-line only interface
Hyper-V Role (additional Roles available/supported)

Microsoft Windows Server 2008 R2 SP1 GUI install + Hyper-V Role

Full feature for-cost OS
Full GUI ‘typical’ Windows interface
Hyper-V Role (additional Roles available/supported)

Compare features and limitations of the free/Standard/Enterprise/Datacenter versions of Hyper-V

http://technet.microsoft.com/en-us/library/ee815283(v=WS.10).aspx

A few pro/cons for CORE vs GUI OS versions

CORE – pro

Fewer patches than GUI = fewer reboots due to maintenance
Smaller attack surface than GUI
Fewer ‘casual’ logons/administration due to lack of typical tools/consoles available on the GUI versions of the OS

CORE – con

Separate/additional build to maintain from GUI version of OS
Admin skillset for managing a command-line OS is not as prevalent as GUI versions of the OS
Some 3rd party apps/agents/tools have requirements for some of the GUI elements that CORE lacks

Determine the VM host storage architecture/model

Single node host

Direct-attached storage (DAS) - predominantly SAS but becoming SSD

Two-node Failover Cluster

DAS – predominantly SAS
SAN - predominantly iSCSI or fibre channel

Multi-node Failover Cluster ‘farm’

SAN – predominantly iSCSI or fibre channel

Determine the storage architecture details for the VM host(s)

RAID requirements
Controller redundancy requirements
Controller cache requirements

Determine the hardware requirements of the VM Host servers

CPU(s)
RAM
Local storage
NIX
SAN connectivity
Out of band mgmt of the host server(s)
Consider the additional overhead of one or more Cluster Failover events and the additional load from the VM guests when they are migrated onto the remaining node(s)

Finally, a few great resources for some specific Hyper-V details:

The Virtualization Calculator: http://www.microsoft.com/en-us/server-cloud/windows-server/hyper-v-benefits.aspx#calculator

This is in addition to the ROI and cost savings information/reports which can be found in the Assessment and Planning Toolkit mentioned earlier.

Hyper-V Security Accelerator: http://technet.microsoft.com/en-us/library/dd569113.aspx
Hyper-V Patch/Hotfix Wiki (keep up with your Hyper-V patches well-beyond the monthly updates): http://social.technet.microsoft.com/wiki/contents/articles/1349.hyper-v-update-list-for-windows-server-2008-r2.aspx
W2k8 R2 Performance Tuning Whitepaper (includes a Hyper-V section but it’s all excellent information): http://msdn.microsoft.com/en-us/library/windows/hardware/gg463392.aspx
Hyper-V Clustering Survival Guide: http://social.technet.microsoft.com/wiki/contents/articles/239.aspx
Here's a fun site created when Hyper-V first released in WS 2008
http://www.microsoft.com/canada/windowsserver2008/serverunleashed/default.html
Some cool videos can be accessed by clicking on the "Meet IT 24-7" box
My personal fave is “The Command Line”

Hopefully, the information presented here provides some food for thought regarding your Hyper-V deployments.

Cheers!

Central Store and ADM Removal Q&A (with an updated script!)

Tom Moser [MSFT] — Wed, 14 Mar 2012 12:00:00 GMT

We’ve recently received some great feedback from several customers and other PFEs on the ADM template post that Mark and I wrote a month or two back. I thought I’d take some time to respond to some of those questions and to post an updated version of the script. Going forward, we plan to spend some time responding to questions, either on previous posts or new topics, so keep them coming!

Question:

The dates in the script for the out-of-box ADM templates are different than what I have on SYSVOL. Why is that?

Answer:

We tried to track down all of the possible dates for out-of-box ADM templates and missed a few. I’ve added those to the list, as well as added a new switch to the script –NoDateCheck. Using this switch will cause the script to remove all ADMs where a matching ADMX template is found, regardless of time stamp.

Question:

What’s with all of the dates for the ADMs in the script, anyway?

Answer:

When Mark and I were originally discussing this post and the script, we decided to err on the side of caution. We wanted to be 100% certain that nobody had modified the original ADMs and that there wouldn’t be unexpected behavior, or anything missing, when the migration to ADMX was complete. Checking for the time stamps ensured that things were good. The original plan was to have script users add their own “known good” ADM dates to the script, then run it to verify ADM consistency. For example, if I created a custom ADM and added it to GPO A, then modified that custom ADM again and deployed with GPO B, I’d have two different versions of the ADM template. After, I convert the ADM from GPO A to ADMX and place it in the store. I delete all of my ADMs then I attempt to edit GPO B. Since it’s using an ADMX template that differs from the ADM it was using originally, you may be missing settings or options. We wanted to prevent that.

If you’re 100% certain you won’t see this issue, feel free to use the –NoDateCheck switch I mentioned above to remove all ADMs where a matching ADMX name is found.

Question:

Can I use the script if my organization hasn’t implemented the GP central store? (See: An Alternative to the Central Store in this AskDS post).

Answer:

With the old script, no. With the new one, it’ll check for the central store, first. If it doesn’t exist, it will check the default policy definitions folder (C:\windows\policydefinitions) and use that as the ADMX source. If you’ve got any non-standard ADMX templates, and you aren’t using the central store, you’ll need to ensure those ADMX/ADML files are in c:\windows\policydefinitions.

Question:

There’s a switch in the script, -ADMCSVPath, and I’m not sure why it’s there. So why is it there?

Answer:

Good question. I forgot to remove it. Thanks for reminding me.

Question:

Is it possible to add my own ADMs to the script so that I can check for and remove Office ADMs, custom ADMs, and more?

Answer:

You sure can. If you go to line 130 in the script where all of the out-of-box ADMs are listed you can add each of your own ADMs, as well as the date stamp you’re expecting to find on that ADM. This is only necessary if you want to check specific dates on the ADMs. If you’re comfortable running the script with –NoDateCheck, it’ll remove any ADMs it finds with a matching ADMX regardless of the date stamp.

Question:

I’ve run the script and cleaned up all of the ADMs on sysvol. How can I keep ADMs from re-appearing?

Answer:

First, make sure that all GPO admins are on Vista SP1/Server 2008, or later (read this if you have Vista/7 mixed). If you’re worried about a rogue GPO admin showing up and using an older OS, enable the group policy setting, Always use local ADM files for Group Policy Editor. This setting, outlined here, will force GPMC to use ADMs from %systemroot%\inf instead of storing and reading the ADMs from sysvol.

Question:

What happens if I don’t have ADMX templates in my central store or local policy definitions and I remove the ADMs? Is that a resume generating event?

Answer:

In short, you’ll probably irritate your Group Policy administrators. Removing the ADM template, and not having the ADMX present, means that when attempting to manage a GPO the administrator will not be able to modify, or view, any of the GPO settings specific to that ADM.

Here’s a quick illustration. Assume that you’ve created a GPO which uses an Office12 ADM template (office12.adm). If you view the details on that GPO in GPMC, you’ll see a “Unique ID” or GUID. When you navigate out to SYSVOL on one of your DCs, and go to the Policies folder, you should see a whole bunch of GUIDs. Find the one that matches your GPO GUID and navigate to it. Then open the ADM folder. In the case of my GPO, I see this:

I delete it and let the change replicate around, then edit the policy in GPMC.

When I open the GPO to view the Office 2007 settings, I can’t find them.

What I would have seen before I deleted the template is this:

Fixing it is easy. Find the missing template (because of course you made a backup), right click on Administrative Templates in the GPO and click Add/Remove Templates. In the Add/Remove Templates dialog, click Add… and navigate to the missing ADM. Click Close. If you go back out to SYSVOL and take a look at that Adm folder on the PDCe, you’ll see the template is there and all of the group policy admins have stopped yelling at you.

So don’t worry about that resume, unless you’re looking for a job.

Conclusion:

Now you can get out there, convert your ADMs to ADMX, clean up sysvol, and save yourself a ton of disk space.

Also, here’s a link list to the Office ADMX templates because Mark insisted I include it.

Office 2010 - http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=18968
Office 2007 - http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=22666
Office 2003 - Just kidding. You’ll need to get the ADMs and convert to ADMX. Or consider an upgrade.

- Tom Moser

So….. You THINK you Removed DNS from Your Server

dougga1 — Fri, 09 Mar 2012 20:36:04 GMT

Hi, my name is Doug Gabbard and also a Premier Field Engineer (PFE) with Microsoft for about the past 8 years. I hope to add to the Ask PFE Platform blog with the others already here. I get the opportunity to see challenges our customers face and how they resolve them. I hope to share what I learn that is technically interesting.

This blog post is a lesson on DNS storage and behavior. Read on to learn more.

Joining in a conversation……

“….I used server manager to remove the DNS role.” Or

“….I uninstalled DNS from my domain controller. “

“....This means, DNS data is now removed from my domain controller, right?”

Well, probably not.

Background on Storing DNS Data in the Active Directory Database

Let’s hit some basics first to make sure we are all on the same page. If you follow the history of Active Directory integrated DNS from Windows 2000 to 2008 R2 you will find some changes along the way. The one change I want to focus on here is “Where in the database” is the DNS stored and how do you remove DNS.

Those of you that have met me during ADRAPs have found that I encourage understanding where things are located in the database. Knowing this helps when it comes to troubleshooting or understanding this post. So let’s start with a diagram of the database and where DNS can be stored. Fellow PFE wrote a great blog on DNS zones, which summarizes where DNS data is stored in Active Directory. Let’s borrow his diagrams. You’ll note that DNS data can be stored in the non-shaded partitions.

Let’s map these partitions back to the configuration options in the DNS Management console. To see this: open the DNS Management console; connect to a DNS server; Left click on your DNS zone (forward or reverse zones), right click on that zone and choose properties. In the properties dialog box select Change (next to the “Replication:…)” to display the radio buttons showing the options for replication scope. The four choices of radio buttons should now make more sense to you:

1. To All DNS servers running on domain controllers in the forest: contoso.com:

This is the forestDNSzones application partition. Meaning any DC in the forest with DNS installed will participate in the replication of the DNS information for that zone.

2. To all DNS servers running on domain controllers in this domain: contoso.com

This is the DomainDNSzones application partition. Meaning any DC in the this domain with DNS installed will participate in the replication of the DNS information for that zone.

3. To all domain controllers in this domain (for Windows 2000 compatibility): contoso.com.

Notice the subtle change in wording on this one. There is no mention of DNS. Just that it is replicated to all domain controllers in the domain. Meaning, even if you do not install DNS on all DCs, ALL DCs in that domain will participate in replication of the zone.

4. To all domain controllers in the scope of this directory partition:

This is the custom partition if you are using the custom partition. Only those domain controllers with DNS that YOU choose will participate in the replication of the zone. (Read below for examples why you would want a custom partition). This option is grayed out if there are no custom DNS application partitions.

So, “Class for DNS storage for AD integrated zones 101” is dismissed. On to the topic of the title:

“So you THINK you removed DNS from Your Server”

Let’s say I uninstall DNS from a DC: what happens?

Scenario 1: DNS data stored in the Domain Partition:

Not much other than the DNS service is uninstalled. You can still see all the DNS records in the AD database using your favorite LDAP browser on port 389. Be careful, though. If you delete data, the deletions will replicate and disappear from “real” DNS servers.

Scenario 2: DNS Data stored in Application Partitions (ForestDNSzones, DomainDNSzones, and Custom DNS application partitions):

You would think – maybe – that the records would no longer reside on a DC after removing DNS from that DC. Sorry, if you thought that. They do exist and are replicating just fine as before. It turns out the DC still replicates the application partition; it just no longer exposes the data through the DNS service.

There is a little cleanup if you decide to completely remove a domain controller from the replication of the DNS information. You can leave the remnants of the DNS if you like because nothing is broken. However, there is risk from accidental deletion of records or the zone if an engineer discovers the partition and does not understand why it is there and attempts clean-up. Also, if the intent was to not expose the DNS records on a domain controller or minimize its replication footprint, you have additional steps after removing a DNS Role. I will start with the custom application partition first because it is easier and the assumption is that you want to Remove the DNS role from the domain controller.

Forcing a DNS Application partition removal, after the removal of DNS:

1. Uninstall DNS/Remove the DNS role

2. From a command prompt run:

Dnscmd <DCNAME> /unenlistdirectorypartition <foo.com>

Where <DCNAME> is the name of the DC and <foo.com> is the name of the application partition.

Forcing the removal of the ForestDNSZones or Domain DNSzones, after the removal of DNS:

1. Uninstall DNS/Remove the DNS role

2. Use Ntdsutil to remove the partition to remove it from the replication group.

a. Remove NC replica dc=forestDNSzones,dc=contoso,dc=com dc2.contoso.com

(Note: I am purposefully not giving you all the syntax here, because I want you to practice this in a lab before you try this in production. Also, there is a Delete NC command. Do NOT use this unless you want to delete the partition globally)

A Final Note: “Why would I use a custom application partition for DNS Zones?”

Although there are many reasons, here are some that I have observed:

1. 100% control of exactly which domain controllers participate in replication of the DNS information.

2. Custom Application partitions can replicate to domain controllers across domains (in the same forest).

3. Easier to smoothly retire a DNS server

I was on site when a customer using custom application partitions gave me a great example. In most large environments, when you want to retire a server of any role there is always a concern of: Is anyone still using that server or service? Think of the impact of removing DNS if an application is depending on that server for DNS. With a custom application partition, we can “unenlist” from the replication (leaving the DNS service running – and if that was the only zone it becomes a caching DNS server). Then forward requests from this caching only DNS server to a DNS server that still hosts the zone. Finally, with monitoring, discover who is still using the DNS server you wish to retire.

4. If you have a DNS zone that is only needed in a few locations.
If you have many domain controllers in your environment and you need the zone in just a couple of locations, this is a great option so that not all DNS servers need to replicate the data

This was interesting to me and I hop you learned some new things about DNS.

Doug Gabbard, Senior PFE

How to Track the Who, What, When and Where of Active Directory Attribute Changes – Part I (The Case of the Mysteriously Modified UPN)

Ray Zabilla [MSFT] — Mon, 05 Mar 2012 13:45:00 GMT

Hello our names are Ray Zabilla and Rick Bergman. Ray and I are Dedicated Premier Field Engineers with Microsoft and work with the same customer. We were both away from our customer during the same week attending training, when we had an issue pop up that needed our help. We were pulled into a situation where we needed to help quickly find a solution to identify what or who was changing the Active Directory Universal Principal Name, UPN, value for our customer Contoso.

The blog post for this interesting issue is going to be discussed with the solution and details in a two part blog post. Part I will include the issue definition and approach to solving the problem and in Part II we will share the details and lessons learned.

The story you’re about to read is true. Only the names have been changed to protect the innocent.

Several months ago Contoso began a Migration to Office 365 and the design requirements required the use of the Active Directory “User Principal Name” attribute for authenticating to Office 365 with ADFS. Contoso design requires that the Active Directory UPN must match the Primary SMTP Address. Initially the UPN for the Contoso user accounts did not match the Primary SMTP address, and prior to the start of the migration they ran a script to update the UPN for all users to match the Primary SMTP address.

Shortly after the first pilot users’ mailboxes were migrated to Office 365, the pilot users’ UPN value began mysteriously reverting back to the original value. This change prevented any of the migrated users from logging into their mailbox in the cloud with their UPN and accessing their mail. We spent some time trying to identify the source of the changes but we weren’t successful. This included pulling reports from their enterprise audit collection system which collected the security logs from each DC. In the interest of continuing the pilot, the decision was made to just run a script to correct the UPNs as a scheduled job.

Now they wanted to start aggressively migrating mailboxes to Office 365, but put the project on hold because they couldn’t find what was changing the UPN value.

The Challenge

Some unknown process, running on some unknown computer, at some unknown time was changing the UPN on the Active Directory user accounts. Let’s stop and really think about the challenge statement.

Question: Where can a user object have its UPN valued changed?

Answer: On any writeable domain controller.

Question: How domain controllers are in Contoso’s environment and how many are writable?

Answer: There are more than 60 domain controllers in the environment all being writeable.

Question: What versions are the domain controller OS and what is their Forest Functional Level and Domain Functional Level?

Answer: The domain controllers are Windows Server 2003 X64 SP2, which means the FFL and DFL cannot be above 2003.

Question: How come you couldn’t get information out of the enterprise security log collection system?

Answer: We asked the team which ran the enterprise security log collection system to provide us a report of all the users who had their UPN values changed and we never received a report that contained a proper Event ID 642.

What do we do now? What is the approach for finding who is changing the UPN values?

We are going share the approach on what we did to find the mystery machine/process/account that is changing the UPN value in AD. While we needed to be aware of the all the technical considerations, we also needed to take into account the “political considerations” which we refer to as layers 8 & 9 of the OSI model.

Current Status and Next Steps

Top priority, “political consideration,” was to keep the pilot moving forward and keep the pilot users working as expected. The initial approach used by the IT group at Contoso was to try to modify the ACL on the UPN attribute to limit the accounts that could modify the attribute. This approached worked to some extent but proved problematic for a number of reasons, most importantly because it did not prevent the attribute from being modified. The IT group at Contoso continued to investigate the source of the UPN change and had focused on their identity provisioning system as the likely culprit, but they were unable to correlate any of the UPN changes to any events in the provisioning system.

The Approach – Just the facts Ma’am

The Quandary

At this point, Ray and I needed to come up with a better plan, because the approach we recommended seemed like a relatively straight forward approach which should theoretically identify the source of the change is relatively short order. However in reality this was not the case. Contoso has over 60 domain controllers in multiple sites worldwide. Further, Contoso’s security department required significant auditing which, given the size of the security logs, only provided about 15 minutes of data before the log file wrapped. Contoso also had a tool to archive the log files and while it did discover and few isolated UPN change events and the associated accounts making the change, they were unable to provide enough detail to identify the source of the mysterious UPN changes. So, how to identify an unknown process, using an unknown account, making a change to a UPN attribute, on any one of over 60 domain controllers at some unknown time?!

NOTE: On Windows 2003 Domain Controller, the Event ID 642 in the security event log indicates the new value of the attribute and it does not show failures. By default on a Windows 2008/R2 Domain Controller in the security event log, the event numbers have changed to Event ID 4738. For more information on Window Server 2008/R2 User Account Management event IDs, go to TechNet. http://technet.microsoft.com/en-us/library/dd772693(WS.10).aspx

Below is a sample from Windows Server 2003 R2

NOTE: All the following examples are from our lab for Office 365 testing.

Figure 1 – Windows Server 2003 Event ID 642

Figure 2 – Event ID 642 in Text form

The tool of Choice – REPADMIN

One of the topics during Ray’s Active Directory training class that week was on Active Directory replication and that provided the inspiration that led Ray to call upon the AD tool of choice, REPADMIN. Without getting into a lot of detail about how Active Directory replication works, we will save that for another blog post, at a high level changes in Active Directory occur at the attribute level on an originating Domain Controller identified with a Unique Sequence Number (USN), corresponding version number and a date/time stamp. In order to keep an attribute change in Active Directory from replicating around in an endless loop the originating USN from the originating Domain Controller for the attribute is stored in the objects’ metadata.

The light bulb went on! After a little creative thinking and with an understanding of the Active Directory replication process it occurred to me the same attributes maintained by Active Directory to manage replication would provide us the answer to when the attribute was changed in the entire AD Forest and give us the originating domain controller for the change. Yup, REPADMIN was tool to do it.

Rick’s Comment – After Ray suggested using Repadmin /showobjmetatdata, I started drinking V-8 juice, because why didn’t I think of that? J

REPADMIN is one of the primary and most helpful tools for troubleshooting Active Directory replication. REPADMIN has quite a few options to display information about Active Directory replication status , some of which most of you are probably pretty familiar with such as “/showrepl” or “/replsummary” . However, for this problem we going to use one of the options you may not be so familiar with and that is the “/showobjmetadata” option.

According the help displayed from the command line the “/showobjmeta” option “Displays the replication metadata for a specified object stored in Active Directory, such as attribute ID, version number, originating and local Update Sequence Number (USN), and originating server's GUID and Date and Time stamp.” See the full help listing below in Figure 3.

Figure 3 - REPADMIN /showobjmeta help

We could use “REPADMIN /showobjmeta” to display the metadata for a user object where UPN had been changed. This would identify the originating domain controller as well as the date/time of the change. You can see from the output of the REPADMIN /showobjmeta in the figure below, you supply the DN of the AD user and the tool provides the information in the metadata. In this example, the UPN for user “test5455” was changed on DC02NA at 14:27:31 on Feb 3, 2012. Success!

Figure 4 - REPADMIN /showobjmeta output

The Approach

First we needed more time to be able to get/pull the Event ID 642 from the DC where the change was made. Since all the domain controllers in this domain were running Windows Server2003 x64, the first thing the Contoso IT team did was increase the Security Event log size to 2GB. We had done some research and found there had been a few incidents reported on Server 2003 x64 domain controllers when the log files were set to the maximum 4GB so we recommend Contoso set the log file size to 2GB which should give us enough data to capture the 642 event but be well below the max size. After increasing the log file size some quick analysis found they were now had 3.5 – 4 hours before the Security event log wrapped

With the information provided by the REPADMIN /showobjmeta we would only have to search the Security log on the domain controller where the change was made to find the Event ID 642 for the object that had changed. We would also know the precise time when the changed occurred in AD so our search would be specific and limited. Once we found the 642 event in the appropriate Security log we would know the AD account that made the change and we would now have identified 4 of the 5 key variables (who, where, when, what), which would hopefully provide enough information to lead us to the process making the change.

In the example 642 event text listed below, you can see the change was made to test5455 on DC02NA which we knew from the metadata. But now we know the other key piece of missing information, the AD account initiating the change, which in this case is Administrator.

Figure 5 - REPADMIN /showobjmeta output

Armed with this knowledge, we created a couple of PowerShell scripts to identify user accounts where the UPN that had been set to an incorrect value. The scripts we created for finding the Event Id 642, will further discussed in the second post.

The Contoso IT team was working in parallel to our efforts by tracking down all of the applications, processes and scripts that were making changes to Active Directory. This was a good exercise because based on the research the Contoso IT team had already done, once they knew the AD account making the change, they were able to identify the offending process in about 15 minutes and get the migration back on track.

The story you have read is true. Only the names have been changed to protect the innocent.

The culprit was found using the tools and solid data gathering approach. Who was the culprit? It was a script that someone had written over 7 years ago to keep the UPN in the same format as sAMAccountName@contoso.com. The script and the server the script ran on had been forgotten about when there was staff turnover. In fact, the server went through hardware upgrades and had been P2V’d, moved from a hardware server to a virtualized server.

There are a number of lessons learned in this post.

1. The tools and approach that can be used to help you discover what is making the change to the UPN values or other AD attributes.

2. Keeping track of what is accessing and making updates to your Active Directory. There should be some tool in which items are tracked that makes updates to Active Directory. The tool can be a SharePoint site, a component of your Enterprise Management tool or something as simple as the “Enterprise Spreadsheet”. Tracking this information would help drive good operational discipline in the organization.

3. The last and really important lesson is using a data driven trouble shooting approach. The key to be being able to solve difficult complex problems such as this one is using a systematic, careful, detailed, data driven troubleshooting methodology. Too many times we see customers struggling to solve problems by unstructured methods and processes from guessing to trying things without any logical process or results tracking which sometimes can lead to more problems.

The second part of the post will include details on the scripts we wrote, performance issues we ran into, and methodology we used working through the issues.

Part II - How to Track the Who, What, When and Where of Active Directory Attribute Changes

We hope you found this blog post helpful and is something you can use in your environment.

Ray Zabilla and Rick Bergman, Microsoft PFE

Taming Perfmon: Data Collector Sets

Jake Mowrer [MSFT] — Mon, 27 Feb 2012 14:17:00 GMT

There I was, the computer clock showed 1AM, I was trying to figure out how to fulfill what seemed to be a simple customer request.

“How do I get a daily report from 12AM to 11:59PM every day on how my server is performing?”

Now you may be thinking “Jake, why the heck were you up at 1AM working on this?!?” It’s because 48 hours prior I was on a customer call and I didn't have a quick answer for this seemingly straightforward request. I also got the feeling the customer was not so happy with me not having an immediate answer for him, especially since I’m their “Platforms” go to guy! This bothered me, so I decided not to sleep until I figured it out, which I did (eventually). To start things off, let’s take a look at Performance Monitor (AKA Perfmon) on a Windows Server 2008 R2 machine.

OK. So what. It looks the same as it did with maybe some different wording and layout. Let’s dig in a bit more and look at Data Collector Sets, which is what we use to capture historic data.

Here you can see that I do not have any User Defined Data Collector Sets. I’m going to create a set using the built in System Performance template and accept the default all the way through the wizard. If you have questions about how to create this set, check this resource out:

Create a Data Collector Set Manually
http://technet.microsoft.com/en-us/library/cc766404.aspx

Now I have my data collector called “This is easy”. At this point, I have not set any schedules or tweaked any advanced settings so this data collector will only collect data when I start it manually. Let’s checkout the properties of my new collector, specifically the Schedule tab since I
want this collector to run every day from 12AM to 11:59PM.

OK, easy enough, looks like I have my schedule configured correctly to start every day at 12AM. Note I did not set an Expiration Date as I don’t want the collector to quit anytime soon. Next, we’ll look at the Stop Condition tab.

Here’s where it gets a bit tricky. Important Note - Reports are compiled oncethe collector set stops. This means I can’t let this collector just simply run forever because I will never get a report. I need it to stop at 11:59PM so it can create a cool report that looks like this:

Have you figured out what’s missing yet? Let me give you a hint by showing you what the Schedule tab looks like on a Counter Log in Windows Server 2003:

Yea, there’s no “Stop At” setting so I can’t set a stop time! So after some pondering, I decide to roll with 1439 minutes for the Overall duration setting.

How did I come up with that? The customer wanted a report from 12AM to 11:59PM which is 23 hours and 59 minutes total run time. (23*60) + 59 = 1439 minutes (sorry for the grade school math flashback there).

So I sent this over to the customer and he seems happy. The next day he tells me it didn’t work. Impossible! Seeing is believing, so I looked at the reports on his system and sure enough, the job ended at something like 11AM, missed the 12AM schedule, and was not going to start again until 12AM that night so he will miss 13 hours of reports. No good.

How could this be? Pondering some more, I figured out that he had manually started the collector right after creating it around 11AM the day prior. Since it will run for 1439 minutes from start time, it ran until 11AM and the job at 12AM didn't kick off since it was already running. Meh, there must be a better way! I went back to the drawing board.

So back to where this post started, 1AM on Tuesday, I had a very simple idea (perhaps via inception):

What underlying mechanism controls the schedule for these data collectors to start and stop?

Task Scheduler!

Opening Task Scheduler, we’ll navigate to Task Scheduler Library > Microsoft > Windows > PLA. This is where you will find your data collector set schedules.

Getting the properties on my collector set, on the Triggers tab, if I edit the Trigger you can see even this screen does not have a way to end at a certain time. At first I thought I could select the Repeat Task option but every hour is the maximum configurable time.

Exploring a bit more I went into the Settings tab, one setting caught my eye:

If the task is already running, then the following rule applies: Stop the existing instance

Ah ha! What if I didn't configure a Stop Condition and used this setting to stop the collector as the next collector instance starts? Guess what, it worked. Now my collector starts at 12AM and finishes at 12AM when the next instance starts up. The extra value on this is that they got 1 additional minute of reporting, 11:59PM to 12AM.

Victory? Not quite. Once I tested this I noticed that the report generation broke. Great, yet another area of Perfmon that has always just worked so how am I going to fix this one? Switching back to Perfmon, there are some additional data management settings that are not so obvious. Right-clicking on the data collector revealed an option I had not touched before called Data Manager.

Here you can control the disk space consumption of the folder that houses data from the data collector in question.

If you browse out to where Perfmon puts data collector files, you will see various files as seen below. The ETL is the Kernel ETW tracing that is enabled in my data collector. The .BLG file is the performance counter data, and the report files are, well, the report J. The rules.log is fun to look at, but I've never messed with it. It corresponds with the Rules tab back on the Data Manager window.

Not very interesting on the surface yet when I went out to view the data for my data collector that was stopped by the new instance starting, I noticed none of this data was there, just a .CAB file. Why? Let’s go back and look at the Actions tab on the Data Manager.

If you look at the Actions tab here, you’ll notice that there is a Folder action at the top that basically says after 1 Day, doesn't matter what size it is, create a CAB and delete the data. So essentially when my data collector stopped, it was at one day so the data was being deleted and placed in a CAB before the tracerpt.exe process could parse the ETL and BLG files and generate the report. To fix this I edited the Folder action and changed the Age to 2 Days and OK’d all the way out.

Did it work after that? Yes. Was I happy with the solution? No, it shouldn’t be this hard for something so simple. Is this the only way to do this? It’s the only way I could think of. Why did you all take out the Stop condition as a definitive Time? I don’t know, but it sure made this a lot more interesting to figure out!

Until next time, thanks for reading, I am going to sleep now!

2008 R2 Active Directory: Schema Updates, LCS, OCS, and LYNC

Greg Jaworski [MSFT] — Mon, 20 Feb 2012 14:00:00 GMT

I recently ran across an unexpected issue regarding LCS 2005, OCS 2007, LYNC 2010 and AD 2008 R2 schema updates. Unfortunately for my client we discovered the issue too late, but hopefully this article will provide you with the necessary understanding and information for you to be forewarned and therefore forearmed.

Background – KB982020

While the issue is not directly related to this KB article, you should know that if you have already extended your schema for LCS 2005, OCS 2007 or OCS 2007 R2 and you subsequently extend your schema for AD 2008 R2, there is a known issue. See http://support.microsoft.com/kb/982020 for all the details. In short, you’ll have to re-apply the LCS/OCS schema extensions after the 2008 R2 schema extensions.

Customer Situation

Since we were updating the Schema for 2008 R2, we needed to do a quite a bit of planning, coordination and change control preparation. Additionally, the customer decided to install

Editorial Comment: Many administrators I run across these days don’t think twice about schema updates. I think this may be a testament to the reliability and stability Microsoft has developed in AD. While it is rare and very unlikely you will have an issue just remember the bottom line is if something does go wrong with the schema updates and it creates an issue with the Active Directory database, the only real roll-back option is a complete forest recovery. (Sounds like another blog post).

the Exchange 2010 and LYNC 2010 schema updates during the same change control window to expedite an upcoming project.

We installed the 2008 R2 Schema Updates (adrep /forestprep; adprep /domainprep adprep /RODCprep) without any issue. We followed our plan and of course did all our validation checking, etc. So far, so good, so on to the Exchange schema extensions. Again no issues with Exchange so onto the LYNC 2010 schema updates.

This was the “fatal flaw” in our plan although we had no idea at time, which is in fact the reason for this blog post. Our customer as you may remember was running LCS 2005 and they had intended to go directly to LYNC 2010. This made perfect sense at the time so no need to do anything with OCS, just install the LYNC 2010 Schema updates. So we did. That was a decision as they say, we would live to regret. Well O.K., maybe not quite that dramatic, but as you will find out in a minute this innocent, well-meaning decision is causing a lot of agonizing and anxiety for the customer at this very moment.

Obligatory Disclaimer

First, let me offer a bit of a disclaimer in that there has been a lot of investigation and follow on research of the following issue that I have not been involved with so the situation may have changed or new information may be available. So as with any information, use this blog as a guide and do you own research and come to you own conclusions.

So what’s the Problem Anyway?

Several months later the customer was planning their LYNC 2010 upgrade and discovered there is no coexistence strategy between LCS 2005 and LYNC 2010. Without getting into a lot of details, the migration to LYNC 2010 would require an extended outage which was not acceptable to the customers’ business model.

The word around was to do an interim update to OCS 2007 to keep coexistence with LCS 2005 and then once migrated to OCS 2007 they would have coexistence between OCS 2007 and LYNC 2010. O.K., so it sounds like some extra work and time but not a show stopper.

This is the Problem.

As it turns out it is a problem. The problem is that it is not supported to install the OCS 2007 Schema updates once you’ve installed the LYNC 2010 schema updates. This is causing a major issue for this customer while they regroup and try to come up with a workable plan. I know they had been working with some of the really smart people at Microsoft and they may already have come up with a resolution at this time. If I find out the resolution I’ll update the post however there is a relatively simple preventative solution.

The Recommended Approach.

As a result, the recommendation would be if you have LCS and are planning to migrate to LYNC 2010, install the OCS schema updates regardless, before you install LYNC 2010. It should not cause any issues and it provides the opportunity to install OCS if for some reason this became necessary.

I expect this is not a very likely or probable situation for most of you but if it helps anyone avoid the situation than it was worth the time to write it up.

Ray Zabilla

Disaster Recovery

Michael Hildebrand - MSFT — Mon, 13 Feb 2012 16:55:00 GMT

Have you ever tried to restore a server? What about a Production server? How about in the middle of the night? It never goes smoothly. Your cellphone never stops ringing. Often, the only thing that gets the server recovered is your own ingenuity and rock-star efforts. Let’s spend some cycles and try to get ahead of this.

As PFEs, one of our major roles and responsibilities is to help our customers realize “the gaps” and assist them in addressing them proactively. After an eye-opening conference call discussing recovery plans, or lack thereof, I felt even more compelled to create a post with some DR considerations. Hopefully, this will stir some thoughts and discussions (and ACTIONS!) around the matter of recovery.

Recovery can be defined as (among other things):

To return to health
To return to normal state
To gain back something which was lost

In our World of IT, we could be doing any or all of these actions during what we often refer to as “Disaster Recovery,” or “DR.”

It could be from a natural or man-made disaster or other large-scale event.

Fire/flood/storm
Terrorism or war
Facility malfunction

It could be a rogue admin or disgruntled employee. Often, it was due to an IT Pro making an innocent mistake – either small or large-scale.

Even with the confirmation prompts of most actions within Windows, people are still, well, ‘human.’
Anyone been on the recovery end of a script running with Admin-level credentials but not behaving as expected? Whoa daddy. That’s likely the time when you discover that backups have been failing. Since the spring. Of 2008.

Consider the statement: We do full backups of the ‘whole’ server, so in order to recover after an outage, we would simply do a full recovery of the box and be done.

BUT

Many times, a ‘full’ server backup doesn’t get key files – such as those files that are in use. DBs, transaction logs, application exe files, etc, are often not backed up during backup jobs via default settings or without special agents. We usually don’t realize this until we’re in dire straits. Or, perhaps, there is a Scheduled Task that is supposed to pause/quiesce the app/DB so the backup can get a copy of the proper flat file(s)? However, the Task isn’t being monitored and it hasn’t run for 9 months (since the svc acct got locked out and we’re not monitoring it with SCOM). Also, since that last backup 9 months ago, the app owner has upgraded the app two versions.

Consider the statement: We test recovery of our systems at the annual/recurring DR exercise/effort/mtg (you do have one of those, don’t you?)

BUT

However, as a “year in the life” passes for a system or server, it gets patched, service packed, drivers updated, settings changed (or drift), etc. Sometimes, the steps that enabled you to recover the system during the last DR exercise no longer work and the recovery suffers an epic failure.

BE PREPARED – as much as you can. Like many things, DR is always a work in progress and always changing as our systems evolve, get patched, updated or otherwise changed. Be vigilant! Be disciplined! Add Recovery to your normal work routine so it doesn't catch you off-guard. Consider recovery before a system is even deployed. Make sure it is part of the design. Test the recovery design prior to deployment and again at regular intervals.

One tip is to add recovery testing to your own day-to-day work items.

Consider using Outlook and Recurring Appointments with Reminders
- Monthly – test recovery of a test OU and its test contents
- Quarterly – test recovery of a complete test server and it’s test applications/services
  - Isolated or other offline environment
  - Bi-annually – test recovery of an entire Domain Controller (a test DC or other non-production impacting)
  - Annually – perform a more formal shared DR exercise
The Outlook Calendar method helps by blocking out Calendar time for this
You can also Invite others to these Outlook events
The Outlook Calendar method makes it all just a bit more official and formal

Now for a few DR pointers. Much of this is obvious and self-evident. It is painful, though, how often we neglect or forget the obvious.

Document. Document. DOCUMENT!

Have two or more locations for Documentation such that a disaster to the system(s) that store your Docs doesn’t render you completely scrambling.
Don’t underestimate the value of a hard-copy, even if it is a bit dated, it’s better than nothing
Make sure there are application-specific docs that get tested/reviewed
- Often, the app was installed 6 years ago and no one on the current team even knows where the install bits are stored. The woman who knew the app left the company and took to a life of wandering the forests; she hasn’t been heard from since the spring of 2004.
- Application pre-requisites/details
  - DotNet versions?
  - Service accounts? (local or Domain-based)
  - Specific or non-standard NTFS or registry permissions?
  - Non-standard User Rights or other local Policies or Group Policy settings
Track application service releases/updates/etc – so you’re able to get back to where you are via clean install + updates, if needed
Have a selection of these accessible:
- CD/DVD blanks
- USB thumb and bigger drives
- 3 ½” floppy disks – if you need one of these, they can be very hard to find these days
Some folks have mature “Configuration Management Database” systems (CMDB) to track server/application personality Information and settings

SCCM can help automate a great deal of this personality information via Inventory jobs
CMDBs are extremely helpful but many times, they are not running on a ‘highly available’ system and during a DR (exercise or real) might not be available. Examine your environment to see if you’ve painted yourself into a corner like this

Again, don’t be afraid of hard-copy – just be sure to secure it. There’s nothing better than a big ol’ DR binder when you need it.
Consider storing the following info as a good start

HDD sizes (especially C:\)
C:\WINDOWS or C:\WINNT?
Service Pack levels
Standard/Enterprise/Datacenter/R2?
x86 vs x64?
Windows Firewall – custom ports/settings
Custom or non-standard Local Policies, reg entries, GPOs
Local Admin pwd (hopefully as part of a process that is managed/on-going)
TCP/IP info

Static routes
NIC settings and info
- Don’t forget NIC speed/duplex
Hardware config/info
- Driver versions
- BIOS versions and custom settings (i.e. virtualization, power mgmt, etc)
- Storage/array configs/logical drive layout

For AD-specific recovery, consider the following as a start:

GPOs – are you backing up your GPOs?

Consider Powershell and/or GPMC scripts

OU information along with GPO link information

Note, GPMC backups do not backup the GPO links (they’re an aspect of the OU, not the GPO itself) but the link information is recorded in the GPO report within the backu
OU permissions/delegations

Consider Powershell and/or a DSACLs script

Directory Services Restore Mode (DSRM) Password

This is set on EACH DC independently and is very often poorly managed (if at all)
However, this can now be sync’d to a Domain account

http://support.microsoft.com/kb/961320

Current, accurate location of servers

In a large datacenter, simply finding the right physical server can be a maddening and high-calorie-burn endeavo
Virtual servers have their own set of ‘hide and seek’ issue

Tested recovery/boot CDs for pwd reset, dead server revival/data-harvesting/etc

Many times, the storage drivers on these need to be updated or they won’t ‘see’ the drives and can’t find the Windows installation
The Microsoft DaRT tool can help in this regard

http://technet.microsoft.com/library/ee532075.aspx

Hopefully, the information here reminds you of DR, gets you thinking about DR, brings up an idea or two about DR, or even stirs you to setup some Outlook appointments.

Now, take action and be at least a little better prepared.

Cheers!

Now, seriously… are you ready to become a PFE?

Paolo Matarazzo — Fri, 10 Feb 2012 18:00:00 GMT

After Greg gave you some great tips about how to join Microsoft, and Mark explained what it's like being a PFE, it's time to join us!

If you are interested (and I bet you are), Microsoft Services is hiring in Charlotte. We will have an Open House on Saturday, Feb 25th 2012 so hurry up and register!

We are not only hiring for Premier Field Engineer roles, but also for Technical Account Manager, Escalation Engineer, Senior Support Engineer, Software Development Engineer, Program Manager and Service Delivery Specialist roles.

You will have the opportunity to meet Microsoft engineers, ask more details about the different positions and find out about the prerequisites.

Bring your resume, we will be glad to review it and pre-screen you for a technical interview.

For more information and to complete the registration, visit the following page.

Am I Seeing Double? The case of "Multiple copies of the same DNS zone"

dgreg - MSFT — Sun, 05 Feb 2012 22:30:00 GMT

Introduction

With the introduction of Windows 2003 and the new DNS application partitions, I have helped numerous customers resolve the issue of having multiple copies of the same DNS zone. So, today we're going to cover the following:

1.) What exactly does this mean?

2.) What are the symptoms?

3.) How does this scenario occur?

4.) How to resolve the problem?

What Does this Mean?

Quick history lesson...With the introduction of Windows 2003, Microsoft created two new DNS-related application partitions. Let's quickly discuss why Microsoft did this. To do this, we'll have to take a look at how Windows 2000 implemented DNS. With Windows 2000, when you created a new primary Active Directory integrated DNS zone, it automatically got stored in the domain partition. This is the same partition that contains your users and groups. You could locate these DNS zones by navigating to Domain Partition>System>MicrosoftDNS. All domain controllers in the domain receive this partition regardless of whether they're actually running DNS. Additionally, all GC's in the forest receive a read-only, partial-attribute copy of this partition. Based on this explanation, if a DNS zone is stored in the domain partition, it will replicate to a domain controller that NOT running DNS. And when it replicates to all the GC's in the forest, it will also replicate to a domain controller that is NOT running DNS. Ultimately, we were replicating DNS information to a DC that can't even use it..Now, doesn't that sound really inefficient. Microsoft went back to the drawing board and came up with this new idea of DNS application partitions that ONLY replicate to DNS servers. Brilliant!

With the new DNS-related application partitions you see above, we could now put DNS zones in these locations, which would only replicate to domain controllers that are also DNS servers. But Microsoft also kept the ability to store DNS in the domain partitions, for backwards compability purposes. DomainDNS will only replicate to DNS servers in the same domain and ForestDNS will only replicate to DNS servers in the entire forest. No more inefficient replication! The caveat here now is that DNS now has three places to store a DNS zone, which can lead to the possibility that you could have multiple copies of the same DNS zone stored in different locations:

The key to remember here though is the order of precendence: Domain Partition, then DomainDNS, and then ForestDNS. If DNS finds the same DNS zone in multiple partitions in Active Directory, the above precedence will determine which copy it uses.

What are the symptoms?

Inconsistent DNS records - The first symptom you'll probably notice on a child domain DNS server is that the DNS records are not the same as within the same DNS zone on a another domain's DNS server

Event ID's - Additionally, when this happens, you see the following Event ID on the affected DNS servers:

Event ID: 4515
Event Source: DNS
Event Type: Warning
Event Description: The zone contoso.com was previously loaded from the directory partition ForestDnsZones.contoso.com but another copy of the zone has been found in directory partition contoso.com.

How does this happen?

Scenario #1

This first scenario involves a single forest, single domain environment. Before the problem started occurring, the contoso.com DNS zone was configured to replicate to all DNS servers in the forest via the ForestDNS application partition. If you open up the DNS console and go to the properties of the DNS zone and under the replicate scope click change, the option selected was "Replicate to all DNS servers in the Active Directory forest contoso.com":

One day, the admin has to stand up a new domain controller in some remote location that has really slow WAN link. He runs DCPromo, everything seems to be working fine. He knows that without DNS, nothing will work so he opens the DNS console and notices that it doesn't have the contoso.com DNS zone yet. He gets really impatient and decides to manually create the contoso.com DNS zone hoping to give it a kick in the butt. So he creates a new DNS zone called contoso.com, and decides to replicate it in the domain partition within Active Directory by choosing "Replicate to all domain controllers in the Active Directory domain contoso.com":

He thens waits 15-30 minutes depending on the replicates times and immediately his phone is ringing that all enterprise name resolution is not working. He starts connecting to all the DNS servers in the domain and notices all the DNS records that existed previously are GONE!

What happened? In this scenario, he actually has multiple copies of contoso.com. Although their was already a copy of contoso.com in the ForestDNS application partition, due to the slow WAN link, it hadn't replicated to this DNS server yet. He got impatient and created another copy of contoso.com in the domain partition. Based on the precedence we discussed earlier, DNS started using the newly created domain partition copy rather than the one in the ForestDNS application partition. If we were to use adsiedit and connect to the following partitions, we would have found the following:

Scenario #2

This scenario is the more common one but is slightly more complex because the environment has multiple domains. Nonetheless, what we learned earlier still applies here. This environment is a single forest with six domains. Before the problem occurred, the admins decided to replicate the forest-root DNS zone, contoso.com to all DNS servers in the forest via the ForestDNS application partition.

One day, a domain admin in one of the child domains opens up the DNS console, notices that he has a copy of the contoso.com DNS zone. He also notices that it is replicating to all DNS servers in the forest. He wants a copy all to himself so he changes the scope of replication on the contoso.com DNS zone to "Replicate to all domain controllers in the Active Directory domain child.contoso.com".

Under the hood, DNS copies the contoso.com DNS zone into the child.contoso.com domain partition and then attempts to delete the copy stored in the ForestDNS application partition. Since he isn't an Enterprise Admin or Domain Admin in the forest-root domain, he doesn't have permission to delete it from the ForestDNS partition. The DNS servers in child domain now have two copies of contoso.com; one in their domain partition and another one in their ForestDNS application parition. If we were to use adsiedit and connect to the following partitions, we would have found the following:

Based on the precedence of DNS we discussed earlier, all DNS servers in this child domain will operate off of the copy in the domain partition while the other DNS servers in the rest of the AD forest will continue to operate off of the copy in the ForestDNS application partition. Over time, this misconfiguration and problem will only become magnified.

Over time, the two DNS zones will get further and further out of sync because all the DNS updates/changes that are occurring throughout the forest are being replicating to the ForestDNS copy while the domain partition copy begins to get stale. Perhaps even the scavenging process runs on the child domain partition copy of contoso.com and they get further out of sync. Eventually, the admin discovers this, don't know what to do so they manually start making records for the resources just to get name resolution working.

Resolution

Warning: Before executing on the solutions below, please confirm that the above scenario apply. If you are not sure, please contact Microsoft support. Also, due to the critical nature of DNS, please perform these resolutions during off-peak hours.

First off, if either of these scenarios occur, they will impact all the domain controllers that are DNS servers in the entire respective domain. If you suspect this is happening, connect to the folllowing partitions in adsiedit and document what you find:

dc=Domain,dc=Com

dc=DomainDNSZones,dc=domain.dc=Com

dc=ForestDNSZones,dc=ForestRootDomain,dc=Com

Next, find out how many domains exist in the forest. Use the following guidelines to resolve this issue:

Only 1 domain: If the problem just started happing, from within adsiedit, determine which DNS zone copy has the most recent, up-to-date DNS records and delete the other DNS zone. From adsiedit, right-click and delete the respective DNS zone. If the problem has been occurring for a month or longer, delete the other copies of the DNS from either dc=DomainDNSZones,dc=Domain,dc=Com or from dc=ForestDNSZones,dc=ForestRootDomain,dc=Com.

Multiple domains: In the entire forest, determine how many DNS servers are operating off the one DNS zone vs. how many are operating off of the other DNS zone. Keep the one that is being used by more DNS servers. The logic here is that the more DNS servers that are using it, the more update-to-date records it must have. For example

1 forest with 6 domains and each domain has 6 domain controllers. If the problem exists in only 1 child domain then:

5 domains x 6 DNS servers = 30 DNS servers or 83% are using the ForestDNS copy.

1 domain x 6 DNS servers = 6 DNS servers or 17% are using their Domain or DomainDNS copy.

Since 83% of the DNS servers are using the ForestDNS copy of DNS, that is the one we will keep. Consequently, on the affected child domain DNS servers, open up the DNS console, right-click the DNS zone and choose export list. This allows us to back-up any static records. Next right-click and select delete. The DNS service polls for changes to DNS zones every 3 minutes and will then find the ForestDNS copy and load it. Lastly, restart the netlogon service on the domain controllers in the affected child domain to ensure the domain controllers re-register their necessary SRV records.

Prevention

Scenario #1: The only way to avoid scenario #1 is to be patient and allow AD replication to populate the already existing contoso.com DNS zone.

Scenario #2: Based on the default permissions set within Active Directory, the only way to avoid this scenario is through education and awareness, or eliminating all child domain admins :) Joking aside, make all child domain admins aware that certain DNS zones will be replicating forest-wide and that their scope of replication must not be changed.

What’s it like being a Transactional Premier Field Engineer (PFE)

Mark Morowczynski [MSFT] — Mon, 30 Jan 2012 13:00:00 GMT

Hi, Mark Morowczynski here again, in a previous post, http://blogs.technet.com/b/askpfeplat/archive/2012/01/16/how-to-become-a-premier-field-engineer-pfe.aspx Greg Jaworski, a fellow transactional PFE, touched on how to become a PFE and defined what a transactional PFE is and a dedicated PFE. If you recall transactional PFEs don’t have specific customers they always see and dedicated PFEs tend to be assigned to 1-4 specific customers. Also our work is classified into two categories, proactive (think along the lines of health checks, workshops and chalk talks) and reactive (there is a service outage or something is not working properly). Today I’m going to give you an overview of what it’s like for a transactional PFE based on questions I routinely get from customers, friends, and family.

“Do you go to a different customer every week?”

Generally yes. We see a lot of customers and with that their environments. This provides us with great insight on a how a product is used depending on a variety of factors. The size of the company, their industry, their business requirements and the history of the environment all play a role in how a product is deployed and used today. It amazes me still how creative Microsoft customers are in designing solutions and with that comes some very “creative” issues.

“Do you ever go back to the same customer twice?”

Absolutely! Many times a PFE will build up a relationship with a customer and frequently go back. They are familiar with the environment, how the customer works and what their overall goals and challenges are. Depending on your skill set you may go back to the same customer but to a different team within that customer. I’ve personally worked with a customer’s AD team as well as their desktop team on two completely different engagements.

“Do you always have to go on site to help customers?”

Nope, PFEs often provide remote support as well but not in the same way that CTS does. Whereas CTS tends be in a break-fix scenario we tend to help in a guidance scenario. For example, “We have some general questions around raising our domain and forest levels” or “We wanted to start doing X and we wanted to bounce some ideas and make sure we are heading in the right direction.” This type of remote support tends to be 1 to a few hours in length.

“Do you always know how to fix what’s wrong?”

Of course not, though we generally have a good idea on where to start. Having a good and deep understanding of how things are supposed to work really helps troubleshooting a problem you’ve never seen before. We also have an amazing internal support network of PFEs and of course all of Microsoft who are extremely helpful if you are really stumped.

“Do you have an on-call rotation?”

For the most part, we do have times were we are dedicated to respond to any “CritSits” which are basically Severity 1 or Severity A cases. We can be called for those even when we are not designated as well.

“How much do you travel?”

We do some traveling; some of us do A LOT of travelling. This is probably one of the most common questions I get asked. “How much traveling is there really?” The answer like all Microsoft answers is, “it depends”. Where do you live? What technology are you supporting? How much do you want to travel? If you live in a fairly major city, support a widely deployed technology and do not want to travel, you generally find customers that need your help close by. However if you want to do some traveling there is plenty of opportunity for this.

“Where do you get to go?

Generally you try to stay in your region if you are US based, (West, Central and East) but timing and availability of customers and PFEs can have you traveling all over the US. Personally I’ve gone as far east as Miami and as far west as Alaska and lots of places in between so this is not a hard and fast rule. International travel is a rare exception. We will try to get a non US based PFE to answer how the travel is for them shortly. Update: Carlos Mayol Berral (who is a PFE out of Spain and will write some blog posts for us in the future) lets me know that being based out of Europe generally means you cover the entire region.

“What are the perks of traveling?”

Perks include, airport delays, dealing with people who travel once a year, lost luggage and bed bugs . Now for the good stuff. We do also get to travel to some pretty amazing places as well. Hawaii, and Caribbean islands. Personally I enjoy going to different major cities and seeing the sights if I have time. We do get to use our airline and hotel points as we personally see fit. So that’s a plus as well.

“How do you keep track of all this it seems like there is a lot going on?”

Planning and controlling where you are going and what you need to do is one of the most important things as a transactional PFE. We beat this into all new hires by saying “control your calendar don’t let your calendar control you”. Knowing where you need to be, what you need to do for past and upcoming customers can get overwhelming fast.

“Do you get training?”

Yes there is plenty of classroom and on your own study. We actually require that PFEs spend 3 weeks of official Microsoft training per year to keep their skills sharp.

“Are you married to your job or are you allowed to have a personal life”

Work life balance is a huge and constant topic among PFEs in general. You can have a personal life you just have to plan for it, seriously. Controlling your calendar is the number one way to do this. If you don’t allow any free time and want to go from customer to customer the job will most certainly allow you to do this. However we do not recommend it as that is how you’ll get burned out.

That is pretty much it in a nutshell. We spend a large amount of our time on site face to face with customers. There is generally some pre and post work required for each customer as each engagement is a bit different. If there are any questions you’d like to ask please put them in the comments and I’ll do my best to answer.

Mark Morowczynski

Keep your friends close, but your cluster node configuration closer. (Comparing differences across failover cluster nodes).

dsymalla — Mon, 23 Jan 2012 13:31:00 GMT

In my role, I often get to review failover clusters. Typically we look for configuration problems or deviations from best practices. One of the most challenging aspects of managing and maintaining a failover cluster is keeping the nodes identically configured. This means identical drivers, driver versions, service packs, hotfixes, services and applications across all cluster nodes.

Failover clusters are usually deployed with each node identically configured; however, as time passes the nodes tend to diverge. Drivers get updated on one node, but not the others. Hotfixes get installed on one node, but not the others. Your challenge, should you choose to accept it, is to keep the nodes the same. What you need are some tools to audit the configuration across the nodes, to look for differences.

How to compare cluster nodes in Windows Server 2008 or Windows Server 2008 R2

In Windows Server 2008 and Windows Server 2008 R2 there are built in capabilities to compare failover cluster nodes. The tool is called “Validate a Configuration Wizard”. Simply run the validation wizard and it will (among other things) compare cluster nodes for differences in BIOS, drivers and hotfixes. (Note that the cluster validation wizard is much more powerful than simply comparing the cluster nodes. If you administer (or plan to administer) 2008/R2 failover clusters you should get to know the validation wizard).

Validate a New or Existing Failover Cluster: http://technet.microsoft.com/en-us/library/cc772450.aspx

What do you do, though, if you want to compare Windows Server 2003 cluster nodes, or systems that aren’t cluster nodes?

Comparing Differences in Other Scenarios - Enter the PowerShell cmdlet: Compare-Object

(Note: All examples are executed from a machine with PowerShell v.2.0. However, the target machines are a mix of 2008 R2 and Windows Server 2003. It is NOT required to have Powershell/.Net/WinRM on the target machines. You must simply have administrative privileges on the target machines and WMI must be working properly)

If you can extract information from two systems, PowerShell can do the comparison for you. For example, use the Get-Hotfix cmdlet to pull hotfix information from two different computers, and then use the Compare-Object cmdlet to compare them.

Start by pulling the hotfix information from a remote computer “ParentDC1”.

Get-Hotfix –computer ParentDC1

It’s Never That Easy, is it?

Notice all the “File 1” garbage? Let’s get rid of that.

Get-Hotfix –computer ParentDC1 | where{$_.HotfixID –like “KB*”}

Now, let’s hold that data in a variable so we can use it to compare to another system.

$a = Get-Hotfix –computer ParentDC1 | where{$_.HotfixID –like “KB*”}

Then, do likewise to collect information from another computer “ChildDC2”.

$b = Get-Hotfix –computer ChildDC2 | where{$_.HotfixID –like “KB*”}

Now, simply compare the two collections with Compare-Object. Note the use of the –property switch so we only compare hotfixIDs (and not hostnames, installed dates or installed by).

Compare-Object $a $b –property HotfixID

To interpret the output from Compare-Object you note the direction of the SideIndicator. When we called Compare-Object we put $a (ParentDC1) on the left and $b (ChildDC2) on the right. Thus, if the SideIndicator points to the left, the hotfix appears on ParentDC1 (and not ChildDC2). If the SideIndicator points to the right, the hotfix appears on ChildDC2 (and not ParentDC1).

What if you also want to see the hotfixes installed on each node that are the same? Include the –IncludeEqual switch

Compare-Object $a $b –property HotfixID -IncludeEqual

Note on Get-Hotfix, from http://msdn.microsoft.com/en-us/library/windows/desktop/aa394391(v=vs.85).aspx. “Starting with Windows Vista, this class returns only the updates supplied by Component Based Servicing (CBS). These updates are not listed in the registry. Updates supplied by Microsoft Windows Installer (MSI) or the Windows update site (http://update.microsoft.com) are not returned by Win32_QuickFixEngineering.” This means get-hotfix may not return all installed hotfixes when run against Windows 7 or Windows Server 2008 R2. If you want to know more about the new component-based servicing model, see the following blog: http://blogs.technet.com/b/askperf/archive/2008/04/23/understanding-component-based-servicing.aspx.

That’s pretty cool and relatively easy. How about comparing services across systems? Same logic, but use the Get-Service cmdlet to get the data.

$a = Get-Service –computer ParentDC1
$b = Get-Service –computer ChildDC2
Compare-Object $a $b

If you’d like to only consider services that are running, you can modify your commands, as follows:

$a = Get-Service –computer ParentDC1 | where {$_.status –eq “running”}
$b = Get-Service –computer ChildDC2 | where {$_.status –eq “running”}
Compare-Object $a $b

How About Something Harder – Like Driver Differences?

Now things start getting a little interesting (or not, depending on your tastes). Unfortunately, there is no Get-Driver cmdlet. So we need to go into the weeds, and get some data using WMI. Basically, we combine two WMI queries. The first one, gets a list of installed drivers on a system.

$Query = “SELECT Name, PathName FROM Win32_SystemDriver WHERE PathName IS NOT NULL”
$driverList = Get-WmiObject –Query $Query –ComputerName ParentDC1
$driverlist

The second WMI query can take a pathname and return details about the driver (including version)

$Query2 = “SELECT Name, Version, Manufacturer, LastModified FROM CIM_DataFile WHERE Name = ‘C:\\Windows\\system32\\Drivers\\afd.sys”
$DriverInfo = Get-WMIObject –Query $Query2 –Computername ParentDC1
$DriverInfo

So you’ll need to create a collection of driver information on the first system by walking through the driver list. Then create a second collection of driver information for the second system by walking through its driver list. Then you use Compare-Object to compare the two collections. You could report differences in driver names, to report drivers installed on one system, but not the other. Or you could report differences in driver versions, to report differences in driver versions across systems.

If you’ve read this far, you’ve probably figured out that you’ll need to script the solution so you put in the necessary loops and create the custom data collections. I’ll spare you the details, and let you look through the attached script (drivers.ps1). To use the script you can either specify a single hostname and the script will report on driver information for that host.

.\Drivers.ps1 ParentDC1

Or, you can include two hostnames and the script will show you a difference in drivers installed on the systems.

.\Drivers.ps1 MNTools1 MNTools2

Or, you can include two hostnames with the –versions switch to only report differences in driver versions (a driver installed on both systems, of a different version).

.\Drivers.ps1 ParentDC1 ParentDC2 -versions

Finally, you could pass the script two hostnames with the –both switch to report both driver differences and driver version differences. Note that the script will report if no differences are found.

.\Drivers.ps1 ParentDC1 ChildDC2 –both

I hope you enjoy the script. Remember all the usual caveats apply to the script, so use it at your own risk. I hope you now have some tools to keep your cluster nodes more closely configured with respect to hotfixes, services and drivers.

Doug Symalla

How to become a Premier Field Engineer (PFE)

Greg Jaworski [MSFT] — Mon, 16 Jan 2012 12:46:24 GMT

Hello my name is Greg Jaworski. I am a Transactional Premier Field Engineer with Microsoft. A very common question is what it takes to join Microsoft. What kind of skills should I have? How many years of experience do I need? A comment from one of our blog posts is what it takes to join PFE. While others have probably posted some tips it never hurts to post this information again. I will provide some of my own personal background as well as some of the things that we look for. While this does not guarantee you will get hired by Microsoft it will give you a general idea of what we are looking for.

What is Premier Field Engineering?

Before we even get into becoming a PFE we probably need to go over exactly what PFE does and some of the terminology that we use. Many of the people who read this blog may have never even heard of us. In future blog posts we will go over the life of a PFE in greater detail.

Premier Field Engineering is a part of the Microsoft Support organization. Our primary focus is to go onsite to customers that have Microsoft Premier Support and either provide a Proactive Service (we may assess an environment for potential issues or deliver training as well as many other things) or a Reactive Service (troubleshooting a DC that is not replicating). We also may do remote case work as well. Generally these are 1-2 hour calls where we answer questions about a technology a customer is implementing. We don’t handle reactive cases over the phone since we have an entire group at Microsoft that already does that. We are available 24x7x365 to go onsite and provide assistance to Microsoft Premier Customers. We provide solid guidance and advice to our customers on how to run and support their Microsoft software.

We have two facets of Premier Field Engineering. We have transactional PFEs and dedicated PFEs. Transactional PFEs generally go on a different engagement every week. This role requires a fair amount of travel. Transactional PFEs get to see many different environments and one week could be troubleshooting a down DC and the next week delivering a workshop to a classroom full of students. Dedicated PFEs are assigned to 1-4 customers. This role tends to travel less since they have a dedicated set of customers they are working with. In this role the PFE is much more familiar with the environment since they typically are working in it multiple times a week, up to five days a week if they are local. In some cases we have DSEs working out of our support centers. They provide support remotely and will travel to the customer location(s) when needed.

This blog is a Platforms blog and all of the PFEs who post here are Platforms PFEs. Microsoft has a wide range of products and technologies so our resources are broken into skillsets. Platforms PFEs handle items related to the Windows OS and components that are installable as a part of Windows. We also have PFEs for SharePoint, Exchange, Lync, and so on for the products, solutions, and technologies that Microsoft produces.

What are some tips to become a PFE?

Apply -- The first tip is to apply. We have numerous open positions and they are all listed at http://careers.microsoft.com/. The worst that can happen is you don’t get hired. Don’t fret if that happens. Work on your weaknesses and apply again.
Communications -- The PFE role requires good customer service and communications skills. We are onsite with customers just about every day and in the transactional role you are working with different people every week. In most cases you are meeting new people each week and only working with them for 3-5 days. We need to be able to work with the helpdesk staff all the way up to C-level people at the organization. I personally have gone to dinner with the CIO of a major company to discuss SAN issues they were having. Being able to work with different people and communicate effectively is critical to this role.
Leadership -- Leadership is also another key ability. This might sound odd since this is not a management position, but you need to be able to take charge of a situation. In many reactive cases everyone is going in different directions. Taking charge of a situation and making sure the right things are happening many times resolves the issue in a much quicker fashion.
Technical -- Yes this is a highly technical role and requires a deep understanding of how the Microsoft product(s) you support work.
- The PFE role does not look at your years of experience. We look at what you have been doing in your role. This can be hard to express in your resume, but you need to be clear and concise on exactly the type of work you have done.
- How many DCs do you support? Exactly what kind of support do you do? Is it 3rd tier support troubleshooting replication issues? If you do architecture or design type work generally PFE is not the right role. Microsoft Consulting Services does architecture and design type work.
- Troubleshooting – As mentioned above troubleshooting is a major part of this role. Having solid troubleshooting skills and a troubleshooting methodology is something we ask about in our interview process.
- Certification – While we do not have a hard and fast rule on certification you should have your MCSE or MCITP:EA or be working towards it.
- Don’t lie about your skills. If you list Windows Server 2008 R2 AD on your resume and maybe you installed one DC in a lab don’t list it. We will ask you questions about the AD Recycle Bin, ADWS, and so forth. We ask these questions based on the skills you have told us in your resume. If you are going to list a skill on your resume you should be Level 200+ in it.
- Be honest – While I am starting to touch a little bit on some of our interview process this goes the same as above. If you don’t know something don’t try to guess or stumble your way through it. While we look for people who are already technical we are also looking for people who can learn and adapt quickly. We provide significant training and are looking for people who have a solid base of skills to build on. If you think maybe we are touching on an area where you might not be as strong as you thought provide us direction on an area you are strong in.

The Interview Process for PFE

I will just provide a high level overview here, but this will give you an idea of how we hire in PFE.

We have a group of recruiters here at Microsoft dedicated to PFE that look at the resumes and online applications. If they find a candidate with the skills we are looking for they will contact that candidate. The recruiter will discuss the role with the candidate as well as ask some technical questions. If the role is a fit for the candidate and the candidate did well on these technical questions the candidate will be setup with an interview with one or two PFEs.

This process can vary some based on scheduling but the candidate will have one or two technical screens with a PFE or two PFEs in that skillset. If we are trying to fill a Platforms role the PFE will be a Platforms PFE. As mentioned above this is a good place to be honest with the recruiter and yourself. While you may be excited that you are in the interview process maybe you have some Platforms skills but are stronger in Exchange. I have interviewed candidates that are being interviewed for a Platforms position, but based on their resume were stronger in Exchange. They did not do well on the Platforms interview, but maybe they would have been a great Exchange PFE. Another tip here is that the Platforms interview is generally very Active Directory heavy since this is a high demand area for us. If you are stronger in clustering or something else Platforms related you want to make that known especially to the recruiter so the right PFEs are assigned to perform the interview.

If you make it through the technical screen(s) then you will have a manager interview. The managers are the ones gauging your communications and leadership skills. If you are highly technical, but can’t convey your message then this can be a problem. The managers are looking to see how well you will interact with customers as well as with coworkers and other teams inside Microsoft.

My Personal Road to Microsoft and PFE

As I mentioned at the beginning I will provide some detail on how I came to Microsoft as well as PFE. Most of what I have listed above is based on my own personal experience both in the hiring process as well as being someone who interviews candidates.

My first tip above was to apply. Microsoft was a dream job of mine since I had gotten my first 486 PC running DOS 6.0 and Windows 3.1. For some reason though as I progressed in my career and my own personal passion for computing I never applied. Maybe it seemed like one of those unattainable dreams or I didn’t think I was good enough. Finally at the company I worked for previous to Microsoft one of my coworkers was hired by Microsoft. I then thought to myself I should be working for Microsoft. I asked him how it was to work for Microsoft and what kind of jobs they had, but due to how busy he was we were never able to connect. My wife finally prodded me and said why are you waiting for him just go and apply yourself and I did. Several months went by and I didn’t hear anything. My wife happened to make a comment and said I guess you aren’t good enough and ironically they called me that day. So long story short apply and see what happens.

Once I was called by the Microsoft recruiter they lined up the first interview. This first interview is an interview with someone in the PFE role as mentioned earlier. I thought ok no big deal I am the go-to person at my company and I am strong in Active Directory and Windows. Well I was not as strong as I thought however as I mentioned above I had a solid base and a wide range of experience in my resume. I had a second technical interview. Where, again I had some weaknesses, but was honest if I didn’t know the answer to the question. I then had the manager interview as I had listed above and did very well. So as I mentioned previously we will provide deep technical training. We look for people that have a solid technical base, can clearly convey their message, and who are willing to learn.

Resources

Interview tips are frequently posted here.

http://www.facebook.com/#!/MicrosoftUSServices

Discussion about Microsoft Careers as well

http://www.facebook.com/#!/MicrosoftCareers

@MicrosoftJobs for Twitter users

Microsoft’s official site for applying for jobs

http://careers.microsoft.com

We are on Twitter as well

@pfeplatforms

Summary

So to recap and close out this blog post if you are a strong leader with great communications skills and a passion for technology we would love to have you join our team. It is a rewarding career that changes almost every week.

Thanks and I hope you found this useful. We are hiring and it is a great place to work.

Greg Jaworski

Failover Cluster Communication Failures

Charity Shelbourne — Mon, 09 Jan 2012 14:00:00 GMT

There are many potential causes for Cluster communication failures including:

Network latency
Network outages
Faulty drivers or network cards, including TCP offload issues
Misconfigured firewall rules
Security software such as anti-virus, intrusion detection, etc.

I was recently working with my customer on an issue where their Windows Server 2008 R2 eight-node Failover Cluster would randomly experience Cluster communication failures and their entire Cluster would go down. On the nodes, we would see events such as these:

In the Cluster.log file, we clearly see a networking related issue:

00003728.000017c8::2011/12/19-12:39:48.993 WARN [NETFT] Failed to send keep-alive ioctl to NetFT: 0xd0000001

00003728.00001654::2011/12/19-12:39:49.507 WARN [NETFT] Failed to send keep-alive ioctl to NetFT: 0xd0000001

00003728.00001654::2011/12/19-12:39:49.975 INFO [Reconnector- Reconnector from epoch 3 to epoch 4 waited 28.000 so far.

00003728.000017c8::2011/12/19-12:39:50.022 WARN [NETFT] Failed to send keep-alive ioctl to NetFT: 0xd0000001

00003728.000017c8::2011/12/19-12:39:50.537 WARN [NETFT] Failed to send keep-alive ioctl to NetFT: 0xd0000001

00003728.000047f8::2011/12/19-12:39:51.005 INFO [Reconnector-] Connection attempt timed out.

00003728.000047f8::2011/12/19-12:39:51.052 WARN [NETFT] Failed to send keep-alive ioctl to NetFT: 0xd0000001

00003728.00004150::2011/12/19-12:39:51.567 WARN [NETFT] Failed to send keep-alive ioctl to NetFT: 0xd0000001

00003728.00004150::2011/12/19-12:39:51.988 INFO [Reconnector-] Reconnector from epoch 3 to epoch 4 waited 30.000 so far.

00003728.000017c8::2011/12/19-12:39:52.081 WARN [NETFT] Failed to send keep-alive ioctl to NetFT: 0xd0000001

00003728.000034f8::2011/12/19-12:39:52.175 INFO [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint :~48450~.

00003728.00003948::2011/12/19-12:39:52.175 INFO [SV] Securing route from (:~3343~) to remote (:~48450~).

00003728.00003948::2011/12/19-12:39:52.175 INFO [SV] Got a new incoming stream from 48450~

00003728.00002a08::2011/12/19-12:39:52.206 ERR node was pruned out by the membership manager (status = 5892), executing OnStop

00003728.00002a08::2011/12/19-12:39:52.206 INFO [DM]: Shutting down, so unloading the cluster database.

00003728.00002a08::2011/12/19-12:39:52.206 WARN [DM] Hive::DatabaseUnloadOnShutdown: Unable to grab the lock (it will not unload the hive)

00003728.00002a08::2011/12/19-12:39:52.206 ERR FatalError is Calling Exit Process.

00003174.00002d4c::2011/12/19-12:39:52.409 WARN [RHS] Cluster service has terminated.

00003b94.00000c18::2011/12/19-12:39:52.409 WARN [RHS] Cluster service has terminated.

00003174.00002d4c::2011/12/19-12:39:52.409 INFO [RHS] Exiting.

00003b94.00000c18::2011/12/19-12:39:52.409 INFO [RHS] Exiting.

00003054.00004764::2011/12/19-12:40:53.447 INFO

Looking at the System Event log, there was no evidence of the public or private networks failing.

We applied the following two hotfixes:

2552040 A Windows Server 2008 R2 failover cluster loses quorum when an asymmetric communication failure occurs

http://support.microsoft.com/default.aspx?scid=kb;EN-US;2552040

2550886 A transient communication failure causes a Windows Server 2008 R2 failover cluster to stop working

http://support.microsoft.com/default.aspx?scid=kb;EN-US;2550886

Yet still the Cluster communication failed.

We isolated the Cluster communication by removing the Exchange replication traffic from the private network.

At that point, NIC teaming of the private network was no longer necessary since the private network was only hosting Cluster communications. We therefore broke the team and removed the NIC teaming software. We also ensured that the network drivers and firmware were at the latest and greatest.

Yet still the Cluster communication failed.

Since these are 10GB network cards, we disabled TCP offload from within the operating system and on the network cards per the following article:

951037 Information about the TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access features in Windows Server 2008

http://support.microsoft.com/default.aspx?scid=kb;EN-US;951037

Yet still the Cluster communication failed.

Despite no signs of there being latency, we increased the SameSubnetDelay to 2000 milliseconds and SameSubnetThreshold to 10 just in case there were momentary blips of latency issues that we were not catching in our traces and network analysis. Please see the following blog for more information on this: http://blogs.technet.com/b/askcore/archive/2010/02/12/windows-server-2008-failover-clusters-networking-part-1.aspx

Yet still the Cluster communication failed.

At this point, things are pretty hot. Their Exchange migration was not going well. We are pretty much to the end of our rope.

Why was the Cluster communication still failing?

Why was there no sign of the private and public networks failing at the time of our Cluster communication failures?

I started looking outside of the typical Cluster communication failures and ran across this:

"On a computer that is running Windows 7 or Windows Server 2008 R2, the network location profile that is selected changes unexpectedly from Domain to Public. Additionally, the firewall settings (these are determined by the network location profile) change to the settings that correspond to the Public network location profile. Therefore, some outgoing connections may be blocked, and some applications may be disconnected."

A light bulb went off. Angels started singing. I started jumping up and down doing my "happy dance". It all made so much sense!

I immediately requested my customers Microsoft-Windows-NetworkProfile Operational Event Log from each of the nodes to check and see if they are experiencing events that are changing and identifying from Public to domain to Public, etc. For those of you not familiar with this event log, it resides in the following location in Event Viewer: Applications and Services\Microsoft\Windows\NetworkProfile\Operational

It was definitely happening. The events were all over the place, very random, and there were some nodes already in this faulty condition.

Looking at the other nodes, we saw the same thing happening over and over again and it lined up with our previous Cluster outages.

12/12/2011 10:13:18 AM Microsoft-Windows-FailoverClustering 1135 Cluster node 'CONTOSONODE1' was removed from the active failover cluster membership.

In the NetworkProfile log, it was on the Domain Profile instead of Public.

12/12/2011 10:13:14 AM Microsoft-Windows-NetworkProfile 10001 "Network Disconnected

Name: contoso.com

Desc:contoso.com

Type: Managed

State: Disconnected

Category: Domain Authenticated

But then changed to the Public Profile.

12/12/2011 10:13:24 AM Microsoft-Windows-NetworkProfile 4001 Entered State: Identifying Network Interface Guid: {491C2D84-B062-41B2-805A-0905DC53976C}

12/12/2011 10:13:25 AM Microsoft-Windows-NetworkProfile 10000 "Network Connected

Name: Identifying...

Desc: Identifying...

Type: Unmanaged

State: Connected,IPV4 (Local)

Category: Public

12/12/2011 10:13:26 AM Microsoft-Windows-NetworkProfile 4003 Transitioning to State: Unidentified Network Interface Guid: {C83435F5-B9D8-464A-85F5-9054C3B92044}

12/12/2011 10:13:27 AM Microsoft-Windows-NetworkProfile 10000 "Network Connected

Name: Unidentified network

Desc: Unidentified network

Type: Unmanaged

State: Connected,IPV4 (Local)

Category: Public

On this node, it did change back to a Domain Profile and the Cluster Service started again. But on some of the nodes, it would stay stuck on the opposite of what was needed and a reboot would be required to bring the node back online.

12/12/2011 10:19:46 AM Microsoft-Windows-NetworkProfile 10000 "Network Connected

Name: contoso.com

Desc: contoso.com

Type: Managed

State: Connected,IPV4 (Internet)

Category: Domain Authenticated

12/12/2011 10:21:22 AM 7036 Service Control Manager The Cluster Service service entered the running state.

We had two options:

1) Open up port 3343 for Cluster Communications on all networks.

2) Apply the following hotfix to all nodes and reboot:

2524478 The network location profile changes from "Domain" to "Public" in Windows 7 or in Windows Server 2008 R2

http://support.microsoft.com/default.aspx?scid=kb;EN-US;2524478

My customer went with option 2. Their NetworkProfile Operational event logs have been clean ever since and their Cluster communications have not failed again.

Now you may be wondering how I found the hotfixes mentioned in this blog post and that is a very good question. Some of it was just through some good ole Bing searches. Check out the first two results:

Additionally, Microsoft does this great thing for Failover Clusters and some of our other products as well. They create and update Knowledge Base articles with a list of recommended hotfixes for customers to proactively apply. I highly recommend checking these out and applying any hotfixes that fit (after some initial testing in a test environment of course).

Windows Server 2008 R2 (no service pack):

980054 Recommended hotfixes and updates for Windows Server 2008 R2-based server clusters

http://support.microsoft.com/default.aspx?scid=kb;EN-US;980054

For Windows Server 2008 R2 SP1:

2545685 Recommended hotfixes and updates for Windows Server 2008 R2 SP1 Failover Clusters

http://support.microsoft.com/default.aspx?scid=kb;EN-US;2545685

Happy Clustering!!

~ Charity Shelbourne

PFE Troubleshooting Series - Second Post (and Happy New Year!)

Michael Hildebrand - MSFT — Mon, 02 Jan 2012 21:03:00 GMT

Troubleshooting Series

New Year greetings and salutations from Hilde and the rest of the PFEs out there! This is the first posting to this blog in 2012 and the second post in a multi-part series on troubleshooting. In this installment, I’ll be covering a real gem – the Event Viewer.

NOTE: The details of the tools covered in this series will be specific to the versions in Win7/W2k8 R2.

A prize in every box!

In Windows 7 and/or Windows Server 2008 R2, you are able to utilize many excellent troubleshooting tools without additional AdminPacks, Support Tools or other add-ins. These tools are part of the OS and you can count on them just being there.

Installment #2 – The Event Viewer

Many IT Pros are well-versed in translating data they find in the Event Viewer into actionable information. The newer Event Viewer offers some GREAT enhancements and features, and is even more helpful to IT Pros.

Event Log “Sub-system”

Completely re-done back in Vista/2K8 - known as “Windows Eventing 6.0”
Like many aspects of newer Microsoft products, the new Eventing subsystem relies heavily on XML standards
This makes searching, filtering and overall performance of the Event Viewer much speedier
- Especially apparent in large event-volume situations such as the Security Event Log
- In W2k3, trying to “massage” the Security Event log on an enterprise-scale DC with best-practice auditing enabled in AD was painful and in many cases, not really even workable.
In Win7/2K8 R2, you can manipulate/filter/sort/search the Event Logs and actually have it be an effective and valuable use of your time
- Searches quickly return results
- Filtering or re-sorting the Events doesn’t lock up the box while it processes
Older versions of the 32-bit OS had a maximum combined size for ALL Event Log files of around 300-400mb. If the files got near that limit, unpredictable results could occur including missing events. Those space/file-size limitations are no longer present (up to 2 TB event logs can be set – not recommended, but possible).
- http://support.microsoft.com/kb/957662
GPOs have been added/enhanced for very fine-grained management and control around most facets of the Event Logs, granular auditing, etc

Be sure to understand the inter-play of “legacy” Event Log GPO settings and the newer Event Log GPO settings (newer settings take precedence)
- http://technet.microsoft.com/en-us/library/dd349798(v=WS.10).aspx
Be sure to understand the inter-play of “Basic Audit Policy” settings and the newer “Advanced Audit Policy” settings (be careful in a mixed environment)

http://technet.microsoft.com/en-us/library/ff182311(WS.10).aspx

Custom Views

They’ll persist after you close the Event Log – you won’t lose your favorite View(s) when you close the Event Viewer

Use “Filtering” to narrow down the results you’re looking for to quickly weed out the noise and find only what you’re looking for.

Import/export them – make your favorite Views(s) and share them with your team
Consider this idea:
- Combine Filtering with a Custom View and you can make “application-specific” views of Events that you can save/export across a server farm or distribute to the application-specific team who supports the app.

Subscriptions (aka ‘Event Forwarding’)

Ever wish you could gather specific Events (even from multiple machines) to a central machine with relative ease? Now you can!
Note – this is not a viable alternative for an enterprise monitoring system like SCOM but in a pinch or for a small-scale or narrowly-focused situation, this could be just the ticket

Event-triggered Tasks

Ever wish you could kick off a script or command, or even an email, right when an event occurs? Now you can!

Two ways:
- Basic Task – from the Event Log entry itself > right-click > “Attach a Task to this Event” (or log)
- For more advanced options, open Task Scheduler and drill-down to Event Viewer Tasks > Create Task

Enter all appropriate info and on the ‘Triggers” tab, choose “On an event”

· Application and Services Logs

Provide detail on a vast array of OS activities
GPO processing
- Why aren’t my GPO(s) applying?
- Don’t need to enable this like USERENV logging
DNS Client processing
- Why isn’t my DNS record(s) updating in DNS?
Scheduled Tasks processing
- Why is my Scheduled Task failing?
Windows Backup processing
- Why isn’t my Backup job completing?
DHCP Client
- Why isn’t my client getting an IP?

· Save Selected Events

Save a subset of Events to their own EVTX file for further analysis/filtering and easier portability
Instead of copying a 250MB file across the WAN from a remote server, you can copy over a 32kb one

· FIND the needle in the haystack

Right-click a Log, click “Find”, enter a User ID, Event ID, keyword, etc and let the magic begin
Quickly find who rebooted the server recently?

Don’t wait another minute - jump in and explore the Event Viewer. Progress your troubleshooting and glean more actionable information and details about the system/situation with these great features.

Come back next time for a discussion of yet another tool waiting for you “in the box” of Win7 and 2K8 R2…

CHEERS!

Hilde

In Search Of…. Roaming Active Directory Clients. (How to scriptomatically identify missing Active Directory Subnet Definitions)

dsymalla — Mon, 26 Dec 2011 14:45:00 GMT

If you want detailed information on how Active Directory sites and subnets work to help clients find their closest domain controller, there is good information on TechNet. In short, you need to ensure that you’ve defined (in Active Directory Sites and Services) Active Directory sites for all of your physical locations that have domain controllers. Additionally, you need to specify which subnets are in use for your AD clients. Then, you map every subnet to a corresponding AD site. Clients then magically find their nearest domain controller.

However, even the best plans go awry. For example, what if your network team decides to deploy a new subnet for their shiny new wireless infrastructure, but they conveniently overlook the need to tell you about it? You’ll eventually become aware of the situation when you receive the complaints. “Why is Active Directory so slow?” An investigation reveals that wireless clients in a conference room in your headquarters are using the services of a Domain Controller in your most remote location for authentication and group policy. There is no subnet defined in Active Directory for this new wireless subnet, so any clients on the undefined subnet will randomly roam the planet in search of a domain controller.

How can you proactively avoid these unpleasant scenarios? The ideal solution is to actually communicate with your network team regularly, so you know that all your AD sites/subnets are complete and accurate. And they know they should tell you when things change. But since most of us have never seen this ideal world of unicorns and fairies, you should be looking for signs of trouble.

Find the “Roamers” in the LogFiles

Every domain controller logs when they encounter a roaming client (a client that authenticates from an IP address that does not belong to defined AD subnet). This information is written to the Netlogon.log file (default location c:\windows\debug). If you crack open a Netlogon.log file you will see a list of roaming clients:

Simply look for lines in the log with NO_CLIENT_SITE, and note the hostname and IP address. Then you may be able to figure out the missing subnet definition, add the missing subnet definition to AD and associate it to the appropriate AD site. Alternatively, ask your network team about the IP address/subnet and let them tell you which AD site is appropriate.

Automate the Review of Log Files

If you have more than one domain controller (and I hope you do), reviewing log files manually is a chore you’re not likely to cherish. So now you should figure out how to script it.

Step 1: Supply or Dynamically Create a List of Domain Controllers

First, you need to know of all the domain controllers in the forest, so you can pull information from their netlogon.log files. You could just create a text file with all the DC names and read them in. I would prefer; however, to dynamically generate a list from Active Directory. Furthermore, I like to create a PowerShell function for this task, since it’s something you do often with AD-related scripts. Finally, my code remains a bit dated, because I’m never sure if a customer has old (2003-style) or new (2008R2) domain controllers. So I try to avoid some of the 2008R2 AD cmdlets that can make your code much simpler. The logic of the code is to find any Server objects in the configuration partition. Then, we validate that the server object has an NTDS Settings object associated with it. This should accurately identify any DCs.

### DCDiscovery - All DCs in the Forest
Function EnumerateDCs
{
   $arrServers =@()
   $rootdse=new-object directoryservices.directoryentry(LDAP://rootdse)
   $Configpath=$rootdse.configurationNamingContext
   $adsientry=new-object directoryservices.directoryentry(LDAP://cn=Sites,$Configpath)
   $adsisearcher=new-object directoryservices.directorysearcher($adsientry)
   $adsisearcher.pagesize=1000
   $adsisearcher.searchscope="subtree"
   $strfilter="(ObjectClass=Server)"
   $adsisearcher.filter=$strfilter
   $colAttributeList = "cn","dNSHostName","ServerReference","distinguishedname"
   Foreach ($c in $colAttributeList)
   {
      [void]$adsiSearcher.PropertiesToLoad.Add($c)
   }
   $objServers=$adsisearcher.findall()
   forEach ($objServer in $objServers)
   {
      $serverDN = $objServer.properties.item("distinguishedname")
      $ntdsDN = "CN=NTDS Settings,$serverDN"
      if ([adsi]::Exists(LDAP://$ntdsDN))
      {
         $serverdNSHostname = $objServer.properties.item("dNSHostname")
         $arrServers += "$serverDNSHostname"
      }
      $serverdNSHostname=""
   }
   $arrServers
}

Step 2: Reach Out and Grab the Netlogon.log Files from Each DC

For simplicity sake, let’s assume that the DCs have default settings for the OS installation, so the netlogon.log file can be found at C:\Windows\Debug\netlogon.log. The code does check that this location/file exist before proceeding. If the DC cannot be reached, or that location/file does not exist, write to the screen that the log file for that DC cannot be found DC and move on. Note how the cmdlet Test-Path can be used to validate connectivity to the log file. Any errors with connectivity, permissions, or missing path/file should be handled with Test-Path.

The code also includes some error checking (Try{}; Catch{}; Finally{}). If we fail to get content from the netlogon.log file (it’s empty, for example), the Try{};Catch{};Finally{} should handle the errors.

Finally, the code has to do some string gymnastics to pull out specific content from the netlogon.log files. Specifically:

1. Only the last 500 lines of the netlogon.log file will be analyzed
2. The line must contain “NO_CLIENT_SITE”
3. We only want the part of the line that has “HOSTNAME IPADDRESS”

The remaining lines containing only “HOSTNAME and IPADDRESS”, across all DCs, are combined into a single array.

### Collect the FQDN of all DCs in the Forest
$allDCsinForest = EnumerateDCs
### Collect the last 500 lines of the Netlogon.log file from each DC, pick out lines with NO_CLIENT_SITE and combine them
$combinedNetLogon = @()
ForEach ($DC in $allDCsinForest)
{
   $NetlogonPath = “\\$DC\c$\Windows\Debug\netlogon.log”
   If (Test-path $NetLogonPath)
   {
      Try {$netlogon = (Get-Content $NetLogonPath)[-1..-500]}
      Catch {write-host "Can't read Netlogon.log for $DC. It may be empty." -foregroundcolor red;$NetLogon=$null}
      Finally
      {
         If ($NetLogon)
         {
            forEach ($line in $Netlogon)
            {
               If ($line -like "*NO_CLIENT_SITE*")
               {
                  $startPos = $line.IndexOf("NO_CLIENT_SITE") + 16
                  $length = $line.length
                  $length = $length-$startPos
                  $line = $line.SubString($startPos,$length)
                  $combinedNetlogon += $line
               }
            }
         }
      }
   }
   Else {write-host "Can't find Netlogon.log for $DC" -foregroundcolor red}
}

Step 3: Reach Out and Grab the Netlogon.log Files from Each DC

To make data analysis/manipulation easier, create a new table/array which contains IP Address and Hostname of roaming clients. This involves splitting each line into IP Address and Hostname. Then, create an object with two properties – IP and Host. Finally, add this object to an array. The resulting array can then be sorted to remove any duplicate entries.

### Walk the array with NO_CLIENT_SITE hostnames/IPs. Create a new table/array of IP addresses and Hostname. Remove
### duplicates.
$colIPs = @()
ForEach ($line in $combinedNetLogon)
{
   $splitLine = $line.split(" ")
   $hostname = $splitLine[0]
   $IPadd = $splitLine[1]
   $objIP = New-Object System.Object
   $objIP | Add-member -type NoteProperty -name IP -value $IPadd
   $objIP | Add-member -type NoteProperty -name Host -value $hostname
   $colIPs += $objIP
}
$listofRoamingClients = $colIps | sort-object -property IP –unique

Step 4: Report the results

Now just spit out the results – either to the screen or to a CSV-formatted log file.

### Comment out the line below, to not display results on screen
$listofRoamingClients
### Dump list to a file
$listofRoamingClients | export-csv .\listofroamingclients.txt

Run/Re-Run Your Script, and Enjoy the Fruits of Your Labor

Now you’ve got no excuses for roaming clients. You can identify them as often as you wish and fix your subnet definitions appropriately. If you still find your network team less than helpful in resolving issues, simply create a new subnet definition for their workstation IP address(es) and associate it with the most remote AD site in your environment.

Happy cleaning.

Doug Symalla

PFE Troubleshooting Series - Initial Post

Michael Hildebrand - MSFT — Sun, 18 Dec 2011 14:13:00 GMT

Hello, my name is Mike Hildebrand (aka 'Hilde') - I'm a Dedicated Premier Field Engineer with Microsoft. Welcome to the first in a multi-part series from us on troubleshooting. From mindsets to toolsets, a variety of Microsoft Premier Field Engineers (PFEs) will discuss a variety of skills, tips and tricks to help you build your ability to troubleshoot issues from the simple to the complex. These will all build on a foundation of patterned “thought” that most of us do in our day-to-day lives without even thinking about it.

For example: your car won’t start
- Do I have the right key?
- Does the starter/engine turn over or do I only hear clicks?
- Is the car in gear?
- Do the lights work?
- Is the clock blinking “12:00”?
- Likely culprit - the battery is having issues - investigate further.

Installment #1 - The <Not Always So> Obvious

We all have faced similar situations to the dead battery issue but often, in our “IT lives”, many of us are driven into an almost immediate panic in the face of a severe problem (i.e. a complete datacenter failure or a single, but REALLY mad user or VIP). We start changing a variety of things hoping we’ll get it fixed quickly. We begin pulling cables, rebooting servers, and often cause more harm or do nothing to reduce the time to resolution for the current event/issue.

First, remind yourself to breathe. Keep your head. Breeeeeathe. Control the situation as much as you can (rather than only reacting to it) and lean on a methodical, repeatable process to help guide and support you while you begin to work the issue.

Second, clearly define the problem as much as possible. You’ll often need tact here to keep from fanning the flames. Have you ever asked a super angry end-user who’s reporting a PC problem “Is the PC powered on?” The eerie silence on the other end of that conversation is enough to make anyone’s forehead start to bead up with sweat. Here are a few examples of “What’s reported” vs “what’s really wrong:”

Month-end printing is down – No one can print

Printer was powered on but disconnected from the network

AD is down – no one can login

Network cable not connected for one VIP user

Ask the focus questions:

Who all is impacted (other users, other sites, other ____, etc)
When did it start (just now, after I upgraded my PDF software, after my last reboot, two weeks ago but just now calling in, etc)
What is broken? What changed, if anything, that might have caused the issue?
What has been done so far to troubleshoot (few issues will get to you prior to any other changes being made to the ‘situation’)

Clarify what the problem is, as well as clarify what the problem is NOT

Email is not working – VS – I can send emails, but I’m not getting any
The Internet is down – VS - I can get to external websites but not internal ones

Consider likely causes even at the expense of seeming obvious - just be ready to duck.

Is the printer turned on? Are you sure? What does the display read (if anything)? Is the network cable (skinny blue wire) connected to the printer? Oh, there were painters there over the weekend? Perhaps they moved/unplugged it?
Is the cable connected to the VIP’s desktop? Are you sure? Do you see link lights?

If the obvious fails, start diligent but simple troubleshooting at the physical layer and work your way up to the more complex systems/environments (think back to your early MCSE tests and the “OSI model”).

Is network connected? Try reconnect/reseat. Examine cord/plug(s).
Is there link activity - green/yellow lights? Flashing or steady?
TCP/IP settings and name resolution

Never underestimate the power of PING (but don't forget that it might be blocked by firewalls)
IPCONFIG
Static or DHCP-assigned IP?
If DHCP, did you get an IP?
Is the IP 169. ?
Is gateway defined? Is it accurate?
Can you ping the gateway?
Is DNS and/or WINS server(s) defined? Are those entries accurate?
Can you ping the DNS/WINS IP address(es)?
Can you ping an external web site (www.msn.com)?
Perhaps a static DNS entry or IP was set for home ISP access?
Can you get to the intranet?
Can you get to the Internet?
Can you send/receive emails?
- Internally?
- Externally?

Once you think you’re onto something, try to make one change at a time to actually discover the root cause.

This is often VERY difficult to do
- Mgmt in your face – PRESSURE!! FIX IT!!! IS IT BACK UP YET?!?! BUSINESS IS DOWN!! THE SKY IS FALLING!!! SEV-1!! SEV-A!! CRITSIT!!!! OH THE HUMANITY!!!!
- Other people/teams involved possibly making changes
  - Covering up or repairing their ‘oops’ - "roll-back that SAN controller firmware update we pushed"
  - Trying to help but working/making changes in silos
  - Unaware that you’re working the same issue - "I didn't realize your business-unit/site/etc was affected, too?"

Continue to expand on the steps and thought-framework presented here and you’ll continue to be better equipped to manage difficult situations and make progress on simple or complex problem resolution.

Also, be sure to check out a blog entry by our Dir Service folks on troubleshooting – it is a skill that you should be constantly evolving and growing.

http://blogs.technet.com/b/askds/archive/2011/12/08/effective-troubleshooting.aspx

Tune in next time for a discussion of the first tool installment in this series where we’ll take a look at the World Famous Event Viewer.

Cheers! Michael Hildebrand