April, 2010

Hello there fellow humanoids, Ned here again. Last week the Mail Sack was a bit thin. This week I had to wrestle it under control. If your interesting question doesn’t show up here, it may just be in the backlog – nothing personal, cobber, maybe next week.

Let’s move out.

• Accidental deletion protection in Win2003
• Forestprep rollup behavior
• DFSN memory usage patterns
• DFSR sync time estimation
• DFS management status check
• GPP LDAP item-level targeting

Question

In previous blog posts AskDS has talked about setting “Protect object from accidental deletion” in Windows Server 2008 and later.

I run Windows Server 2003, and that checkbox is not available for me. I tried adding the Everyone group to DENY delete on a test OU, but I can still delete it as the Administrator. What am I missing?

Answer

For Win2003, follow this step-by-step article:

http://technet.microsoft.com/en-us/library/cc739350(WS.10).aspx

It covers how you have to also set specific deny permissions on the parent object (in my case, the domain root contoso.com). If you do it correctly, when you attempt to delete an OU you will get:

Remember, this is not a panacea – it’s only preventing accidental deletions. An admin that really wants to zap this OU still can, as they can remove the DENY perms easily. The only way to prevent an admin from deleting an OU is to fire him.

Question

I am running a Win2003 forest currently and I will soon be deploying new Win2008 R2 DC’s. Do I need to run ADPREP.EXE /FORESTPREP for Windows Server 2008, then for Windows Server 2008 R2?

Answer

Nope. Every version of ADPREP we release contains all previous Windows schema updates. If you wanted you could upgrade your schema from Windows Server 2000 all the way to 2008 R2 in one go.

It’s a reasonable question – not like most folks are constantly upgrading schemas…

Question

Do you know the rough memory consumption I’d see in DFS Namespaces with X number of Links (i.e. “Folders”)? This would be within Windows Server 2008 R2.

Answer

For a test I created 100 links in a V1 (Windows 2000 style) namespace through a few quick FOR loops - you’d be amazed what you can do with MD, NET SHARE, and DFSUTIL in a pinch. I found the following after restarting the DFS service on that hosting root…

Here is private working set and committed memory with no DFS Links:

Here it is with 100 Links:

As you can see, not much difference between 100 links and 0 links. Memory went up ~300KB private working set within the user-mode heap memory. Backing Kernel memory of pool and non-paged pool were pretty much unaffected.

So then I took it to 1000 Links:

That made it roughly 3MB higher than the usage at 0 links. So it is actually quite linear and predictable in a simple repro. 100 links was ~300KB, 1000 links was ~3MB.

Finally, I converted the namespace to V2 (Windows Server 2008 Style):

Added a bit more per link, but not much. This is because we have secret relationships with hardware vendors that require us to have more RAM as we release later operating systems. Nah, just kidding, it’s because V2 namespaces sacrifice a bit of memory for higher reliability and scalability. Take off your tinfoil hat, fella.

DFSMGMT.MSC has an effect here as well – the more I used it and navigated around a namespace on that server, the memory usage in the service kept climbing slightly as it retrieved data to send to the snap-in. But that should be rare. As should having even 100 links, much less 1000.

This question came from our pal Mark, who asks always questions that force me to repro. ;-)

Question

Is there some way to estimate initial sync time in DFSR?

Answer

Think of how accurate the progress bar is when you are using Windows Explorer or Internet Explorer over a slow WAN or the Internet – often very inaccurate, right? That is a very synchronous operation where you are typically copying only one file and it will not be changed by anyone in the middle of being copied. The progress bars tends to move fast, then slow, it freezes, gives outlandish times, and then suddenly finishes.

Now imagine you are having to track progress on 16 files at a time to 30 different servers on 20 different networks of varying speeds and quality, which are also servicing other network data. Impossible to do, pointless to estimate – it will always be wildly wrong. So we don’t bother.

Question

We recently upgraded from Win2003 and started using the new DFSMGMT.MSC console for our DFS Namespace administration. The old DGSGUI.MSC had a little “check status” option that I liked, which is gone now. Can I get that back or use something else?

Answer

DFSDIAG.EXE will tell you most things about the health of your environment, not just shares. For example, here I have a 3-server link and one of them is offline:

The old DFSGUI.MSC way of doing this was inherently flawed – it missed a lot of other problems and mainly gave a false sense of security. All it checked was that the share could be enumerated. A common complaint I got supporting Win2003 was “What do you mean DFS isn’t working, it says right here that it passed the status check!?!”

An even better idea than staring at DFS in the management tools is to run System Center or a third party, and have them check status for you. Then it can tell you when things aren’t working, leaving you to catching up on your reading of Windows 7 Phones at Engadget. WANT!

Question

I am using LDAP Query item-level targeting in Group Policy Preferences and trying to provide %USERNAME% as a variable to part of my filter, but it’s not working. I’ve already installed the KB976398 hotfix.

Answer

GPP doesn’t necessarily care about Windows environment variables – actually, no application is required to. To see all the variables that GPP will accept as part of configuring a policy or targeting, click on any field in the editor and then press F3.

So in this case, you’d want to use %LogonUser% to get the same info that %UserName% provides:

And for this particular case, we’d use this to apply a policy to a user:

What I don’t necessarily understand is why you’d want this filter. Seems like it would always apply, so why not just make it a user policy at the domain? Oh well, it’s a useful example. :-)

====

Finally, I sent my Pop, Uncle, Aunt, and step-mother to a Cubs game on Thursday. Great seats, right along the home dugout in Wrigley Field. And in true Cubs fashion, the game went like this:

http://mlb.mlb.com/mlb/gameday/index.jsp?gid=2010_04_29_arimlb_chnmlb_1&mode=wrap

13 runs by the Diamondbacks. 13. That’s a football score!

Until next time.

- Ned “$%^#&&* Cubs” Pyle

Ahoy hoy. The BPA release release cycle has just ticked over once for Windows Server 2008 R2. This means that you can now install – through Windows Update or the Download Center – add-ons that snap into Server Manager and will tell you if you are following MS best practices for your installed roles. Simply install the update, look at the role, and click “scan this role”. After some noodling, BPA will kick out info.

For example, it appears I stink at running DFSN…

But I rule at running DFSR!

Here’s what just shipped:

• Update for Best Practices Analyzer for HYPER-V for Windows Server 2008 R2
• Update for Best Practices Analyzer for Network Policy and Access Services for Windows Server 2008 R2
• Update for Best Practices Analyzer for Active Directory Rights Management Services for Windows Server 2008 R2
• Update for Best Practices Analyzer for Application Server for Windows Server 2008 R2
• Update for Best Practices Analyzer for File Services for Windows Server 2008 R2
• Update for Best Practices Analyzer for DHCP Server for Windows Server 2008 R2
• Update for Best Practices Analyzer for Windows Server Update Services for Windows Server 2008 R2

Download them all from here.

Read more about them all here.

You need Windows Server 2008 R2 to use any of this stuff, so add it to your list of reasons to upgrade if you haven’t already. More BPA’s coming out when they… come out. Including updates to these existing ones, in theory.

Hey Mahesh, where’s your post?

Ned “beat filecab to the punch for once”  Pyle

KB

Blogs

Hi there world. It’s been a particularly gnarly week: not too many questions that most people would find relevant, plus it was just crazy busy (stupid Windows 7 and R2, being all popular and whatnot, leads to a lot of USMT work for me… D-: ). Hence – late posting with not much sirloin.

Get to the choppah!

• DFSN V2 root requirements
• Default AD ACLs

Question

I’ve installed my first few Windows Server 2008 R2 computers and tried to make them DFS Namespace V2 (i.e. “Windows Server 2008 Style”) root servers. I am having a bunch of issues setting it up though. For example, using DFSMGMT.MSC or running this command:

DFSUTIL.EXE ROOT ADDDOM \\TESTSRV\Test V2

always returns:

Could not execute the command successfully
SYSTEM ERROR - The version of the operating system installed on the server is incompatible with the functional level of the domain or forest.

I’ve had various Win2008 servers for a while now and they add as V2 roots just fine in the same domain and forest. I also cannot delete previously created links in V2 namespaces using the R2 servers, I get error:

The folder cannot be deleted. Cannot complete this function.

What’s up here?

Answer

You need to raise the forest functional level to Windows Server 2003 or higher; right now it’s at Windows 2000, I’ll wager. Windows Server 2008 R2 DFSN requires the higher level due to how it does some AD object creation operations differently than Win2008. Confirmation here.

Question

Did the default SACL’s in Active Directory change between Windows 2000 and Windows Server 2003? It seems that when Directory Services Access auditing is enabled on a Win2003 domain, the logs are much quieter, but Win2000 is noisy as heck. If true, when I upgrade a Win2000 domain to 2003 will it get less chatty?

Answer

Indeed, they did change based on the experience we had with Win2000.

vs.

Yeowza! Win2000 has very aggressive settings, but Win2003 makes you go set SACL’s as needed for nearly everything. This is definitely the better approach as every company will have a different idea on what they want to audit.

And no, they are not changed again by subsequent domain upgrades. They are a function of the first DCPROMO in a domain only, not any later ones. If you wanted to make an upgraded domain less chatty, examine the domain root DN; you will see where most of the SACL’s are being inherited from. :-)

Statement

I am going to $^%#&*^$&*# destroy you for posting my email address on the Internet! I hate your face so much! I get nothing but <redacted, but hilarious> spam now! AAARGGGHHH!!!!

Signed, Mike O’Reilly.

Reply

I lol’ed.

Have a good weekend, Earth.

Ned “has on-call phone, so expect grumpy replies Monday” Pyle

KB

Blogs

Hello folks, Ned here again. Recently I was asked to provide a technical assessment of the risks of continuing to use the File Replication Service (FRS) and the benefits of migrating to DFSR, all regarding SYSVOL on domain controllers. I thought I’d find a decent set of documentation on TechNet, polish it up and send it along – I was wrong; I had to spend several hours coming up with a complete list.

Now you can reap the benefits. Hopefully this helps you convince yourself or your management that the time has come to cut the cord on FRS, especially if you have already deployed your Windows Server 2008 DC’s.

I sure hope you like bullet points!

The risks and downsides of FRS and SYSVOL

• FRS code is in maintenance mode, where Microsoft does not accept design change requests or bug fixes except when related to security.The last FRS bug fix update was released in KB939667. It was for Win2003 and nearly 3 years ago; Win2008 has never gotten an FRS bug fix update in its history.
• Additionally, the FRS component began deprecation starting in Windows Server 2003 R2:  
  • The Microsoft product team stopped investing in FRS in Windows Server 2003 R2, when it was decided to build DFSR and have that replace FRS even for SYSVOL replication
  • DFSGUI.MSC FRS management tool was removed in Win2008
  • FRS component no longer installable in Win2008 R2 except for SYSVOL replication on DC’s
  • FRS component automatically uninstalled during in-place upgrade of Win2008 R2 non-DC’s 
• FRS scalability and performance are significantly lower than DFSR, especially with frequently modified files, larger data sets, larger files, and slow wide area networks. FRS always replicates an entire file regardless of modification type (i.e. a security change, data change, attribute change, or file name change each replicate the entire file)
• FRS does not include a public development interface (API or WMI) for monitoring, and it’s interface for management is limited
• FRS does not have a native, supported health reporting mechanism.
• FRS does not have a native, supported monitoring solution from Microsoft System Center. Only has legacy unsupported tools like Sonar, Ultrasound, CONNSTAT, etc. with limited MOM 2005 integration
• FRS has limited performance monitoring counters through PERFMON/ETW
• FRS does not have a working self-healing system for problems like database corruption,  journal wraps, and morphed folders
• FRS does not fully support RODC SYSVOL replicas and allows data to become unsynchronized without chance of automatic resynchronization
• FRS does not support the inter-site change notification flag, leading to artificially slow replication between DC’s in different AD logical sites
• FRS does not have significant built-in instrumentation (debug logs, event logs) for troubleshooting and debugging

The improvements and upsides to DFSR and SYSVOL

• DFSR code is in active development with full product DCR and QFE support. Hotfixes for feature improvements as well as bug fixes are regularly released and also integrated into new Service Packs.
• DFSR scalability and performance are designed to be superior to FRS. This includes:
  
  • Ability to replicate partial file changes using RDC (block-level delta replication) rather than entire files
  • Support for cross-file RDC that can construct new files from similar files, rather than replicating the new file over the wire (when using Enterprise edition)
  • A more efficient file compression on staged files
  • The number of files that can be replicated inbound and outbound simultaneously is significantly increased
  • Support for unstable and slow networks with asynchronous RPC
  • Support for more efficient OS kernel mechanisms introduced in Win2008 like unbuffered I/O, low priority I/O, and asynchronous I/O’s
  • No staging of smaller files (<=64KB by default)
  • Staging compression can be controlled on a per-file type basis
  • Scalable to a supported (not hard) limit of 10 terabytes of data. Although if you have 10TB in SYSVOL, you are doing it wrong buddy.
• DFSR has a public interface (using WMI/DCOM) managing and monitoring all aspects of DFSR, including backlog (and files currently on the wire in Win2008 R2).  It also includes command-line tools that give feature parity with the GUI management tools
• DFSR has a native, supported health reporting mechanism that is available through the GUI or command-line and generates HTML/XML outputs
• DFSR has several releases of native, supported monitoring solutions from Microsoft System Center via management packs. The new Win2008 R2 File Services MP is also in final stages of beta
• DFSR has more complete performance monitoring counters through PERFMON/ETW
• DFSR has a self-healing system for problems like database corruption or journal wraps. Due to improved replication performance and the ability to enable content freshness protection, it is also very unlikely to ever see a journal wrap in the first place. DFSR also does not create morphed folders like FRS and instead uses a conflict resolution algorithm
• DFSR supports RODC SYSVOL replicas and does not allow SYSVOL’s to remain out of sync in Win2008. In Win2008 R2 originating I/O in SYSVOL is completely blocked with a filter driver on RODC’s
• DFSR  - while it does not directly support the AD DS inter-site change notification flag – always replicates SYSVOL immediately and continuously with its own internal change notification as long as the schedule is open; these scheduled windows are in 15 minute blocks and are assigned on the AD DS connection objects. If the current time matches an open block, you replicate continuously (as fast as possible, sending DFSR change notifications) until that block closes. If the next block is closed, you wait for 15 minutes, sending no updates at all. If that next block had also been open, you continue replicating at max speed.
• DFSR has significant built-in instrumentation for troubleshooting and debugging, including considerable event logging and a large number of highly verbose debug logs (1000 debug logs maintained under compression by default in Win2008 R2, at the second to highest level of verbosity by default)

A table

Here’s a different way of looking at it, as I know executives love their matrices:

Now go migrate. For most customers it will be a few hours of work. Your manager may not even have time to buy you lunch on a Saturday.

Stay tuned for another article about the benefits of using FRS. Its title will be “the shortest blog post ever written” and will contain only a picture of my dogs eating their toys. Here’s a preview.

A special thanks to Mahesh from the DFSR product team for his timely review and contributions to this write up. You rock dude.

Until next time,

Ned “nom nom nom” Pyle

Hi all, Ned here. A new Technet operations guide has been published that walks you through how to migrate from FRS to DFSR for non-SYSVOL folders running on Windows Server 2003 R2 and Windows 2008:  

DFS Operations Guide: Migrating from FRS to DFS Replication
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=a27008a8-4b28-49cc-80b5-05b867440af9

We (Mike Stephens and I) are also working on a migration tool to be available from CodePlex. There is no ETA on this as even though we are code complete, we are entering the lawyer phase that is out of our control. The tool gives you a way to list, export, and delete your old FRS replicas then configure DFSR to replicate the same files on the same servers, all through a command-line interface, all pretty automagically. More on that when I have something concrete to tell you - again, please don't keep asking for ETA in the meantime, I am just going to ignore you. :-)

Ned "thank goodness that's done" Pyle

Hello all, Ned here again. After a brief absence, the rocket sled that I use to carry my disembodied head around has brought me back to AskDS headquarters. The coup is over and I have emerged triumphant again. You won’t be hearing from Jonathan until the truth serum wears off.

So let’s talk some talk.

• NTRIGHTS in Win2008+
• USMT free space
• Schema Master isolation
• KB954968 and FSRM in Win2008+

Question

There was a tool called ntrights.exe in the Win2003 resource kit tools, but we couldn't find one for Windows Server 2008. I need a command-line based tool to add security privileges for users.

Answer

The ntrights.exe tool still works fine even in Windows Server 2008 R2 and Windows 7 so feel free to use it. You could also use secedit.exe /configure with a custom INF file that added the user rights (good idea Mike). Not to mention group policy – adding privs with the command-line sounds like a lot of extra work to me.

Question

How much free space is needed for temporary files doing a USMT 4.0 scanstate? I grok that it arbitrarily requires at least 250MB as stated here, but could I need more? I plan to have the store file written to a network drive.

Answer

By default, the USMT temp/working folder is the operator's %TEMP% folder (obviously, this is local to the computer). The full set of files is not gathered here; the store is updated in a serialized fashion directly. The temporary file that USMT 4.0 creates is used purely to track work and back the stores catalog data and non-file data.

When running scanstate /p the estimator for space figures how big the backing file will get, then adds an additional 1MB of "fudge factor". The binary size of gathered user data files never matters -just the quantity of units to be migrated.

For example, in a repro I had a Windows 7 client with eight profiles. This created a temporary backing file that was 44MB. Then when I cut the migration down to a single user profile the temporary file was only 9MB. When I added 300+MB of data to my profile (so only 20 files, but each being very big), the temporary space usage estimate did not get appreciably larger.

<?xml version="1.0" encoding="UTF-8"?>
<PreMigration>
  <storeSize>
    <size clusterSize="4096">96075776</size>
  </storeSize>
  <temporarySpace>
    <size>10576664</size>
  </temporarySpace>
</PreMigration>

<?xml version="1.0" encoding="UTF-8"?>
<PreMigration>
  <storeSize>
    <size clusterSize="4096">425594880</size>
  </storeSize>
  <temporarySpace>
    <size>10617624</size>
  </temporarySpace>
</PreMigration>

Also, you can use the USMT_WORKING_DIR override environment variable to make the temporary folder a remote server path. But the migration is going to get much slower. My repro scanstate ran ~2-3 times slower because I had traded fast local I/O for comparatively slow network I/O. That was on gigabit network with no contention. A hard-link migration would be much faster.

Question

Is there a way to isolate a DC in order to do an AD Schema upgrade? I cannot find any documentation on how to do this.

Answer

Isolating the Schema Master for ADPREP /FORESTPREP is not tested by the Product Group and not recommended*; we intentionally try to block you from this scenario starting in Win2003 SP1. Attempting to do so will return:

“Adprep was unable to extend the schema.
[Status/Consequence]
The schema master did not complete a replication cycle after the last reboot. The schema master must complete at least one replication cycle before the schema can be extended.
[User Action]
Verify that the schema master is connected to the network and can communicate with other domain controllers. Use the Sites and Services snap-in to replicate between the schema operations master and at least one replication partner. After replication has succeeded, run adprep again.”

This was added back in Win2003 SP1, based on the fact that customers were causing horrendous issues trying to isolate their Schema Master FSMO servers during a migration or never verifying that the Schema master was healthy, then incorrectly (or never) reattaching them to their domain while the now split schemas diverged.

Our supported and recommended methodology is for you to test the migration in your lab with a copy of your current forest/schema; if there are going to be problems in the schema upgrade, they will happen in your lab. Likewise if there are going to be problems with the Schema itself, they would occur there as well. Prior to upgrading your schema, we recommend that you get a good System State backup on all DC’s; but we recommend you do this every day, not just for Schema upgrades. If there was some irreconcilable issue you could restore your forest from backup using those system states using our forest recovery info here: http://technet.microsoft.com/en-us/library/planning-active-directory-forest-recovery(WS.10).aspx

This was an especially excellent question – sometimes we imply through an absence of documentation rather than stating things flat out, unfortunately.

* And to be clear here , yes it is possible to disable replication temporarily. Older documentation even used to say things like "disconnect your schema master" or "block outbound replication". Newer documentation does NOT, as we now have a decade's worth of experience with customers using those techniques in lieu of proper testing. And dealing with the fallout of that! We've had customers disable the replication then forget to ever turn it back on again; guess what happened after 61 days?

When the AskDS team says something is possible, it often gets construed as it's recommended and supported. It's not. Testing your schema update in a lab costs nothing thanks to free virtualization products aplently. Do that and you cannot go wrong.

Question

Do the registry values in KB954968 apply to Windows Server 2008 and 2008 R2 also, in regards to configuring FSRM hard quotas to work with DFSR?

Answer

The registry values still work, yes. But they shouldn’t be as necessary in 2008/2008 R2 DFSR because all of the folders and files that FSRM would count against quota are now under a reparse point. The reparse point will prevent the quota from being enforced in this circumstance.

So for example, if you set an FSRM quota against c:\condelrf, it would not affect the contents of the c:\condelrf\dfsrprivate folder:

Because that is actually this reparse point target location:

So the data in there is not covered for quota. The KB and registry change from 2003 R2 were necessary because back then, dfsrprivate was a real folder under the DFSR replicated folder. When quota was hit there, kaboooooom.

You still need to make sure that you approach hard quotas with extreme caution though:

http://technet.microsoft.com/en-us/library/cc773238(WS.10).aspx#BKMK_064

DFSR and FSRM do not really have a good interop story – using them together is not something I’d personally recommend, after many, many support cases fixing the fallout of inappropriately configured hard quotas.

Finally, some sad news. Our fearless manager Mike O’Reilly - he of the swapped desk and the cubicle tree - has left us for greener pastures. At least as green as pastures get in Newfoundland. Mike is now a director at a large construction firm back on his native island in his pseudo-country we call America’s Hat. In fond memory, here is his email address: moreilly1974vw@hotmail.com. I sure hope it doesn’t get crazily inappropriate spam, what with it being out here on the Internet forever.

That’s all, have a nice weekend folks,

Ned “ “ Pyle