Active Directory Blog : Client Side Caching and Folder Redirection

All The Logging In The World

Tim Springston — Mon, 08 Oct 2007 19:40:00 GMT

There’s normal troubleshooting and then there’s the stuff you do when the basic troubleshooting doesn’t get things resolved. Normal troubleshooting can be things like selecting “last known good” on a reboot after installing a new driver and having a blue screen. Or perhaps uninstalling and then reinstalling an application, or altering settings for the application or operating system to alleviate a problem. Sometimes we have to dig in and find out more. Many admins out there in the world live that every day. Which is why we add methods to find out more into our products. This post is all about listing all of the data gathering methods that a Directory Services person may ever need to know. Since there are so many it will be difficult to organize well in one uber post but I’m going to put out here for you all anyway, disorganized or not....(read more)

Tracking User Environment Creation

Tim Springston — Tue, 21 Aug 2007 05:40:00 GMT

In my soliloquy of AD logon you heard some broad generalities intended to give a general understanding of the intended design and how it all fits together. In this post I hope to give you a more detailed idea of how it works.

How to enable user environment debug logging in retail builds of Windows

http://support.microsoft.com/kb/221833

For years we at Microsoft have told people how to enable USERENV logging and information, sometimes detailed information, on what you’ll see. I’m going to give you my highlights on what I do when I review a USERENV debug log, and some of the more frequently relevant things to look at.

First of all, USERENV is a contraction of “user environment”, the creation of which is what this logging monitors. This does not mean that there is a service or process on your computer as you logon named “userenv” however. USERENV debug entries originate in the Winlogon.exe process instance that is instantiated when a user logs onto the computer. This may raise a question in your mind of “what if more than one user is logged onto the computer, like on a terminal server?” The answer is that each user will have their own instance of Winlogon.exe. All instances of Winlogon.exe, however, will spew their debug logging to one "universal" USERENV log on that computer.

So, all of the various users logon details in the above terminal server scenario will dump to the same USERENV debug log file (as described in the KB article on how to enable the logging, link above).

Let's assume you’ve enabled the debug logging on a supported system and you want to review your log to see what it says about you. Don’t get ahead of yourselves folks, it won’t tell your horoscope, just what happened during the user logon.

When I open the log I usually have the intention of looking for what happened to a specific user. It helps as a starting point to open the file in Notepad and choose EditàFind then search for that username. This should take you to a section that looks like below, essentially. Note that you could just as easily search for LoadUserProfile and to find your starting point, but you could end up looking at the wrong user’s logon section if you do that.

=========================================================

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: Entering, hToken = <0x73c>, lpProfileInfo = 0x6e3e0

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: lpProfileInfo->dwFlags = <0x0>

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: lpProfileInfo->lpUserName = <sombrerobob>

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: NULL central profile path

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: lpProfileInfo->lpDefaultPath = <\\hatsrus\netlogon\Default User>

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: NULL server name

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: In console winlogon process

USERENV(1f4.1f8) 15:19:56:468 In LoadUserProfileP

USERENV(1f4.1f8) 15:19:56:468 =========================================================

So now we have a time reference for when the user logon began (15:19:56:468) for user SombreroBob (the man who makes his own shade). This is good for the reason of having a starting point in reviewing the log, but it’s also useful for tracking slow user logon issues that take place after your user provides their credentials and presses enter, in addition to answering questions like “what policy processing is taking place?” in detail.

The next entries give us excellent information regarding locating a user profile for the user to logon with. Keep in mind that whether a user has a roaming user profile or not, and where it is located, is mandated by the user account properties in AD. Nice little tie-in from the Soliloquy, right?

The entries a few paragraphs above show SombreroBob logging on where he does not have a roaming profile specified so the logon will look for a prior local one or create a new profile for this iser from defaults located either in the netlogon share of the domain or on the computer being logged into. I’ve made the difference boldface text below.

USERENV(1f4.1f8) 15:19:56:468 =========================================================

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: Entering, hToken = <0x73c>, lpProfileInfo = 0x6e3e0

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: lpProfileInfo->dwFlags = <0x0>

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: lpProfileInfo->lpUserName = <sombrerobob>

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: lpProfileInfo->lpProfilePath = <\\hatrack\profiles\sombrerobob>

USERENV(1f4.1f8) 15:19:56:468 LoadUserProfile: lpProfileInfo->lpDefaultPath = <\\hatsrus-dc1\netlogon\Default User>

For instances of a user logon where a profile is already existing locally on the computer then the remote copy and the must be compared for the latest versions of the users profile files. For the remote profile case this will incur a good amount of SMB traffic on the wire as files are compared. The reason for this is quite good: consider that you have a terminal server farm and the user may never (or at least infrequently) logon to the same terminal server twice. Since by default there will be a local copy of that profile we can save some time, local computer resources and network usage by comparing that up-to-date roaming profile with the files that were left behind when the user logged off that terminal server last and then only copying over those files which need to be copied. In other words, the deltas or changes between what the user had at last logoff from that server and what has since changed.

The USERENV log gives a lot of entries from these comparison routines and you can quickly move to the appropriate section of the log if your needs mandate that by searching for ReconcileFile.

Another interesting technique that people use at times is to compare the directory path of the profile being loaded with the SID (security identifier) of the user. This can be interesting if you need to look in detail at what profile is being loaded versus what you have in the local computers registry regarding profile paths at HKLM\Software\Microsoft\Windows NT\CurrentVersion\ProfileList\SID.

Group policy processing at user logon (as well as at computer startup and GPO refresh intervals-unless you have Vista or 2008 which have an event log for that) will appear in this log as well. The basic rundown is that the list of the GPOs which are linked to the OU where the user’s object resides in AD is examined and assessed to see if the user has permissions to them. This will appear in the USERENV log similar to below and generates LDAP traffic on the wire from client computer the user is logging onto to the domain controller.

USERENV(1f4.1f8) 15:20:11:234 ProcessGPOs: Calling GetGPOInfo for normal policy mode

USERENV(1f4.1f8) 15:20:11:234 GetGPOInfo: ********************************

USERENV(1f4.1f8) 15:20:11:234 GetGPOInfo: Entering...

USERENV(1f4.1f8) 15:20:11:234 GetGPOInfo: Server connection established.

USERENV(1f4.1f8) 15:20:11:234 GetGPOInfo: Bound successfully.

USERENV(1f4.1f8) 15:20:11:234 SearchDSObject: Searching <OU=PremiumHatWearers,OU=Sales,OU=Users,DC=hatsrus,DC=com>

USERENV(1f4.1f8) 15:20:12:010 SearchDSObject: Found GPO(s): <[LDAP://cn={23F09E20-DFAB-441C-B656-6CB543A029CF},cn=policies,cn=system,DC=hatsrus,DC=com;0][LDAP://cn={2CBDE770-B492-424A-8AB0-60DF2CF25B7B},cn=policies,cn=system, DC=hatsrus,DC=com;0][LDAP://cn={E6A34850-7F6E-445G-97E2-95D0A44B86F9},cn=policies,cn=system,DC=hatsrus,DC=com;0][LDAP://cn={978EBEEF-EADE-4AF0-953D-CAFEC4B72B3B},cn=policies,cn=system,DC=hatsrus,DC=com;0][LDAP://CN={F0DF6A59-2D70-76YF-BDF9-F34D766FFA27},CN=Policies,CN=System,DC=hatsrus,DC=com;0][LDAP://cn={E6D33304-X49H-4285-B6F0-AAA7434D461E},cn=policies,cn=system,DC=hatsrus,DC=com;0][LDAP://cn={C109DD8D-8F46-4647-E335-935FB6A534E1},cn=policies,cn=system,DC=hatsrus,DC=com;0]>

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: ==============================

So now we have a list of GPOs which are linked to the OU where the user account resides in AD. Next we should see some entries which determine whether user has access to the GPO enough that he or she should be able to process and apply it, indicated by the “GPO passes the filter check” entry.

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: ==============================

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: Searching <CN={23F09E20-DFAB-441C-B656-6CB543A029CF},cn=policies,cn=system,DC=hatsrus,DC=com>

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: User has access to this GPO.

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: GPO passes the filter check.

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: Found functionality version of: 2

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: Found file system path of: <\\hatsrus.com\SysVol\hatsrus.com\Policies\{23F09E20-DFAB-441C-B656-6CB543A029CF}>

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: Found common name of: <{23F09E20-DFAB-441C-B656-6CB543A029CF}>

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: Found display name of: <Non-Propeller Hat Wearers Folder Redirection Policy>

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: Found user version of: GPC is 80, GPT is 80

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: Found flags of: 0

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: Found extensions: [{25537BA6-77A8-11D2-9B6C-0000F8080861}]

USERENV(1f4.1f8) 15:20:12:010 ProcessGPO: ==============================

So now we have the GPOs which will be processed for this user at logon and queued up (so to speak), ready to process.

From here it’s crucial to understand that GPO processing “where the rubber meets the road” is really all about client side extensions (CSEs). A CSE is the portion of our code that gets the setting to where it needs to be, and there are CSEs for different types of settings. For example, registry settings are applied by the Registry CSE. Scripts are processed by the Scripts CSE. I think you can all see where this type of thing is going.

In the USERENV there is a list of extensions which represents the types of settings the GPO specifies, listed in the Found Extensions portion above. Each of these GUIDs represents a CSE. In the above example, the CSE which this GPO has settings from is Folder Redirection. You can view a list of the CSEs and the GUIDs which relate to them (usable as a reference when seeing them in a USERENV log) by opening Regedit.exe and viewing the properties of the keys located here, listed by GUID: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\GPExtensions.

At a certain point in the USERENV log there’s a check done to see if these settings have been applied previously and how the CSE should behave if so. The general idea is that, for performance reasons, you may not want to apply policy settings at every policy processing if these settings have already been applied successfully. Don’t expect great detail here in the log, but do expect that the settings are things governed by looking to the registry on the client computer and deciding what to do based on what is there:

USERENV(1f4.1f8) 15:20:13:544 ReadExtStatus: Reading Previous Status for extension {25537BA6-77A8-11D2-9B6C-0000F8080861}

USERENV(1f4.1f8) 15:21:01:028 ReadStatus: Read Extension's Previous status successfully.

<snip>

USERENV(1f4.1f8) 15:21:01:028 ProcessGPOs: -----------------------

USERENV(1f4.1f8))15:21:01:028 ProcessGPOs: Processing extension Folder Redirection

USERENV(1f4.1f8) 15:21:01:143 CompareGPOLists: The lists are the same.

USERENV(1f4.1f8) 15:21:01:143 ProcessGPOList: Entering for extension Folder Redirection

USERENV(1f4.1f8) 15:21:01:143 UserPolicyCallback: Setting status UI to Applying Folder Redirection policy...

USERENV(1f4.1f8) 15:21:01:143 UserPolicyCallback: Setting status UI to Applying your personal settings...

USERENV(1f4.1f8) 15:21:01:143 ProcessGPOList: Extension Folder Redirection returned 0x0

A similar story can be told for each CSE as it processes. Is the USERENV the best place to look for details on how the CSEs do their job? Well, no, not really. Each CSE may have its own debug logging detailing how that CSE works and giving good info on problems if any are found. But we’re not going there in this post.

In this post we’ve gone over user logon and the user environment creation process, the intention being to provide an understanding of how user logon works. For Windows 2000, XP and 2003 clients this post can be used as a walkthrough for reviewing a USERENV log you create. For Vista and Server 2008, since USERENV logging is now a tracing format in those operating systems, you can use this post a reference point. In either event I hope you’ve gleaned some value in this post.

Finally, here’s a good Technet article we have that has more information:

How Core Group Policy Works

http://technet2.microsoft.com/windowsserver/en/library/eb0042e3-699b-4c49-abcc-e3526dbecc0e1033.mspx?mfr=true

Soliloquy for AD Logon

Tim Springston — Sat, 28 Jul 2007 07:15:00 GMT

Some readers may scoff a little when I talk about how under appreciated the whole Active Directory scheme is. Not schema but scheme. I’m talking about the entire client and server interaction and how they work together to provide all of the distributed services that make up Active Directory and user logon.

That’s a bold statement, you may say. Let me back it up with some general facts.

If you are going to have a secure business environment then you need to have a secure client computer (workstation) for your information worker to use. We have that in Windows Vista and earlier. These operating systems provide a logon mechanism to that workstation which can allow user identity verification by way of a network discussion with a centralized repository that keeps track of users and passwords and things like that. Without a validated logon in this manner no user shell is created, no Explorer will run, no access to resources in the environment can take place in an authenticated user’s context. Without things being done comprehensively in that context guaranteeing security for files, other objects and actions that can be done on that computer and on the network.

Of course that central repository is the Active Directory database. Think of this interaction as a symphony of various types of network traffic, each representing another part of the secure user logon experience our products provide which comes together as a harmonic to give to you the entire Microsoft Directory Services suite. Poetic, ain’t it?

So let’s go over a few of the types of network communication you’ll see when a domain computer starts up and a user logs on.

Finding a domain controller as the computer starts up is an important piece of things. If the domain client computer doesn’t know where to send its requests then nothing else will happen well. So Microsoft client computers, when they are joined to the domain, know that they can query the DNS zone which for that domain. Service, or SRV records, will answer DNS queries from the client computer with the IP address of a server that has LDAP, GC, Time and other services based on the closest site.

User logon (and the computer being authenticated at startup) with authentication of the user in a secure encrypted manner is provided by Kerberos network traffic from workstation to domain controller. You’ll see this in UDP and TCP traffic along port 88. Most parsers filter Kerberos easily (Wireshark.exe for example will identify all such traffic by typing Kerberos in the parsing line) and in a manner that shows not just the ticket requests and responses but also the various transport protocols which negotiate using Kerberos.

User logon with roaming profiles is basically comprised of a short query to a domain controller for location and then a whole lot of SMB traffic as the files are compared with anything that is already present on the client computer and then pulled down across the network. Depending on the size of the profile, and the use of folder redirection and client side caching, that can mean a lot of traffic.

Group Policy processing will first use LDAP queries to assess all of the policies which would apply to the user (or computer earlier on) based on where the user object is located in AD and what GPOs are linked to that object. This is followed by a client side assessment of whether the user has permissions to the GPO, and if it’s got the applicable type of settings (user or computer).

There will later be group policy traffic as the settings which the GPOs are giving are passed to the appropriate client side extensions for processing (scripts, folder redirection, software installation, registry, security, disk quota, EFS, Internet Explorer, and IP Security). These various CSEs, if they are going to process based on whether the GPOs say they should and if they haven’t already, will each use their own types of network traffic based on the remote resources they need to connect to in order to get their job done. Here’s a little not-so-secret: CSEs have their own logging to see what’s happening under the hood.

For Windows Server 2003, XP and 2000 you could see this on the client computer, with time indices, in the userenv log if you had it enabled. This was a pretty comprehensive way to see all that was happening on the client computer and when used in conjunction with a network trace you can really have a good understanding of what is happening on your client computer at computer startup or user logon. Let me confess, ahem, that Windows Vista and Server 2008 have this logging as well but it is not as easy for you, the customer, to gather useful data from it. That’s a topic for another day, but rest assured that Microsoft support is there for you to help glean data from it when needed.

Finally, let me end this diatribe by bringing some relevant links that may be useful to you when it comes to understanding and troubleshooting user logon…

How to enable user environment debug logging in retail builds of Windows

http://support.microsoft.com/kb/221833

Troubleshooting Group Policy Client-Side Extension Behavior

http://support.microsoft.com/kb/216358

And one of my favorite articles that is still largely applicable even in newer operating system releases:

Windows 2000 Startup and Logon Traffic Analysis

http://www.microsoft.com/technet/prodtechnol/windows2000serv/deploy/confeat/w2kstart.mspx

Vista, Client Side Caching, and Office...

Tim Springston — Fri, 16 Mar 2007 00:32:00 GMT

We have a lot of folks out there who use client side caching as a way to have consistent copies of the work they do as they travel and are not always well-connected to their corporate networks.

If you redirect a folder like My Documents, such as via group policy, having a copy of the user’s data on a share that is backed up, and the user having a local copy they can travel with on their laptop, is a pretty slick piece of technology.

There’s the added aspect that the user can make changes to the copy they are travelling with and be a more productive and industrious person. Then, when they have returned and are once again connected to their corporate network, their files that have changed will automatically synch with that remote share and pass along those updates.

There is one permutation of the above that several customers have seen lately that I wanted to pass along to you all out there. Hopefully it will arm you with the tool you need to address it as needed. Particularly if you are one of the many IT organizations that provides their executives with the folder redirection and client side caching technology so they can keep working on the road.

If you are using Word or Excel on a client side cached file and edit that file while you are in an offline state you may run into a situation where you updates are not synched as expected when you go online once again. So, alterations to the file made in that offline state appear to be lost.

The reason for this is that some Office applications need share permission access greater than Read and Change-they need Full Allow. Note that I’m talking about Sharing permissions-not NTFS permissions. From a Vista client, the code that reconciles how to deal with the updated TMP files seems to need a little more access to get the job done.

From a general diagnosis standpoint, other than the share permissions, you will see SMB traffic between client workstation and remote share that it is syncing witt. This traffic will include SMB sessions to the <filename>.DOC and the <GUID>.TMP file mentioned. A typical SMB error received for this will be STATUS_PRIVILEGE_NOT_HELD for the .DOC file, and a less cryptic STATUS_ACCESS_DENIED to the .TMP file.

The solution for this is to allow Full Allow Share permissions for the appropriate user(s) or a group they are a member of. This, of course, needs to be done on the remote share side of the folder redirection.

Generally speaking we don’t recommend that folks set sharing permissions for access to a share anyway. NTFS permissions are much more granular and configurable. There’s the added concern that NTFS permissions are a bit more reliable, in a long term manner, than the share permissions simply because they are stored on the volume file system as opposed to the registry.

As always folks just post to the blog if you have questions or comments, or if I can help out.