Blog - Title

June, 2009

  • DC’s and VM’s – Avoiding the Do-Over

    Hello everyone, Mark from DS again. With more and more companies using virtualization, such as Microsoft Virtual Server, Server 2008 Hyper-V or VMWare, in their environments these days you may end up in the following situation I recently worked on:

    1) Customer wanted to roll back one of his DC’s in his test environment to basically “back out” of some changes that had been made recently. This was a single domain forest that consisted on two Domain Controllers. Both of the DC’s were running Windows 2003 SP2.

    2) Virtual Machine snapshots were being taken instead of normal system state backups.

    3) They restored one of the DC’s from one of the snapshots.

    4) Replication was broken.

    Replication symptoms consisted of the following:

    1) The Netlogon service is in a paused state.

    2) In the Directory Service event log a replication error was logged, Source was NTDS Replication with the Event ID 2095.

    3) Also in the Directory Service event logs were two warnings, Source was NTDS General with the event ID’s 1113 and 1115.

    Here are samples of the Directory Service event log events with the description of the event.

    Event Type: Error
    Event Source: NTDS Replication
    Event Category: Replication
    Event ID: 2095
    Date:
    Time:
    User:
    Computer:
    Description: During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers. Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC. If not resolved immediately, this scenario will result in inconsistencies in the Active Directory databases of this source DC and one or more direct and transitive replication partners. Specifically the consistency of users, computers and trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data may vary, affecting the ability to log on, find objects of interest and perform other critical operations. To determine if this misconfiguration exists, query this event ID using
    http://support.microsoft.com or contact your Microsoft product support. The most probable cause of this situation is the improper restore of Active Directory on the local domain controller. User Actions: If this situation occurred because of an improper or unintended restore, forcibly demote the DC.

    Event Type: Warning
    Event Source: NTDS General
    Event Category: Replication
    Event ID: 1113
    Date:
    Time:
    User:
    Computer:
    Description: Inbound replication has been disabled by the user.
    Event Type: Warning
    Event Source: NTDS General
    Event Category: Replication
    Event ID: 1115
    Date:
    Time: 
    User: 
    Computer: 
    Description: Outbound replication has been disabled by the user.

    If you run the command repadmin /options <The DC Name> you can verify that inbound and outbound replication is disabled. You will see something similar to this:

    Current DC Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL

    With more and more companies using Virtualization to replace actual physical hardware, especially in test environments, I believe we are going see more issues such as this one. This can also happen in situations where you are converting physical hardware to virtual machines which we refer to as “PtoV” (physical to virtual).

    First we need to understand some basic background information regarding Active Directory (AD) replication. Domain Controllers (DC’s) use Update Sequence Numbers (USN’s) to track the updates that need to be replicated between replication partners. Every time a change in made to the data in the directory the USN is incremented to indicate a change was made. For each directory the DC stores, USN’s are used to track the latest updates that a DC has received from each source replication partner. Each DC also has a table where it knows about every other DC highest USN that stores a replica of that directory partition. Each DC also has a value on its NTDS Settings object called an invocation ID. This value is used to indentify its version of its local AD database.

    There are two values that use USN’s during the replication process. One is the up-to-dateness vector, the other is the high water mark. The up-to-dateness vector is a value that the destination DC maintains for tracking the originating updates that are received from its source DC’s. When the destination DC requests its updates for a directory partition it supplies its up-to-dateness to the source DC who can use that value to reduce the set of attributes it needs to send to the destination DC. The source DC will send its up-to-dateness vector value to the destination DC once the replication cycle has completed. The high water mark is a value that the destination DC maintains to keep track of the latest change it has received from a specific source DC for an object in a specific directory partition. This value prevents the source DC from sending out changes to the destination DC that have already been applied by the destination DC.

    The invocation ID is a GUID value that identifies the directory database running on a DC and is maintained separately from the identity of the server object. The server object identity never changes but the identity of the directory database (invocation ID) will change when a system state is restored by using the Microsoft API’s. All the domain controllers keep track of the directory database on its source replication partners. Both the up-to-dateness vector and the high water mark refer to the invocation ID so that other DC’s know which copy of the AD the replication is coming from.

    I know this can be confusing so let’s add some graphics that may help to understand this better. Let’s say we have two DC’s, DC1 and DC2. Both of these DC’s are running as Virtual Machines on a host machine running your favorite Virtualization Software. For all intents and purposes we are assuming that replication is working fine and both of the DC’s are up to date on replication. Before we start, we take a “snapshot” of DC1. As we can see below we add a new user “Jeff Smith” on DC1. The USN is incremented from 4710 to 4711 on DC1.

    image

    Now we replicate the new user to DC2. DC1 will notify DC2 that it has changes that it needs to replicate. DC2 will then request the changes and send DC1 what it thinks is DC1’s high water mark is. In this case DC2 thinks that value is 4710 so that is what it sends. When they are done replicating DC1 will send DC2 its up-to-dateness vector so DC2 will have the new value.

    image

    Now let’s suppose that other changes in the environment are occurring and replicating as they should. “Jeff” logs on and changes his password. When he does this DC2 is the DC where the change takes place. This will increment the USN on DC2 as it was 2452 and we increment the USN for DC2 to 2453.

    image

    Next we replicate that password change over to DC1. DC2 tells DC1 that it has changes it needs to get. DC1 will send DC2 what it thinks DC2’s USN is, in this case DC1 thinks DC2 is at 2452.

    image

    Once they are done replicating DC1 USN will be 5040 and DC2 will know it DC1 is at 5040. DC2 will be at 2453 and DC1 will know that value as well.

    Now you want to roll that one DC back. You apply the snapshot to the DC as a restore procedure. When this happens, the invocation ID remains the same, the USN’s are “rolled back” to the time the snapshot was taken. Now when the replication process starts the “snapshot” DC requests changes from its source DC it sends the old up-to-dateness vector to the source DC. The source DC sees this value and it knows what the value should be and they are different. The value sent has a lower value then the source DC has in its table for the destination DC. The response sent back to the destination DC by the source DC basically telling the destination DC its database is out of date. When this happens we have built-in protection so that the destination DC will take measures not replicate with other. This is referred to as a “USN rollback” situation.

    The protection that the USN rollback system will take will be is:

    1) Pause the Netlogon service.

    2) Disable the inbound and outbound replication.

    To correct this situation we need to do the following on the DC that has the roll back issue.

    1) Forcefully demote the DC by running dcpromo /forceremoval. This will remove AD from the server without attempting to replicate any changes off. Once it is done and you reboot the server and it will be a standalone serve in a workgroup.

    2) Run a metadata cleanup of the DC that was demoted per KB article 216498 on one of the replication partners.

    3) If the demoted server held any of the FSMO (Flexible Single Master Operations) roles then use the KB article 255504 to seize the roles to another DC.

    4) Once replication has occurred end to end in your environment you can rejoin the demoted server back to the domain then promote to a DC.

    To prevent this from happening adhere to the following best practices:

    1) Do not use imaging software to take an image of the DC.

    2) Do not take or apply snapshots of the DC.

    3) Do not shut the Virtual Machine down and simply copy the virtual disk as a backup.

    4) If you have the ability to “discard changes” as you do if you are running “Virtual Server 2005 R2”, do not enable this type of setting on a DC Virtual Machine.

    5) Use NTBACKUP.EXE, WBADMIN.EXE, or any third party software that is available as long as it is certified to be AD-compatible to take system state backups.

    6) Only restore a system state to the DC or restore a full backup.

    References:

    875495 How to detect and recover from a USN rollback in Windows Server 2003

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;875495

    Appendix A: Virtualized Domain Controllers and Replication Issues

    http://technet.microsoft.com/en-us/library/dd348479.aspx

    Backup and Restore Considerations for Virtualized Domain Controllers

    http://technet.microsoft.com/en-us/library/dd363545.aspx

     

    - Mark Ramey

  • Migrating from PolicyMaker to Group Policy Preferences with GPPMIG

    Mike here. PolicyMaker customers rejoice—Microsoft has a way for you to migrate from PolicyMaker 2.x to the new Group Policy preferences released with Windows Server 2008 and included in the Remote Server Administration Tools for Windows Vista Service Pack 1 or higher.

    Download GPPMIG: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=35791cb6-710b-48c4-aaa1-90db170bcf2a

    PolicyMaker to Preferences… how to get there

    If you’ve been using PolicyMaker then you already know how to use Group Policy Preferences. It is all managed using the Group Policy Management Console included with Windows Server 2008 or, using Windows Vista Service Pack1 by installing the Remote Server Administration Tools. However, Group Policy preferences cannot process PolicyMaker data and vice versa. Therefore, you need to have a strategy to migrate from PolicyMaker to Group Policy Preferences. Hopefully, this should help. Everything discussed below is also included in the GPPMIG installer as the ‘GPPMIG Migration Guide'.

    Prerequisites

    I want to take a few minutes to discuss some of the prerequisites before we jump right into the migration strategy. We have two categories Management and Client.

    Management

    Policymaker’s management looks and feels the same as managing other Group Policy setting. The same look and feel returns using Preferences. One thing to consider is-- each instance (PolicyMaker or Group Policy Preferences) cannot edit the others data. For this reason, you may need to leave one or more Windows XP computer, with the PolicyMaker administrative tools installed, until you’ve completed your migration. If your migration follows a staged approach, then you may encounter a small period of time where you may need to manage using both Windows Vista and Windows XP. Or, you may be the weekend warrior type and have your migration complete from Friday to Monday. The choice and freedom are there, but the requirement remains—PolicyMaker administrative additions can only edit PolicyMaker items. Server 2008 and the RSAT tools can only edit Preferences. Read Microsoft Knowledgebase article 941314, Description of Windows Server 2008 Remote Server Administration Tools for Windows Vista Service Pack 1 for more information.

    Client

    The critical component that actual makes PolicyMaker and/or Preferences work is the client side extensions (CSEs), which you must install on the client computer. The CSEs make normal Group Policy processing PolicyMaker/Preferences aware. The same rules apply to the client portion—PolicyMaker CSEs only process PolicyMaker data and Preference CSEs only process Preference data. Also, installing the Group Policy Preference CSEs automatically removes PolicyMaker CSEs. The new Group Policy preference client side extensions installs on

    • Windows Vista RTM and Service Pack 1
    • Windows Server 2003 Service Pack 1
    • Windows XP Service Pack 2

    Both Windows Server 2003 and Windows XP require the installation of XmlLite prior to installing the CSEs. Preference CSEs are included in Windows Server 2008. Read Microsoft Knowledgebase article 943729, Information about new Group Policy preferences in Windows Server 2008 for more information.

    Testing

    It goes without saying—you can never test enough and this scenario is not any different. Make sure you have backups… and they actually work. If you are going to use GPMC to backup your GPOs, then remember to use the correct version. GPMC backups are not interchangeable. If you backup with pre-Server 2008 GPMC, then you must restore with the same version. Back up some of your most complex or important GPOs and then important them into isolated test GPOs in a test OU with a single user and computer. Run through your entire migration strategy—noting what works and what does not— refining the plan with each pass. All efforts spent in planning usually pay off during implementation.

    Group Policy Preference Migration utility

    Now that we have the planning stuff out of the way—on to the good stuff. GPPMIG is a console application developed with version 3.0 of the .NET framework. Use GPPMIG to migrate PolicyMaker items to Group Policy Preference items into the same or a different Group Policy object. GPPMIG does not migrate PolicyMaker Application or Mail Profile data as Group Policy Preferences do not included client-side extensions for these items.

    What it does

    Let us take a few moments to discuss how GPPMIG works. For starters, GPPMIG always uses the domain of the currently logged on user. You’ll want to remember this so you can log on with domain administrator account for the domain GPOs you want to migrate. And, you must be a domain administrator as GPPMIG write to SYSVOL and Active Directory. One last point is that GPPMIG always connects to the PDC of the user domain—for reading and writing to Active Directory and SYSVOL. So, you’ll want to run GPPMIG from a computer close (same subnet) as the PDC emulator.

    With GPPMIG, you can target a single GPO to migrate or, you can choose to migrate all GPOs. GPPMIG performs a paged LDAP query to the PDC to retrieves a list of all the Group Policy objects in the user’s domain. GPPMIG then filters out any GPO in the list that is not configured for PolicyMaker items. Then, GPPMIG iterates through each GPO in the final list, looking for PolicyMaker specific client side extensions in each GPO. The entire GPO is evaluated before moving to the next. If a PolicyMaker setting is found, then GPPMIG ensures there is not an equivalent Group Policy Preference configuration, as it will not migrate PolicyMaker items into existing Group Policy Preference items. When GPPMIG completes its search for PolicyMaker items in the GPO, it then updates the Group Policy object to included Group Policy Preference client side extensions and then increases the version number for the user, computer, or both depending on what PolicyMaker items it migrated. In no way does a migration alter any PolicyMaker items for the GPO. All PolicyMaker items remain configured and available in the GPO. GPPMIG creates a migration log in the directory from which it ran.

    Usage

    You can use GPPMIG to migrate to Group Policy Preferences in staged approach or, you can create brand new GPOs to hold your new Group Policy Preference items and migrate to those new GPOs. The staged approach is a planned migration strategy and is the approach I’ll document here. After reading this, you should be able to alternate this strategy to best suit the needs of your environment. Generally, you’ll migrate from PolicyMaker to Group Policy Preferences in three stages (after you’ve done your testing).

    • Stage 1— Identify GPOs containing PolicyMaker items and use GPMC 1.x to back up those GPOs
    • Stage 2— Migrate PolicyMaker items to Group Policy Preference items in the same or a new Group Policy object. Then, deploy the Group Policy Preference CSEs to your client computers.
    • Stage 3— Confirm Group Policy Preference items migrated and are successfully applying to user and computers. Use GPMC 6.0.0.1 to backup your GPOs (to a different back up location then Stage 1. Then remove PolicyMaker items from GPOs, if applicable

    Download GPPMIG: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=35791cb6-710b-48c4-aaa1-90db170bcf2a

    Commands

    GPPMIG contains four basic commands:

    • Whatif — display all the Group Policy objects that contain PolicyMaker items
    • Migrate— migrates PolicyMaker items to Group Policy Preference items in the same GPO
    • MigrateTo— migrates PolicyMaker items to Group Policy Preference items to a different GPO
    • Remove— removes PolicyMaker items from a GPO

    Stage 1 – Identify PolicyMaker GPOs

    Begin your migration process by identifying GPOs containing PolicyMaker items. You can do this by using the –whatif command. Use the –all command afterwards to search all the GPOs in the user’s domain or, you can use the –name command and provided the display name of the GPO. Use GPMC to backup all of the GPOs identified to have PolicyMaker items.

    Stage 2 – Migrate PolicyMaker Data to Group Policy Preferences

    Next, you’ll want to migrate PolicyMaker items to Group Policy Preference items. You have a choice to migrate the setting within the same or to a different Group Policy object.

    clip_image001Note
    The migration does not modify PolicyMaker items, regardless of the migration action you choose.

    Use the –migrate command to migrate PolicyMaker items to Group Policy Preference items within the same GPO. Use the following syntax:

    Gppmig –migrate –name:gpo_name

    Alternatively, you replace the –name argument with –all to migrate all the GPOs in the users domain that contain PolicyMaker items.

    clip_image003

    Figure 1 GPPMIG In-place Migration

    You may prefer to keep PolicyMaker GPOs separate from Group Policy Preference GPOs. You use the –migrateTo command to accomplish this task

    clip_image004Important
    You must create the target GPO before using the -migrateTo command. GPPMIG does create the target Group Policy object.

    clip_image006

    Figure 2 - GPPMIG Source-target Migration

    The –migrateTo command requires two additional arguments: -source: and –target: follows by the display name of the Group Policy object. Enclose the name of the GPO in quotes if the name contains spaces. Also, the –migrateTo command does not support the –all argument.

    Deploy GPP Client

    You’re now ready to deploy the Group Policy Preference client-side extensions after you’ve migrated all of your GPOs to include Group Policy Preference items. The migration does not modify any PolicyMaker items; so clients with the PolicyMaker CSE and the Group Policy preference CSEs process the same data

    clip_image001[1]Note
    GPPMIG does not migrate Application or Mail PolicyMaker items. Therefore, Group Policy Preference CSEs do not apply these items to users or computers. Leave the PolicyMaker CSE installed on computers that require these items and do not install the Group Policy Preference CSEs as the installation removes PolicyMaker CSEs).

    You can apply Group Policy Preferences to several Microsoft operating systems. The minimum operating system requirements are:

    • Windows Vista RTM or Windows Vista Service Pack 1 (32 or 64-bit)
    • Windows Server 2003 Service Pack 1 or later (32 or 64-bit)
    • Windows XP Service Pack 2 or later (32 or 64-bit)

    Group Policy Preference client-side extensions are included in Windows Server 2008. You can use the links above to download the client-side extension installation packages. Or, you can download the extensions as an optional update from Windows Update.

    clip_image004[1]Important
    Remember-- installing Preference client-side extensions removes PolicyMaker Client Side Extensions.

    Stage 3

    The last stage in the migration process involves verifying your items migrated and apply correctly. Use GPMC to view the Group Policy object to which you migrated your items. Click the Settings tab to show the Preference items included in the GPO.

    clip_image008

    Figure 3 GPMC Settings Tab

    Next, you'll want to apply the Group Policy object to your client computers. For in-place migrations, you'll want to apply the GPO to computers using PolicyMaker CSEs and computers using Preference CSEs. Also verify user PolicyMaker and Preference items apply to the appropriate user. GPOs that are targets of in-place migrations should apply items to both (PolicyMaker and Preferences). Source-target migrations migrate the PolicyMaker items to Preference items in the newly created GPO. This allows you to keep your existing PolicyMaker GPOs separate from your Preference GPOs. You apply GPOs containing Preference items to computers are users using the Group Policy Preference CSEs.

    Use the Resultant Set of Policy (RSOP) management console to confirm PolicyMaker items are applying to computers or users. Use the Group Policy Results feature within GPMC to confirm Preference items are applying to computers or users.

    clip_image010

    Figure 4 GPMC Group Policy Results for testuser

    The actual migration from PolicyMaker to Group Policy Preferences is complete. Computers running either Preferences or PolicyMaker should be applying their respective items. Source-target migrations contain both PolicyMaker and Preference items. After you've transitioned your client to use the Group Policy Preference CSEs, you'll want to remove the PolicyMaker data, which remains in the GPO. You can use GPPMIG with the -remove option to remove overlapping PolicyMaker and Preference items.

    clip_image012

    Figure 5 Removing PolicyMaker settings

    clip_image001[2]Note
    GPPMIG does not remove PolicyMaker Application and Mail items from the Group Policy object.

    Source-target Migrations do not included PolicyMaker items. Therefore, once you've completed transitioning your client computers to use Preference CSEs, you can delete the source version of the GPO, which contains only PolicyMaker items.

    Conclusion

    You should consider backing up your Group Policy objects after you've completed your migration and cleanup of Group Policy objects. Use the Group Policy Management Console included in Windows Server 2008 and the Remote Server Administration tools to backup all of your Group Policy objects before you proceed with any further changes.

    - Mike Stephens

  • Potential for Kerberos Issues When Using a Cisco VPN/ASA with Win2003 or later DC’s

    Hey everyone, Rob Greene here back after a long hiatus from blogging. I had an interesting case come through that I thought many of you IT pros would be interested in.

    Background

    The customer had an issue with using Cisco VPN and Cisco ASA concentrators and authenticating the user with Kerberos authentication. After they upgraded all their Windows 2000 domain controllers to Windows Server 2003 domain controllers, users could not authenticate through the VPN.

    After doing this they gave Cisco a call and they were told to start DSA.MSC and check the box under Account Options to “Do not require Kerberos preauthentication” for all user accounts that need to connect through the appliances. When they did this as a test it did allow the user to start authenticating through the devices. However the customer did not like this option since:

    1. This change lowers the security of Kerberos authentication by disabling password replay attack protection.

    2. Everything had worked previously with Windows 2000.

    If you want to learn more about preauthentication you can click here and look under the section:

    Example AS Administration.

    If you search Cisco’s website you will find that they have older documentation stating that you must disable this feature on the user’s account before the VPN will start working.

    I don’t have real network traces (as I can’t show you a real customer’s network!), but I can tell you what we saw in the traces that led us to a potential workaround.

    VPN Appliance --> UDP Port 88 / KRB5 AS-REQ --> Windows DC

    VPN Appliance <-- UDP Port 88 / KRB5KDC_ERR_REAUTH_REQUIRED <--Windows DC

    VPN Appliance --> UDP Port 88 / AS-REQ (with Pre-Auth Data) --> Windows DC

    VPN Appliance <-- UDP Port 88 / KRB5KDC_ERR_RESPONSE_TOO_BIG <-- Windows DC

    A KRB5KDC_ERR_RESPONSE_TOO_BIG means that the requested Kerberos data was larger than the maximum default datagram size of 1465 bytes. The next step here usually would be for the client to start the Kerberos ticketing process over using TCP rather than UDP. If this was a Windows XP client instead of the Cisco VPN appliance we would have the customer implement the registry key change outlined in the below KB article.

    244474 How to force Kerberos to use TCP instead of UDP in Windows
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;244474

    So basically the real issue is that the Cisco appliance is being asked by the KDC to use TCP instead of UDP to do the AS-REQ with pre-authentication data present. However, the Cisco appliance does not continue at this point. I spent the past few weeks working with Cisco directly (big shoutout - they have been great!), and they have published the following new articles:

    Cisco VPN 3000: CSCed27444 - VPN3000 does not switch to TCP upon receiving Kerberos error code 52

    Cisco ASA platform: CSCsi32224 - ASA does not switch to TCP upon receiving Kerberos error code 52

    CSCtd92673 -- "Kerberos authentication fails with pre-auth enabled"

    Unfortunately, the Cisco VPN 3000 concentrator has been discontinued so there is no update available for the Cisco VPN 3000 to add this functionality. Also, you will need a support contract from Cisco to logon to those two articles and read them.

    Please contact Cisco directly to find out more information about the issue, and with what IOS version it will be included in; we have provided this information to make it easier to find the content.

    In the Meantime

    So now the question becomes this: If that is what is going on, then how did it work when a Windows 2000 domain controller was used? And what can you do about it until an IOS update is released?

    There was a change in the default datagram reply packet size used by the Kerberos Key Distribution Center (KDC) between Windows 2000 and Windows Server 2003. In Windows 2000 the default size is 2000 bytes, where it is 1465 bytes in Windows Server 2003. For those interested you can review the below KB article and find the registry key of MaxDatagramReplySize.

    837361 Kerberos protocol registry entries and KDC configuration keys in Windows Server 2003
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;837361

    So that means this customer’s Kerberos ticket size for the user accounts is less than 2000 bytes, but more than 1465 bytes. Keep in mind that if you are reading this blog, your environment may still not work with the registry key setting at 2000. This is because the Kerberos ticket size is going to vary based on the number of groups that the user belongs to. You can review the KB below to learn more:

    327825 New resolution for problems with Kerberos authentication when users belong to many groups
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;327825

    The only workaround that we can give a customer is to modify the MaxDatagramReplySize registry value to the size of your user’s largest Kerberos ticket size. It really depends on the group membership size of the user attempting to authenticate because of the PAC data that is included in the TGT. For those customers that have used the MaxTokenSize registry value should equal that value and restart the KDC service. If none of these acronyms are making sense you might want to check out my older blog post Kerberos for the Busy Admin.

    Hey wow, this sounds like a pretty easy fix right? Well, you can actually have other problems because of this change. If your networking gear (switches, routers, firewalls) cannot handle large UDP packets without fragmenting them then all Kerberos authentication could start to fail in your environment. See KB244474 under more information to learn more about this.

    Please keep in mind what you are doing when you implement the MaxDatagramReplySize registry value. You are in fact telling the KDC to never request the client send KRB5KDC_ERR_RESPONSE_TOO_BIG message back to a Kerberos client, and you should consider implementing the custom group policy setting included in KB244474 on all domain-joined machines to use TCP for Kerberos authentication always. If you are running purely Vista/2008 or later though, this is on by default.

    Lastly, if you have to set this value very high, and you know that your networking gear will not support the packet sizes that had to be implemented, you may have to put a domain controller on the same IP subnet as the Cisco appliance to successfully get Kerberos tickets.

    - Rob ‘I mean, you know’ Greene

  • Ghosts in the DFS Management Machine

    Ned here again. Today’s post is a quick technique I use to show how to slap the DFS Management snap-in around when it acts foolishly. Hopefully you find it helpful someday.

    Here’s the situation

    • There are multiple administrators at your company managing DFSR or DFSN.
    • Everyone (naturally) uses the DFS Management snap-in from their own workstation, or from various servers.
    • Someone deletes a large number of DFSR Replication Groups or DFSN roots – maybe they are recreating a new topology, working in the lab, or whatever.

    So let’s say I had this:

    image

    Then I start up DFSMGMT.MSC and now I have this:

    image

    No big deal, I’ll just click the refresh button. Wait, where’s the refresh button? No matter, I’ll just press F5, that always refreshes stuff… crud, still there. Ok, I’ll just press CTRL+A and select everything… you have got to be kidding me! Are you saying that I have to right-click every single object node and choose ‘Remove Replication Group from Display’?

    image

    Not anymore.


    A little background

    The DFS UI development team writes the DFS Management snap-in for optimal performance. They achieve performance gains by caching new expensive data from LDAP. This optimization prevents the management tool from over utilizing the network with unnecessary queries. The snap-in recognizes when information is new and remembers it. However, the behavior changes when we delete information. The concept of deleting setting in masses in large environments was overlooked.


    Never mind that, fix it!

    So where’s this cache? Right here:

    %appdata%\Microsoft\MMC

    Open the path listed above. This path expands to a folder within your profile, like this:

    C:\Users\nedpyle\AppData\Roaming\Microsoft\MMC

    You'll see a file named ‘dfsmgmt’ without a file extension.

    Close the DFS Management snap. Then delete or rename the dfsmgmt file. Start the snap-in. Voila. All the extraneous deleted goo, which was removed by someone else, is now gone. You can use the same file and technique to remove large numbers of DFS Roots. We’re looking into getting a darned refresh button added too. ;-)

    I told you it was quick.

    - Ned ‘Hammer Approach’ Pyle

  • Internet Explorer behaviors with Kerberos Authentication

    Hey Rob here again, I thought that I would share with you some of the things that we see where Internet Explorer Kerberos authentication fails.

    It is important to understand the default behavior of Internet Explorer and its support for Kerberos authentication so that you don’t start ripping out your hair (can’t speak to what Ned does here). I have listed three very common problems that we typically see when Kerberos authentication is failing with web-based applications.

    Scenario 1: Website does not use the standard TCP/IP port of (80/443)

    Web Server Configuration:
    • Webserver1 has two different IIS sites running on it.
    • Website1 runs with a web application pool account assigned to NetworkService.
    • Website2 runs with a web application pool account assigned to a domain user account.
    • The website2 is configured to listen on Port 8080.
    • The following SPN’s have been defined on the website2 application pool account.
      • http/webserver1.contoso.com:8080
      • http/webserver1:8080

    In this scenario you can see why a non-standard port is being used since multiple websites have been configured on the same web server. When this happens you need to specify the port to be used when you add the Service Principal Name, otherwise there is going to be a high likely hood that you will get a Kerberos ticket for the wrong web application pool account.

    Client Workaround

    In this scenario you need to make sure that when Internet Explorer accesses Website2 that it asks for a Service Principal name with the port number defined. However, the default behavior of IE is to not add the port number to the Kerberos ticket request. When this ticket is presented to IIS you will see a KRB_AP_ERR_MODIFIED message back.

    You will need to use the below KB article change the default behavior on all IE client versions. For Internet Explorer 6 it will require the QFE Brach of Wininet.dll to be installed before the registry change will actually work.

    908209 Internet Explorer 6 cannot use the Kerberos authentication protocol to connect to a Web site that uses a non-standard port in Windows XP and in Windows Server 2003 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;908209

    I can tell you that there is not a version of this KB article for IE7 and above, but you do have to make the same registry change for these versions of IE also.

    Server Workaround:

    There really is not a good workaround to the issue other than to use host headers for one of the websites and adding a DNS HOST record for the host header in your DNS configuration. You will see shortly why we are not recommending a CNAME record in DNS.

    Scenario 2: CNAME DNS RR is used for a website.

    Web Server Configuration:
    • Webserver1 has two different sites running on it.
    • Website1 runs with a web application pool account assigned to NetworkService.
    • Website2 runs with a web application pool account assigned to a domain user account.
    • Website2 is configured to us a host header of app1.contoso.com.
    • In DNS a CNAME record for app1.contoso.com was created and pointed to webserver1.contoso.com HOST record.
    • The following SPN’s have been defined on the website2 application pool account.
      • http/app1.contoso.com
      • http/app1

    In this scenario it appears that this should work just fine. When a user goes to app1.contoso.com the client machine is going to do a DNS lookup, and the DNS server is going to respond with the CNAME record and point to the webserver1.contoso.com HOST record. We can also see that the Service Principal Name configuration is properly configured on the web application pool account for website2.

    The default behavior of Internet Explorer is to generate the Kerberos ticket request for the HOST record that is returned from a CNAME record, not the actual CNAME record itself. So IE specifically asks for a Kerberos ticket for http/webserver1.contoso.com which will result in a Kerberos ticket being generated encrypted with the WebServer1 computer’s password. This will in turn generate a KRB_AP_ERR_MODIFIED from IIS back to IE when the user attempts to visit the app1.contoso.com website.

    Client Workaround

    You will need to use the KB articles below to change the default behavior on all IE versions. For IE 6 it will require the QFE Brach of Wininet.dll to be installed before the registry key change will actually work.

    For Internet Explorer 6:

    911149 Error message in Internet Explorer when you try to access a Web site that requires Kerberos authentication on a Windows XP-based computer: "HTTP Error 401 - Unauthorized: Access is denied due to invalid credentials" - http://support.microsoft.com/default.aspx?scid=kb;EN-US;911149

    For Internet Explorer 7 and above:

    938305 Error message when you try to log on to a Web site that requires Kerberos authentication by using Internet Explorer 7: "Access is denied due to invalid credentials" - http://support.microsoft.com/default.aspx?scid=kb;EN-US;938305

    Server Workaround:

    The only work around is to remove the DNS CNAME RR and replace it with a HOST RR.

    Scenario 3: Website works on first access but starts failing 30 minutes later

    Web Server Configuration:
    • Computer Webserver1 one site on it.
    • Website1 runs with a web application pool account assigned to a domain user account.
    • The following SPN’s have been defined on the website1 application pool account.
      • http/webserver1.contoso.com

    When a user visits the website they use the NETBIOS computer name for the URL to visit. For example: http://webserver1.

    In this scenario there does not seem to be much wrong here, except that there is only the FQDN version of the Service Principal Name defined. Yeah, I know all of our documentation around Kerberos always states to add FQDN as well as NETBIOS name versions of the SPN. Believe it or not, we see this all the time where the user did not register both of them, but stick with me here.

    The default behavior of Internet Explorer is to add on the computer’s DNS suffix or use DNS suffix search order if defined on the machine to whatever the user types in the URL if it is not a dotted name. If your DNS configuration is correct it will resolve to webserver1.contoso.com. Once IE finds this name it stores the DNS entry in its own DNS cache. Just like most caches it times out - for IE’s cache it is 30 minutes. After 30 minutes IE again has to resolve the name, however the next time it does not try to resolve the name through DNS again, it tries just NetBIOS name resolution (hopefully there is WINS in the environment; otherwise it will just fail). Based on your configuration you could expect one of the following Kerberos errors:

    • KRB_AP_ERR_MODIFIED – Expect to get this error if web site name is the same name as your web server’s computer name. That is because it is going to ask for an http/webserver1 SPN and will resolve to HOST/webserver1 which is assigned to the computer account.
    • KRB_ERR_S_PRINCIPAL_UNKNOWN – Expect to get this error if the web site name is something like app1.contoso.com. That is because it is going to ask for an http/app1 SPN and will not resolve to any account in the domain.
    Client Workaround:

    Thankfully there is a fix that can be implemented for Internet Explorer.

    899417 You may receive an "Access is denied" error message you use the WWW-Authenticate: Negotiate method of HTTP authentication to connect to a Web server - http://support.microsoft.com/default.aspx?scid=kb;EN-US;899417

    I will tell you that there is not a version of this KB article for IE7 and above, but you do have to make the registry key change for these versions of IE also before the functionality is supported.

    Server Workaround:

    You can register the NetBIOS version of the Service Principal Name to the account, using SETSPN.EXE.

    I hope that you found this post interesting. As always it is easier to spot these type of issues by reviewing network trace taken at the client side (where IE is being used) to find the root cause of the issue.

    - Rob “I Speak Tampa” Greene

  • Debunking the Vista Remote Differential Compression Myth

    Ned here again. Have you ever visited Snopes.com? It’s a terrific urban legend reference where they research folklore. Snopes is the place you go to find out if eating Thanksgiving turkey makes you sleepy (it doesn’t), if Coca Cola can dissolve a tooth overnight (it can’t), or if a man really did live in a Paris airport for 8 years (he did!).

    Today I’m going to talk about another urban legend – that removing the Remote Differential Compression feature from Windows Vista will make your file copying faster over the network.

    Background on RDC

    Remote Differential Compression (RDC) is a Microsoft algorithm that was originally created for DFSR five years ago. RDC divides a file’s data into chunks by using signatures. When a file exists on two computers and the file is modified, only the differing chunks need to be sent to the other computer.

    image

    An application needs to be specifically written to support RDC. Windows Vista and Windows 7 include MSRDC.DLL to allow apps like Windows Live Messenger to use that functionality.

    image

    The feature can be turned on and off within the Control Panel “Program and Features” applet.

    image

    When turned on, the MSRDC.DLL will exist in the %SYSTEMROOT%\System32 directory. When it’s turned off, this DLL is removed.

    image

    The Myth

    Unfortunately, the Internet is full of people telling you that RDC will somehow make your network communication slower. I have no idea how this got started, but this nonsense has been reprinted on thousands of websites by people unfamiliar with the Scientific Method. Folks have actually convinced themselves that turning this feature on or off has some affect on file transfer speeds. While there are a variety of things you can do to speed up Vista file copy performance, this isn’t one of them.

    The Method

    So after hearing this baloney for the umpteenth time, I set out to debunk it once and for all:

    • I setup a Windows Server 2008 SP2 file server and a Windows Vista SP2 client as virtual guests inside a Hyper-V host.
    • Each guest got 1GB of RAM and a virtual gigabit NIC.
    • I dropped a 3GB ISO file onto a share on the file server.
    • I used robocopy.exe (which automatically reports bytes per second and megabytes per minute) to copy that ISO file five times with MSRDC.DLL installed, then five more times with MSRDC.DLL removed.
    • In between every copy, I rebooted both the client and server to ensure that there was no caching of the file that might artificially improve my results.

    The Results

    Here is what I found after the ten total passes, with and without RDC installed:

    Pass

     

    With MSRDC.DLL

     

    Without MSRDC.DLL

     

    1st pass

     

    1237.939

     

    1210.479

     

    2nd pass

     

    1186.415

     

    1330.882

     

    3rd pass

     

    1192.068

     

    1175.328

     

    4th pass

     

    1111.13

     

    1170.281

     

    5th pass

     

    1320.867

     

    1153.863

     

     

    image

    Hmmm… What if I sort my data points highest to lowest?

    image

    Interesting. Let’s look at the actual averages:

    With MSRDC.DLL installed: 1209.6838 MB/min

    With MSRDC.DLL removed: 1208.1666 MB/min

    Wait – so removing RDC actually made it slower? Not really – the variance there is well within any respectable margin of error. These results mean that the two sets of copies were 99.87% identical. Removing RDC did nothing at all. There are going to be various performance differences when copying a file, depending on what else happens on the network, what the computers are doing, and I think when people claim RDC removal made things ‘faster’, it’s because they are not testing repeatedly over time to see variance.

    Maybe you want more proof? Alright, let’s go to the debugger.

    • I installed Notmyfault on my Vista client and configured the computer for a complete memory dump
    • I started yet another file copy, then I listed out the process and service information with TASKLIST.EXE and TASKLIST.EXE /SVC
    • After a few seconds, I intentionally crash dumped (the so-called “Blue Screen of Death”) the Vista computer. I then cracked open the MEMORY.DMP file in the Windows debugger

    First, I listed loaded modules - all libraries and drivers loaded in the kernel memory space:

    kd> lm
    start             end                 module name
    00000000`75f50000 00000000`75f88000   odbcint    (deferred)            
    00000000`77aa0000 00000000`77b6d000   USER32     (deferred)            
    00000000`77b70000 00000000`77c9d000   kernel32   (deferred)            
    00000000`77ca0000 00000000`77e26000   ntdll      (export symbols)       ntdll.dll
    00000000`77e50000 00000000`77e54000   Normaliz   (deferred)            
    00000000`ffec0000 00000000`fff41000   Robocopy   (deferred)            
    000007fe`f4be0000 000007fe`f4d30000   MFC42u     (deferred)            
    000007fe`f9200000 000007fe`f9271000   ODBC32     (deferred)            
    000007fe`fbd90000 000007fe`fbe30000   COMCTL32   (deferred)            
    000007fe`fcc80000 000007fe`fce79000   comctl32_7fefcc80000   (deferred)            
    000007fe`fe420000 000007fe`fe4ac000   COMDLG32   (deferred)            
    000007fe`fe620000 000007fe`fe64d000   IMM32      (deferred)            
    000007fe`fe6f0000 000007fe`fe8c8000   ole32      (deferred)            
    000007fe`fe8d0000 000007fe`fe92f000   iertutil   (deferred)            
    000007fe`fe930000 000007fe`fe937000   NSI        (deferred)            
    000007fe`feb80000 000007fe`fecc3000   RPCRT4     (deferred)            
    000007fe`fecd0000 000007fe`fed6a000   USP10      (deferred)            
    000007fe`fed70000 000007fe`fedb4000   WS2_32     (deferred)            
    000007fe`fedc0000 000007fe`fee33000   SHLWAPI    (deferred)            
    000007fe`fee40000 000007fe`feedc000   msvcrt     (deferred)            
    000007fe`feee0000 000007fe`fefe2000   MSCTF      (deferred)            
    000007fe`feff0000 000007fe`feffd000   LPK        (deferred)            
    000007fe`ff000000 000007fe`ffc53000   SHELL32    (deferred)            
    000007fe`ffc60000 000007fe`ffd5e000   WININET    (deferred)            
    000007fe`ffd60000 000007fe`ffdc4000   GDI32      (deferred)            
    000007fe`ffdd0000 000007fe`ffea3000   OLEAUT32   (deferred)            
    000007fe`ffeb0000 000007fe`fffb8000   ADVAPI32   (deferred)            
    fffff800`01804000 fffff800`01d1c000   nt         (private)
    fffff800`01d1c000 fffff800`01d62000   hal        (deferred)            
    fffff960`000c0000 fffff960`00371000   win32k     (deferred)            
    fffff960`00480000 fffff960`0049e000   dxg        (deferred)            
    fffff960`00600000 fffff960`0060a000   TSDDD      (deferred)            
    fffff960`00820000 fffff960`0082b000   VMBusVideoD   (deferred)            
    fffffa60`00602000 fffffa60`0060c000   kdcom      (deferred)            
    fffffa60`0060c000 fffffa60`00647000   mcupdate_GenuineIntel   (deferred)            
    fffffa60`00647000 fffffa60`0065b000   PSHED      (deferred)            
    fffffa60`0065b000 fffffa60`006b8000   CLFS       (deferred)            
    fffffa60`006b8000 fffffa60`0076a000   CI         (deferred)            
    fffffa60`0076a000 fffffa60`007d0000   volmgrx    (deferred)            
    fffffa60`007d0000 fffffa60`007e4000   NDProxy    (deferred)            
    fffffa60`007e4000 fffffa60`007ef000   Msfs       (deferred)            
    fffffa60`007ef000 fffffa60`00800000   Npfs       (deferred)            
    fffffa60`00808000 fffffa60`008e2000   Wdf01000   (deferred)            
    fffffa60`008e2000 fffffa60`008f0000   WDFLDR     (deferred)            
    fffffa60`008f0000 fffffa60`00946000   acpi       (deferred)            
    fffffa60`00946000 fffffa60`0094f000   WMILIB     (deferred)            
    fffffa60`0094f000 fffffa60`00959000   msisadrv   (deferred)            
    fffffa60`00959000 fffffa60`00989000   pci        (deferred)            
    fffffa60`00989000 fffffa60`0099e000   partmgr    (deferred)            
    fffffa60`0099e000 fffffa60`009b2000   volmgr     (deferred)            
    fffffa60`009b2000 fffffa60`009ba000   intelide   (deferred)            
    fffffa60`009ba000 fffffa60`009ca000   PCIIDEX    (deferred)            
    fffffa60`009ca000 fffffa60`009fd000   netvsc60   (deferred)            
    fffffa60`00a00000 fffffa60`00a3d000   vmbus      (deferred)            
    fffffa60`00a3d000 fffffa60`00a51000   winhv      (deferred)            
    fffffa60`00a51000 fffffa60`00a64000   mountmgr   (deferred)            
    fffffa60`00a64000 fffffa60`00a6c000   atapi      (deferred)            
    fffffa60`00a6c000 fffffa60`00a90000   ataport    (deferred)            
    fffffa60`00a90000 fffffa60`00ad7000   fltmgr     (deferred)            
    fffffa60`00ad7000 fffffa60`00aeb000   fileinfo   (deferred)            
    fffffa60`00aeb000 fffffa60`00af8000   storvsc    (deferred)            
    fffffa60`00af8000 fffffa60`00b55000   storport   (deferred)            
    fffffa60`00b55000 fffffa60`00bdb000   ksecdd     (deferred)            
    fffffa60`00bdb000 fffffa60`00bee000   intelppm   (deferred)            
    fffffa60`00bee000 fffffa60`00bf7000   rdpencdd   (deferred)            
    fffffa60`00bf7000 fffffa60`00c00000   rasacd     (deferred)            
    fffffa60`00c00000 fffffa60`00c0e000   vga        (deferred)            
    fffffa60`00c0f000 fffffa60`00dd2000   ndis       (deferred)            
    fffffa60`00dd2000 fffffa60`00dee000   cdrom      (deferred)            
    fffffa60`00dee000 fffffa60`00df7000   Null       (deferred)            
    fffffa60`00df7000 fffffa60`00e00000   RDPCDD     (deferred)            
    fffffa60`00e00000 fffffa60`00e0a000   Fs_Rec     (deferred)            
    fffffa60`00e0c000 fffffa60`00e5c000   msrpc      (deferred)            
    fffffa60`00e5c000 fffffa60`00eb5000   NETIO      (deferred)            
    fffffa60`00eb5000 fffffa60`00ede000   fvevol     (deferred)            
    fffffa60`00ede000 fffffa60`00f0a000   CLASSPNP   (deferred)            
    fffffa60`00f29000 fffffa60`00f35000   tunnel     (deferred)            
    fffffa60`00f35000 fffffa60`00f4b000   i8042prt   (deferred)            
    fffffa60`00f4b000 fffffa60`00f59000   kbdclass   (deferred)            
    fffffa60`00f59000 fffffa60`00f65000   mouclass   (deferred)            
    fffffa60`00f65000 fffffa60`00f82000   serial     (deferred)            
    fffffa60`00f82000 fffffa60`00f8e000   serenum    (deferred)            
    fffffa60`00f8e000 fffffa60`00f9b000   fdc        (deferred)            
    fffffa60`00f9b000 fffffa60`00fad000   HIDCLASS   (deferred)            
    fffffa60`00fad000 fffffa60`00fb7000   VMBusVideoM   (deferred)            
    fffffa60`00fb7000 fffffa60`00fdc000   VIDEOPRT   (deferred)            
    fffffa60`00fdc000 fffffa60`00fec000   watchdog   (deferred)            
    fffffa60`00fec000 fffffa60`00ff5000   vms3cap    (deferred)            
    fffffa60`00ff5000 fffffa60`01000000   mouhid     (deferred)            
    fffffa60`01000000 fffffa60`01007b80   HIDPARSE   (deferred)            
    fffffa60`01008000 fffffa60`0117d000   tcpip      (deferred)            
    fffffa60`0117d000 fffffa60`011a9000   fwpkclnt   (deferred)            
    fffffa60`011a9000 fffffa60`011b9000   vmstorfl   (deferred)            
    fffffa60`011b9000 fffffa60`011e5000   ecache     (deferred)            
    fffffa60`011e5000 fffffa60`011ef000   crcdisk    (deferred)            
    fffffa60`01208000 fffffa60`01388000   Ntfs       (deferred)            
    fffffa60`01388000 fffffa60`013cc000   volsnap    (deferred)            
    fffffa60`013cc000 fffffa60`013d4000   spldr      (deferred)            
    fffffa60`013d4000 fffffa60`013e6000   mup        (deferred)            
    fffffa60`013e6000 fffffa60`013fa000   disk       (deferred)            
    fffffa60`013fa000 fffffa60`013ff500   VMBusHID   (deferred)            
    fffffa60`02200000 fffffa60`0220b000   flpydisk   (deferred)            
    fffffa60`0220f000 fffffa60`02248000   msiscsi    (deferred)            
    fffffa60`02248000 fffffa60`02255000   TDI        (deferred)            
    fffffa60`02255000 fffffa60`02278000   rasl2tp    (deferred)            
    fffffa60`02278000 fffffa60`02284000   ndistapi   (deferred)            
    fffffa60`02284000 fffffa60`022b5000   ndiswan    (deferred)            
    fffffa60`022b5000 fffffa60`022c5000   raspppoe   (deferred)            
    fffffa60`022c5000 fffffa60`022e3000   raspptp    (deferred)            
    fffffa60`022e3000 fffffa60`022fb000   rassstp    (deferred)            
    fffffa60`022fb000 fffffa60`02395000   rdpdr      (deferred)            
    fffffa60`02395000 fffffa60`023a8000   termdd     (deferred)            
    fffffa60`023a8000 fffffa60`023a9480   swenum     (deferred)            
    fffffa60`023aa000 fffffa60`023de000   ks         (deferred)            
    fffffa60`023de000 fffffa60`023e9000   mssmbios   (deferred)            
    fffffa60`023e9000 fffffa60`023f9000   umbus      (deferred)            
    fffffa60`02401000 fffffa60`0241e000   tdx        (deferred)            
    fffffa60`0241e000 fffffa60`02439000   smb        (deferred)            
    fffffa60`02439000 fffffa60`024a4000   afd        (deferred)            
    fffffa60`024a4000 fffffa60`024e8000   netbt      (deferred)            
    fffffa60`024e8000 fffffa60`02506000   pacer      (deferred)            
    fffffa60`02506000 fffffa60`02514000   nm3        (deferred)            
    fffffa60`02514000 fffffa60`02523000   netbios    (deferred)            
    fffffa60`02523000 fffffa60`0253e000   wanarp     (deferred)            
    fffffa60`0253e000 fffffa60`0258b000   rdbss      (deferred)            
    fffffa60`0258b000 fffffa60`02597000   nsiproxy   (deferred)            
    fffffa60`02597000 fffffa60`025c0000   srvnet     (deferred)            
    fffffa60`025c0000 fffffa60`025de000   bowser     (deferred)            
    fffffa60`02c08000 fffffa60`02c7e000   csc        (deferred)            
    fffffa60`02c7e000 fffffa60`02c9b000   dfsc       (deferred)            
    fffffa60`02c9b000 fffffa60`02cb7000   cdfs       (deferred)            
    fffffa60`02cb7000 fffffa60`02cc5000   crashdmp   (deferred)            
    fffffa60`02cc5000 fffffa60`02cd1000   dump_dumpata   (deferred)            
    fffffa60`02cd1000 fffffa60`02cd9000   dump_atapi   (deferred)            
    fffffa60`02cd9000 fffffa60`02cec000   dump_dumpfve   (deferred)            
    fffffa60`02cec000 fffffa60`02cf8000   Dxapi      (deferred)            
    fffffa60`02cf8000 fffffa60`02d1a000   luafv      (deferred)            
    fffffa60`02d1a000 fffffa60`02d2e000   lltdio     (deferred)            
    fffffa60`02d2e000 fffffa60`02d46000   rspndr     (deferred)            
    fffffa60`02d46000 fffffa60`02de5000   HTTP       (deferred)            
    fffffa60`02de5000 fffffa60`02dff000   mpsdrv     (deferred)            
    fffffa60`03805000 fffffa60`0382c000   mrxdav     (deferred)            
    fffffa60`0382c000 fffffa60`03855000   mrxsmb     (deferred)            
    fffffa60`03855000 fffffa60`0389e000   mrxsmb10   (deferred)            
    fffffa60`0389e000 fffffa60`038bd000   mrxsmb20   (deferred)            
    fffffa60`038bd000 fffffa60`038ef000   srv2       (deferred)            
    fffffa60`038ef000 fffffa60`03980000   srv        (deferred)            
    fffffa60`03c03000 fffffa60`03c9d000   spsys      (deferred)            
    fffffa60`03c9d000 fffffa60`03d53000   peauth     (deferred)            
    fffffa60`03d53000 fffffa60`03d5e000   secdrv     (deferred)            
    fffffa60`03d5e000 fffffa60`03d6e000   tcpipreg   (deferred)            
    fffffa60`03d6e000 fffffa60`03d75000   myfault    (deferred)            

    Note how MSRDC.DLL is not loaded in memory in the Kernel space. It’s still possible that a given process or service might have it loaded though, so then I listed all processes to see which ones would be interesting and likely to be involved in file copies. The TASKLIST output comes in handy here to see which PID is which hexadecimal CID value. In my case though I dumped them all just for exploratory purposes.

    kd> !process 0 0

    <snipped out some>

    PROCESS fffffa8009af9040
        SessionId: 1  Cid: 0790    Peb: 7fffffdf000  ParentCid: 0a08
        DirBase: 34e57000  ObjectTable: fffff880066f60d0  HandleCount:  65.
        Image: Robocopy.exe

    PROCESS fffffa800b9adc10
        SessionId: 0  Cid: 0340    Peb: 7fffffd5000  ParentCid: 0280
        DirBase: 1efc3000  ObjectTable: fffff8800634fb60  HandleCount: 522.
        Image: svchost.exe

     

     

     

     

    I know that the Workstation Service is responsible for SMB file copying, and the robocopy process is definitely doing work, so I examined those.

    kd> .process fffffa8009af9040
    Implicit process is now fffffa80`09af9040
    kd> !peb
    PEB at 000007fffffdf000
        InheritedAddressSpace:    No
        ReadImageFileExecOptions: No
        BeingDebugged:            No
        ImageBaseAddress:         00000000ffec0000
        Ldr                       0000000077db2960
        Ldr.Initialized:          Yes
        Ldr.InInitializationOrderModuleList: 00000000001226c0 . 00000000001373c0
        Ldr.InLoadOrderModuleList:           00000000001225d0 . 00000000001373a0
        Ldr.InMemoryOrderModuleList:         00000000001225e0 . 00000000001373b0
                Base TimeStamp                     Module
            ffec0000 479191ad Jan 19 00:59:09 2008 C:\Windows\system32\Robocopy.exe
            77ca0000 49e0421d Apr 11 03:09:17 2009 C:\Windows\system32\ntdll.dll
            77b70000 49e041d1 Apr 11 03:08:01 2009 C:\Windows\system32\kernel32.dll
         7feffeb0000 49e040cb Apr 11 03:03:39 2009 C:\Windows\system32\ADVAPI32.dll
         7fefeb80000 49e041ea Apr 11 03:08:26 2009 C:\Windows\system32\RPCRT4.dll
         7fef4be0000 49e04151 Apr 11 03:05:53 2009 C:\Windows\system32\MFC42u.dll
         7fefee40000 49e04189 Apr 11 03:06:49 2009 C:\Windows\system32\msvcrt.dll
            77aa0000 49e0420e Apr 11 03:09:02 2009 C:\Windows\system32\USER32.dll
         7feffd60000 49e04114 Apr 11 03:04:52 2009 C:\Windows\system32\GDI32.dll
         7fefe6f0000 49e041cf Apr 11 03:07:59 2009 C:\Windows\system32\ole32.dll
         7feffdd0000 49e041d2 Apr 11 03:08:02 2009 C:\Windows\system32\OLEAUT32.dll
         7feffc60000 49e04252 Apr 11 03:10:10 2009 C:\Windows\system32\WININET.dll
         7fefedc0000 49e041f4 Apr 11 03:08:36 2009 C:\Windows\system32\SHLWAPI.dll
            77e50000 4549b4d2 Nov 02 05:05:22 2006 C:\Windows\system32\Normaliz.dll
         7fefe8d0000 49e04146 Apr 11 03:05:42 2009 C:\Windows\system32\iertutil.dll
         7fefed70000 49e0422d Apr 11 03:09:33 2009 C:\Windows\system32\WS2_32.dll
         7fefe930000 4791adea Jan 19 02:59:38 2008 C:\Windows\system32\NSI.dll
         7fef9200000 49e041c1 Apr 11 03:07:45 2009 C:\Windows\system32\ODBC32.dll
         7fefbd90000 4791ac7c Jan 19 02:53:32 2008 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.6001.18000_none_40ba501d3c2b20ff\COMCTL32.dll
         7feff000000 49e041ef Apr 11 03:08:31 2009 C:\Windows\system32\SHELL32.dll
         7fefe420000 49e041e9 Apr 11 03:08:25 2009 C:\Windows\system32\COMDLG32.dll
         7fefe620000 49e0417d Apr 11 03:06:37 2009 C:\Windows\system32\IMM32.DLL
         7fefeee0000 49e04184 Apr 11 03:06:44 2009 C:\Windows\system32\MSCTF.dll
         7fefeff0000 4791ad25 Jan 19 02:56:21 2008 C:\Windows\system32\LPK.DLL
         7fefecd0000 49e04211 Apr 11 03:09:05 2009 C:\Windows\system32\USP10.dll
         7fefcc80000 49e041e9 Apr 11 03:08:25 2009 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18005_none_1509f8bef40ee4da\comctl32.dll
            75f50000 4549d310 Nov 02 07:14:24 2006 C:\Windows\system32\odbcint.dll

    kd> .PROCESS fffffa800b9adc10
    Implicit process is now fffffa80`0b9adc10
    kd> !peb
    PEB at 000007fffffd5000
        InheritedAddressSpace:    No
        ReadImageFileExecOptions: No
        BeingDebugged:            No
        ImageBaseAddress:         00000000ff820000
        Ldr                       0000000077db2960
        Ldr.Initialized:          Yes
        Ldr.InInitializationOrderModuleList: 00000000002125f0 . 0000000003a08ac0
        Ldr.InLoadOrderModuleList:           0000000000212500 . 0000000003a08aa0
        Ldr.InMemoryOrderModuleList:         0000000000212510 . 0000000003a08ab0
                Base TimeStamp                     Module
            ff820000 47919291 Jan 19 01:02:57 2008 C:\Windows\system32\svchost.exe
            77ca0000 49e0421d Apr 11 03:09:17 2009 C:\Windows\system32\ntdll.dll
            77b70000 49e041d1 Apr 11 03:08:01 2009 C:\Windows\system32\kernel32.dll
         7fefee40000 49e04189 Apr 11 03:06:49 2009 C:\Windows\system32\msvcrt.dll
         7feffeb0000 49e040cb Apr 11 03:03:39 2009 C:\Windows\system32\ADVAPI32.dll
         7fefeb80000 49e041ea Apr 11 03:08:26 2009 C:\Windows\system32\RPCRT4.dll
         7fefd440000 49e0422f Apr 11 03:09:35 2009 C:\Windows\system32\NTMARTA.DLL
            77aa0000 49e0420e Apr 11 03:09:02 2009 C:\Windows\system32\USER32.dll
         7feffd60000 49e04114 Apr 11 03:04:52 2009 C:\Windows\system32\GDI32.dll
         7fefe940000 49e0427e Apr 11 03:10:54 2009 C:\Windows\system32\WLDAP32.dll
         7fefed70000 49e0422d Apr 11 03:09:33 2009 C:\Windows\system32\WS2_32.dll
         7fefe930000 4791adea Jan 19 02:59:38 2008 C:\Windows\system32\NSI.dll
            77e40000 47919b74 Jan 19 01:40:52 2008 C:\Windows\system32\PSAPI.DLL
         7fefdce0000 49e041e3 Apr 11 03:08:19 2009 C:\Windows\system32\SAMLIB.dll
         7fefe6f0000 49e041cf Apr 11 03:07:59 2009 C:\Windows\system32\ole32.dll
         7fefe620000 49e0417d Apr 11 03:06:37 2009 C:\Windows\system32\IMM32.DLL
         7fefeee0000 49e04184 Apr 11 03:06:44 2009 C:\Windows\system32\MSCTF.dll
         7fefeff0000 4791ad25 Jan 19 02:56:21 2008 C:\Windows\system32\LPK.DLL
         7fefecd0000 49e04211 Apr 11 03:09:05 2009 C:\Windows\system32\USP10.dll
         7fefc1c0000 49e0419d Apr 11 03:07:09 2009 c:\windows\system32\es.dll
         7feffdd0000 49e041d2 Apr 11 03:08:02 2009 C:\Windows\system32\OLEAUT32.dll
         7fefc000000 49e041dd Apr 11 03:08:13 2009 c:\windows\system32\PROPSYS.dll
         7fefd510000 49e041ed Apr 11 03:08:29 2009 C:\Windows\system32\rsaenh.dll
         7fefe650000 4791acc9 Jan 19 02:54:49 2008 C:\Windows\system32\CLBCatQ.DLL
         7fefc4b0000 4791adeb Jan 19 02:59:39 2008 c:\windows\system32\nsisvc.dll
         7fefe250000 49e04210 Apr 11 03:09:04 2009 C:\Windows\system32\secur32.dll
         7fefdb10000 49e04202 Apr 11 03:08:50 2009 C:\Windows\system32\CRYPT32.dll
         7fefdcc0000 4791ad5c Jan 19 02:57:16 2008 C:\Windows\system32\MSASN1.dll
         7fefe270000 49e04210 Apr 11 03:09:04 2009 C:\Windows\system32\USERENV.dll
         7fefd8f0000 4791adc3 Jan 19 02:58:59 2008 C:\Windows\system32\credssp.dll
         7fefd4b0000 49e041f1 Apr 11 03:08:33 2009 C:\Windows\system32\schannel.dll
         7fefdfc0000 49e041a5 Apr 11 03:07:17 2009 C:\Windows\system32\NETAPI32.dll
         7fefbc50000 49e04225 Apr 11 03:09:25 2009 c:\windows\system32\webclnt.dll
         7fefba50000 49e04251 Apr 11 03:10:09 2009 c:\windows\system32\WINHTTP.dll
         7fefedc0000 49e041f4 Apr 11 03:08:36 2009 C:\Windows\system32\SHLWAPI.dll
         7fefe4b0000 49e04209 Apr 11 03:08:57 2009 C:\Windows\system32\urlmon.dll
         7fefe8d0000 49e04146 Apr 11 03:05:42 2009 C:\Windows\system32\iertutil.dll
         7fefcc80000 49e041e9 Apr 11 03:08:25 2009 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18005_none_1509f8bef40ee4da\comctl32.dll
         7feff000000 49e041ef Apr 11 03:08:31 2009 C:\Windows\system32\shell32.dll
         7feffc60000 49e04252 Apr 11 03:10:10 2009 C:\Windows\system32\WinInet.dll
            77e50000 4549b4d2 Nov 02 05:05:22 2006 C:\Windows\system32\Normaliz.dll
         7fefbc10000 4791ae1c Jan 19 03:00:28 2008 c:\windows\system32\wkssvc.dll
         7fefda40000 49e04193 Apr 11 03:06:59 2009 c:\windows\system32\IPHLPAPI.DLL
         7fefd9f0000 49e040f3 Apr 11 03:04:19 2009 c:\windows\system32\dhcpcsvc.DLL
         7fefdd00000 49e04119 Apr 11 03:04:57 2009 c:\windows\system32\DNSAPI.dll
         7fefd9e0000 4791ae08 Jan 19 03:00:08 2008 c:\windows\system32\WINNSI.DLL
         7fefd9b0000 49e040f4 Apr 11 03:04:20 2009 c:\windows\system32\dhcpcsvc6.DLL
         7fefdc90000 4791adef Jan 19 02:59:43 2008 c:\windows\system32\NTDSAPI.dll
         7fefd5a0000 4791adf5 Jan 19 02:59:49 2008 c:\windows\system32\WINBRAND.dll
         7fefb7d0000 4549d27e Nov 02 07:11:58 2006 c:\windows\system32\fdrespub.dll
         7fefb500000 49e0423a Apr 11 03:09:46 2009 c:\windows\system32\wsdapi.dll
         7fefb810000 4791ad11 Jan 19 02:56:01 2008 c:\windows\system32\HTTPAPI.dll
         7fefd3a0000 4791ae1a Jan 19 03:00:26 2008 c:\windows\system32\WINTRUST.dll
         7fefe400000 4791ad46 Jan 19 02:56:54 2008 C:\Windows\system32\imagehlp.dll
         7fefcfc0000 4791addb Jan 19 02:59:23 2008 c:\windows\system32\XmlLite.dll
         7fefd290000 4791ace8 Jan 19 02:55:20 2008 c:\windows\system32\FirewallAPI.dll
         7fefd820000 49e04210 Apr 11 03:09:04 2009 c:\windows\system32\VERSION.dll
         7fefb280000 49e0411b Apr 11 03:04:59 2009 C:\Windows\system32\FunDisc.dll
         7fefc840000 4791ac8a Jan 19 02:53:46 2008 C:\Windows\system32\ATL.DLL
         7fefe9a0000 49e041ed Apr 11 03:08:29 2009 C:\Windows\system32\SETUPAPI.dll
         7fefd790000 49e0418f Apr 11 03:06:55 2009 C:\Windows\system32\mswsock.dll
         7fefd400000 4791aeae Jan 19 03:02:54 2008 C:\Windows\System32\wshtcpip.dll
         7fefd810000 4791aea8 Jan 19 03:02:48 2008 C:\Windows\System32\wship6.dll
         7fefacc0000 49e04191 Apr 11 03:06:57 2009 C:\Windows\System32\msxml3.dll
         7fefb140000 4791ae0e Jan 19 03:00:14 2008 c:\windows\system32\ssdpsrv.dll
         7fefafa0000 49e0420a Apr 11 03:08:58 2009 c:\windows\system32\w32time.dll
         7fefdd40000 4791adc8 Jan 19 02:59:04 2008 c:\windows\system32\cryptdll.dll
         7fefd3e0000 49e04118 Apr 11 03:04:56 2009 C:\Windows\system32\GPAPI.dll
         7fefdae0000 49e041da Apr 11 03:08:10 2009 C:\Windows\system32\slc.dll
         7fefb100000 49ee93d7 Apr 21 23:49:43 2009 C:\Windows\System32\vmictimeprovider.dll
         7fefa300000 4791ad84 Jan 19 02:57:56 2008 c:\windows\system32\netprofm.dll
         7fefc900000 4791ad8c Jan 19 02:58:04 2008 c:\windows\system32\nlaapi.dll
         7fefa2a0000 4791adbc Jan 19 02:58:52 2008 c:\windows\system32\upnphost.dll
         7fefb3f0000 4549d324 Nov 02 07:14:44 2006 c:\windows\system32\SSDPAPI.dll
         7fefaf90000 4549d36c Nov 02 07:15:56 2006 C:\Windows\System32\npmproxy.dll
         7fefe070000 4791adb4 Jan 19 02:58:44 2008 C:\Windows\system32\SXS.DLL
         7fefce90000 4791acf3 Jan 19 02:55:31 2008 c:\windows\system32\fdphost.dll
         7fef69c0000 49e04124 Apr 11 03:05:08 2009 C:\Windows\system32\fdwsd.dll
         7fef6980000 4791ad25 Jan 19 02:56:21 2008 C:\Windows\system32\MLANG.dll
         7fef6960000 49e04121 Apr 11 03:05:05 2009 C:\Windows\system32\fdssdp.dll
         7fefd020000 49e0411f Apr 11 03:05:03 2009 C:\Windows\system32\fdproxy.dll
         7fefb840000 4791ad5c Jan 19 02:57:16 2008 C:\Windows\system32\napinsp.dll
         7fef97d0000 4791adb8 Jan 19 02:58:48 2008 C:\Windows\system32\pnrpnsp.dll
         7fefb860000 4791ae09 Jan 19 03:00:09 2008 C:\Windows\System32\winrnr.dll
         7fef9660000 4791ad9a Jan 19 02:58:18 2008 C:\Windows\system32\rasadhlp.dll

     

    Note how neither process has MSRDC.DLL loaded either. It’s simply not being used, and a module that is not being used cannot possibly affect anyone. Remember, an application has to be coded to use RDC. Nothing in the Kernel, in Robocopy, or in the Workstation service uses RDC at all in Vista or Win7.

    Still don’t believe me? Here is the Microsoft Remote File Systems development team stating it as well.

    Changes that can truly improve file copy performance

    By now you want me to get to the helpful part. Here’s a short list of some things that can improve your file copy network performance on Windows Vista and Windows Server 2008:

    • Install Service Pack 2
    • Install the latest NIC drivers from your vendor.
    • Try disabling Receive Side Scaling, Chimney Offload, and NetDMA support, then testing like I did above to see if the results are measurably different after many copies of the same file. Note that this just disables the Windows implementation of those components – your vendor may also support them through their NIC configuration and it will need to be turned off there as well. While these components are intended to help performance, mileage can vary based on how good your hardware and vendor drivers are.
    • Use robocopy.exe rather than Explorer – the price of the friendly shell showing progress and browsing folders is slightly slower performance.

    Wrapup

    It’s amazing that a component that was designed to speed up network file performance can somehow be vilified as a cause of bad performance; especially when it’s not even being used. I welcome people following my steps and telling me what you find out.

    Don’t believe everything you read on the Internet. Unless I wrote it. :-)

    - Ned ‘Rick Rolled’ Pyle

  • RSAT and ADUC for Vista - Update to add tabs for Terminal Services Profile, Environment, Sessions, and Remote Control

    Ned here. After much strife, here is the hotfix to get RSAT AD Users and Computers to include tabs for:

    • Terminal Services Profile 
    • Environment 
    • Sessions 
    • Remote Control 

    960890 Some tabs are not available in the properties of a user account in the Active Directory Users and Computers MMC snap-in after you install Remote Server Administration Tools (RSAT) on a computer that is running Windows Vista
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;960890

    If you have been hacking away at this previously to make it work, make sure to unregister and remove your server DLL's before installing this update.

     Linda Taylor rules.

    - Ned Pyle

  • Implementing an OCSP responder: Part III - Configuring OCSP for use with Enterprise CAs

    Chris here again. As promised I will be covering configuring an OCSP Responder to support Enterprise CA. I will also be covering validating your OCSP Configuration.

    Installing OCSP Responder Role

    The first step is to install the OCSP Responder Role.

    To install the OCSP Responder: Open a command prompt and type: servermanagercmd.exe –install ADCS-Online-Cert.

    Configuring the OCSP Responder

    First we will add a Revocation Configuration to the OCSP Responder.

    Right click on the Revocation Configuration and select Add Revocation Configuration from the context menu.

    image

    The Add Revocation Configuration wizard opens. Click Next to continue.

    image

    Give a Friendly Name to the Revocation Configuration, and click Next. It is a good idea to include the name of the CA for which you are setting up this Revocation Configuration, especially if this OCSP Responder will handle requests for multiple CAs.

    image

    On the Select CA Certificate page, you will need to select a CA certificate. This is where you determine the CA for which you will be providing revocation information.

    Select a certificate for an Existing enterprise CA, and click Next

    image

    Select Browse CA certificates published in Active Directory, and click Browse.

    image

    Select the appropriate CA, and click OK

    image

    Next you will need to select a certificate that will be used for signing OCSP responses. For a particular Revocation Configuration, the OCSP Signing certificate must be issued by the CA for which the OCSP Responder will answer revocation status requests.

    Select Automatically select a signing certificate. If you wish to automatically enroll for the OCSP Response Signing Certificate, make sure the Auto-Enroll for an OCSP signing certificate is checked. Select the certificate template that you configured for use with the OCSP Responder, then click Next.

    image

    On the Revocation Provider page, you can click Provider to select revocation providers. The Windows Server 2008 OCSP Responder can only use CRLs for revocation information. If you have the CDP Extension available in the signing certificate, the Revocation Providers will be populated from the information in the CDP Extension from the OCSP Response Signing Certificate.

    image

    You can add the repository locations for your CRLs and Delta CRLs if appropriate. By default these will be populated from information included in the CDP extension of the Signing certificate. After you have reviewed the configuration or made any changes, click OK.

    image

    That completes the initial Configuration of the OCSP Responder. If you would like to modify the configuration of the OCSP Responder, you can right click on the Revocation Configuration and select Properties from the context menu.

    image

    The Local CRL tab allows you to configure a Local CRL. You can add revocation information for certificates which you wish to consider revoked. It is recommended that you do not use this option, as it adds unnecessary complexity to the revocation configuration.

    image

    The Revocation Provider tab allows you to modify the location of the CRLs and Delta CRLs that will be used for providing revocation information.

    image

    Signing Tab

    In the signing tab you can:

    • Modify the hash algorithm used to sign responses.
    • Do not prompt for credentials for cryptographic operations. This setting may need to be disabled if you are using an HSM to protect the private key of the OCSP Signing certificate. Disabling this setting allows you to be prompted for the password that is associated with the operator card on the HSM.
    • Use renewed certificates for signing certificates. This option is enabled by default, when you use the OCSP Responder with an Enterprise CA and automatically renew certificates. If you use OCSP Responder with a standalone CA, the OCSP responder will use renewed signing certificates even if this setting is not enabled.
    • Enable NONCE extension support allows the user to attach the NONCE sent in the request with the OCSP response. If this setting is used, you will not be able to utilize cached responses.
    • Use any valid OCSP signing certificate. Not recommended if the OCSP Responder is supporting Vista clients since they do not support this option. This allows the OCSP responder to use any certificate that the OCSP Signing configured in the Extended Key Usage extension of the certificate. Vista clients will only accept OCSP responses that are signed by the same CA for which the OCSP Responder is providing revocation information.
    • All responses will included the following Online Responder identifies: This setting determines whether a Key Hash or Subject will be included in the response. RFC 2560 specifies the structure of the response. In section 4.2.1 of the RFC it is specified that the Responder ID field can either be populated with a Name or Key hash. This setting determines which is included in the response. The Key hash is a hash of the OCSP Responder’s public key. The Name is the distinguished name of the subject of the OCSP signing certificate.

    image

    Verify OCSP Configuration

    After configuring the OCSP Responder, you will want to verify that the OCSP responder is functioning properly. The easiest way to verify that the OCSP is functioning is to use the Certutil URL Retrieval tool.

    First request a certificate from the CA. Place a copy of that cert on the file system, and run the following command: certutil –URL <Certificate Name>. This will open the URL Retrieval Tool

    image

    Select OCSP, and click on the Retrieve button.

    image

    If the certificate is valid you will get the following response.

    image

    If the certificate is revoked, you will get the following response.

    image

    And if it fails, the status will be listed as Failed.

    image

    You can also use the PKIView tool to verify the configurations of the OCSP Responder.

    image

    Conclusion

    This concludes configuring an OCSP Responder to support an Enterprise CA. If you follow the steps listed here you now have your OCSP configured to support your Windows Server 2003 or Windows Server 2008 CA. In the next part of this series, I will be configuring an OCSP Responder to support Standalone CA.

    Implementing an OCSP responder: Part I Introducing OCSP
    Implementing an OCSP responder: Part II Preparing Certificate Authorities
    Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
    Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 
    Implementing an OCSP Responder: Part V High Availability
    Implementing an OCSP Responder: Part VI Configuring Custom OCSP URIs via Group Policy

    - Chris Delay