Blog - Title

June, 2009

  • DC’s and VM’s – Avoiding the Do-Over

    Hello everyone, Mark from DS again. With more and more companies using virtualization, such as Microsoft Virtual Server, Server 2008 Hyper-V or VMWare, in their environments these days you may end up in the following situation I recently worked on:

    1) Customer wanted to roll back one of his DC’s in his test environment to basically “back out” of some changes that had been made recently. This was a single domain forest that consisted on two Domain Controllers. Both of the DC’s were running Windows 2003 SP2.

    2) Virtual Machine snapshots were being taken instead of normal system state backups.

    3) They restored one of the DC’s from one of the snapshots.

    4) Replication was broken.

    Replication symptoms consisted of the following:

    1) The Netlogon service is in a paused state.

    2) In the Directory Service event log a replication error was logged, Source was NTDS Replication with the Event ID 2095.

    3) Also in the Directory Service event logs were two warnings, Source was NTDS General with the event ID’s 1113 and 1115.

    Here are samples of the Directory Service event log events with the description of the event.

    Event Type: Error
    Event Source: NTDS Replication
    Event Category: Replication
    Event ID: 2095
    Date:
    Time:
    User:
    Computer:
    Description: During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers. Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC. If not resolved immediately, this scenario will result in inconsistencies in the Active Directory databases of this source DC and one or more direct and transitive replication partners. Specifically the consistency of users, computers and trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data may vary, affecting the ability to log on, find objects of interest and perform other critical operations. To determine if this misconfiguration exists, query this event ID using
    http://support.microsoft.com or contact your Microsoft product support. The most probable cause of this situation is the improper restore of Active Directory on the local domain controller. User Actions: If this situation occurred because of an improper or unintended restore, forcibly demote the DC.

    Event Type: Warning
    Event Source: NTDS General
    Event Category: Replication
    Event ID: 1113
    Date:
    Time:
    User:
    Computer:
    Description: Inbound replication has been disabled by the user.
    Event Type: Warning
    Event Source: NTDS General
    Event Category: Replication
    Event ID: 1115
    Date:
    Time: 
    User: 
    Computer: 
    Description: Outbound replication has been disabled by the user.

    If you run the command repadmin /options <The DC Name> you can verify that inbound and outbound replication is disabled. You will see something similar to this:

    Current DC Options: IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL

    With more and more companies using Virtualization to replace actual physical hardware, especially in test environments, I believe we are going see more issues such as this one. This can also happen in situations where you are converting physical hardware to virtual machines which we refer to as “PtoV” (physical to virtual).

    First we need to understand some basic background information regarding Active Directory (AD) replication. Domain Controllers (DC’s) use Update Sequence Numbers (USN’s) to track the updates that need to be replicated between replication partners. Every time a change in made to the data in the directory the USN is incremented to indicate a change was made. For each directory the DC stores, USN’s are used to track the latest updates that a DC has received from each source replication partner. Each DC also has a table where it knows about every other DC highest USN that stores a replica of that directory partition. Each DC also has a value on its NTDS Settings object called an invocation ID. This value is used to indentify its version of its local AD database.

    There are two values that use USN’s during the replication process. One is the up-to-dateness vector, the other is the high water mark. The up-to-dateness vector is a value that the destination DC maintains for tracking the originating updates that are received from its source DC’s. When the destination DC requests its updates for a directory partition it supplies its up-to-dateness to the source DC who can use that value to reduce the set of attributes it needs to send to the destination DC. The source DC will send its up-to-dateness vector value to the destination DC once the replication cycle has completed. The high water mark is a value that the destination DC maintains to keep track of the latest change it has received from a specific source DC for an object in a specific directory partition. This value prevents the source DC from sending out changes to the destination DC that have already been applied by the destination DC.

    The invocation ID is a GUID value that identifies the directory database running on a DC and is maintained separately from the identity of the server object. The server object identity never changes but the identity of the directory database (invocation ID) will change when a system state is restored by using the Microsoft API’s. All the domain controllers keep track of the directory database on its source replication partners. Both the up-to-dateness vector and the high water mark refer to the invocation ID so that other DC’s know which copy of the AD the replication is coming from.

    I know this can be confusing so let’s add some graphics that may help to understand this better. Let’s say we have two DC’s, DC1 and DC2. Both of these DC’s are running as Virtual Machines on a host machine running your favorite Virtualization Software. For all intents and purposes we are assuming that replication is working fine and both of the DC’s are up to date on replication. Before we start, we take a “snapshot” of DC1. As we can see below we add a new user “Jeff Smith” on DC1. The USN is incremented from 4710 to 4711 on DC1.

    image

    Now we replicate the new user to DC2. DC1 will notify DC2 that it has changes that it needs to replicate. DC2 will then request the changes and send DC1 what it thinks is DC1’s high water mark is. In this case DC2 thinks that value is 4710 so that is what it sends. When they are done replicating DC1 will send DC2 its up-to-dateness vector so DC2 will have the new value.

    image

    Now let’s suppose that other changes in the environment are occurring and replicating as they should. “Jeff” logs on and changes his password. When he does this DC2 is the DC where the change takes place. This will increment the USN on DC2 as it was 2452 and we increment the USN for DC2 to 2453.

    image

    Next we replicate that password change over to DC1. DC2 tells DC1 that it has changes it needs to get. DC1 will send DC2 what it thinks DC2’s USN is, in this case DC1 thinks DC2 is at 2452.

    image

    Once they are done replicating DC1 USN will be 5040 and DC2 will know it DC1 is at 5040. DC2 will be at 2453 and DC1 will know that value as well.

    Now you want to roll that one DC back. You apply the snapshot to the DC as a restore procedure. When this happens, the invocation ID remains the same, the USN’s are “rolled back” to the time the snapshot was taken. Now when the replication process starts the “snapshot” DC requests changes from its source DC it sends the old up-to-dateness vector to the source DC. The source DC sees this value and it knows what the value should be and they are different. The value sent has a lower value then the source DC has in its table for the destination DC. The response sent back to the destination DC by the source DC basically telling the destination DC its database is out of date. When this happens we have built-in protection so that the destination DC will take measures not replicate with other. This is referred to as a “USN rollback” situation.

    The protection that the USN rollback system will take will be is:

    1) Pause the Netlogon service.

    2) Disable the inbound and outbound replication.

    To correct this situation we need to do the following on the DC that has the roll back issue.

    1) Forcefully demote the DC by running dcpromo /forceremoval. This will remove AD from the server without attempting to replicate any changes off. Once it is done and you reboot the server and it will be a standalone serve in a workgroup.

    2) Run a metadata cleanup of the DC that was demoted per KB article 216498 on one of the replication partners.

    3) If the demoted server held any of the FSMO (Flexible Single Master Operations) roles then use the KB article 255504 to seize the roles to another DC.

    4) Once replication has occurred end to end in your environment you can rejoin the demoted server back to the domain then promote to a DC.

    To prevent this from happening adhere to the following best practices:

    1) Do not use imaging software to take an image of the DC.

    2) Do not take or apply snapshots of the DC.

    3) Do not shut the Virtual Machine down and simply copy the virtual disk as a backup.

    4) If you have the ability to “discard changes” as you do if you are running “Virtual Server 2005 R2”, do not enable this type of setting on a DC Virtual Machine.

    5) Use NTBACKUP.EXE, WBADMIN.EXE, or any third party software that is available as long as it is certified to be AD-compatible to take system state backups.

    6) Only restore a system state to the DC or restore a full backup.

    References:

    875495 How to detect and recover from a USN rollback in Windows Server 2003

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;875495

    Appendix A: Virtualized Domain Controllers and Replication Issues

    http://technet.microsoft.com/en-us/library/dd348479.aspx

    Backup and Restore Considerations for Virtualized Domain Controllers

    http://technet.microsoft.com/en-us/library/dd363545.aspx

     

    - Mark Ramey

  • Implementing an OCSP responder: Part I - Introducing OCSP

    The series:

    Designing and Implementing a PKI: Part I Design and Planning

    Designing and Implementing a PKI: Part II

    Designing and Implementing a PKI: Part III Certificate Templates

    Chris here again. For those Security Architects and PKI implementers, you may have known that since Windows Server 2008 we have an Online Certificate Status Protocol (OCSP) responder, and since Windows Vista we have an OCSP client that is integrated with the operating system. I wanted to cover the in and outs of the OCSP responder, and walk through the installation.

    So, you may be asking the question “OCSP what?” First a little background. One of the capabilities of a PKI and in particular a Certificate Authority, aside from issuing certificates, is to publish revocation information.

    For example, let’s say you issue a User certificate to a user for authentication. When the user leaves the company you will most likely want to make sure no one can use that certificate for authentication so you log onto the Certificate Authority and revoke that certificate. Each CA has a period specified when it publishes what are called Certificate Revocation Lists or CRLs for short. When the next CRL is published it will contain the serial number of the certificate, the date and time it was revoked, and the reason that the certificate was revoked. Depending on the configuration the CA it will publish the CRL to a repository such as an LDAP server or a web server. In some instances a task or job must be created to copy the CRL to a repository.

    Aside from CRLs, there are also delta CRLs. Delta CRLs simply contain the revocation information for certificates that have been revoked since the last Base CRL was published. In order to determine revocation status an application would examine the last base CRL, and the latest delta CRL. The reason for publishing delta CRLs is to provide revocation information that has more current data. Also, it can reduce bandwidth since if the base CRL is already cached on the client, just the delta CRL can be downloaded. More on this later.

    In order for applications to determine if a certificate has been revoked, the application examines the CRL Distribution Point (CDP) extension in the certificate. This extension will have information on locations where the CRL can be obtained. These locations are normally either HTTP or LDAP locations.

    The application then can go to those locations to download the CRL. There are, however, some potential issues with this scenario. CRLs over time can get rather large depending on the number of certificates issued and revoked. If CRLs grow to a large size, and many clients have to download CRLs, this can have a negative impact on network performance. More importantly, by default Windows clients will timeout after 15 seconds while trying to download a CRL. Additionally, CRLs have information about every currently valid certificate that has been revoked, which is an excessive amount of data given the fact that an application may only need the revocation status for a few certificates. So, aside from downloading the CRL, the application or the OS has to parse the CRL and find a match for the serial number of the certificate that has been revoked.

    With the above limitations, which mostly revolve around scalability, it is clear that there are some drawbacks to using CRLs. Hence, the introduction of Online Certificate Status Protocol (OCSP). OCSP reduces the overhead associated with CRLs. There are server/client components to OCSP: The OCSP responder, which is the server component, and the OCSP Client. The OCSP Responder accepts status requests from OCSP Clients. When the OCSP Responder receives the request from the client it then needs to determine the status of the certificate using the serial number presented by the client. First the OCSP Responder determines if it has any cached responses for the same request. If it does, it can then send that response to the client. If there is no cached response, the OCSP Responder then checks to see if it has the CRL issued by the CA cached locally on the OCSP. If it does, it can check the revocation status locally, and send a response to the client stating whether the certificate is valid or revoked. The response is signed by the OCSP Signing Certificate that is selected during installation. If the OCSP does not have the CRL cached locally, the OCSP Responder can retrieve the CRL from the CDP locations listed in the certificate. The OCSP Responder then can parse the CRL to determine the revocation status, and send the appropriate response to the client.

    OCSP Components

    OCSP Client

    The OCSP Client is a component that generates OCSP requests based on information stored in the AIA extension of the certificate it is validating. The Windows OCSP client supports the Lightweight OCSP Profile as specified in RFC 5019.

    OCSP Responder

    Web Proxy Cache

    Web Proxy Cache is the Web service that receives requests, sends and caches responses.

    Online Responder Service

    The Online Responder Service is the component that is responsible for managing the configuration of the OCSP responder, retrieving revocation information from the Revocation Providers, signing responses, and auditing changes to the configuration of the OCSP responder (if configured to do so).

    The Online Responder service runs under the Network Service account. When you create the Revocation Configuration you will assign the Signing Certificate that will be used by the Online Responder Service to digitally sign the responses sent back to a requesting client. If you are utilizing the OCSP in conjunction with an Enterprise CA you can choose to enroll for the signing certificate during the Revocation Configuration setup, and you can also choose to automatically reenroll for signing certificates. This eases management because the Signing Certificates are generally set to be valid for a short period of time.

    The reason for the short validity periods is that the OCSP signing certificate contains the id-pkix-ocsp-nocheck extension. This extension tells the client that the certificate is valid for its entire lifetime so the revocation status of the certificate is never checked. The reason why this extension is included is to avoid circular revocation checking. If this extension was not included, the client would contact the OCSP Responder to verify the revocation status for a certificate. The OCSP Responder would then respond with a signed request. The client would then have perform revocation checking for the certificate that signed the response, before finishing revocation checking for the original certificate. At this point if there was an OCSP location specified for the signing certificate, you would run into a loop where the OCSP client would ask for the revocation status for the signing certificate from the OCSP and get a signed response. Then the client would again have to validate the revocation status for the signing certificate. This would occur over and over again. Or alternatively, if a CDP location was specified for the signing certificate, you would then need to download the CRL, and verify the signing certificate, in effect making the OCSP pointless, since you would have to download a CRL to validate the OCSP Signing Certificate. We avoid all of this with the inclusion of the id-pkix-ocsp-nocheck extension.

    So, since we are not checking revocation status for the OCSP Signing certificate you should have a short validity period for the OCSP Signing Certificate to increase security.

    Regardless you will have to the give permissions to the private key of the OCSP Signing Certificate to the Network Service Account since that is the identity under which the service runs. If you are using the OCSP with a Windows Server 2008 Enterprise CA, in the Request Handling tab of a Version 3 Certificate Template there is the option to Add Read permissions to Network Service on the private key. This option is enabled by default on the OCSP Response Signing template.

    image

    If you are using a certificate issued from a Windows Server 2008 Standalone CA, a Windows Server 2003 Enterprise CA or a Windows Server 2003 Standalone CA, you will need to manually grant permissions to the private Key of the OCSP Signing Response Certificate to the Network Service account.

    1. To manually give the Network Service Account access to the private key, open up the Certificates MMC targeted for the Local Computer.

    2. Right click on the certificate, then select All Tasks from the context menu, and then select Manage Private Keys

    image

    3. Click Add on the Permissions dialog box.

    image

    4. Type Network Service, and then click Check Names to resolve the name. Then click OK.

    image

    5. The Network Service only needs read permissions to the Private Key, so deselect the Allow privilege for Full Control, and verify the Allow privilege is granted for Read, and click OK.

    image

    Revocation Configuration

    A Revocation Configuration contains PKI components required to respond to an OCSP request. These include items such as the CA Certificate, OCSP Signing Certificate, and information about the Revocation Provider.

    You can have multiple Revocation Configurations per OCSP Responder allowing the OCSP Responder to provide revocation information for multiple CAs.

    When configuring the Revocation Configuration for the OCSP Responder you will designate the following

    • The certificate of the CA for which you are providing revocation status
    • The Signing Certificate (If the CA is an Enterprise CA, and you are using a certificate template)
    • The Revocation Provider (Limited to Base and Delta CRLs in Windows Server 2008)

    Revocation Provider is the component responsible for retrieving revocation information. In Windows Server 2008 the only revocation provider supported is the CRL based Revocation Provider. In other words the Windows Server 2008 OCSP Responder can only retrieve revocation information from published CRLs.

    High Availability

    OCSP Responders can be configured for high availability by placing the OCSP responders in an Array. The Array itself does not provide fault tolerances, but maintains the configurations of multiple OCSP responders that are part of the Array. The configuration is maintained by the OCSP Responder that is designated as the “Array Controller”.

    Once the responders are arranged in an Array you can use Network Load Balancing to provide a highly available configuration.

    I will cover the process of creating a highly available OCSP configuration in a future blog article.

    Conclusion

    I hope you found the information in this posting helpful. I plan on continuing the series on deploying an OCSP Responder. I will be posting the following blog entries soon, stay tuned:

    Implementing an OCSP responder: Part I Introducing OCSP
    Implementing an OCSP responder: Part II Preparing Certificate Authorities
    Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
    Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 
    Implementing an OCSP Responder: Part V High Availability
    Implementing an OCSP Responder: Part VI Configuring Custom OCSP URIs via Group Policy

    - Chris Delay

  • Implementing an OCSP responder: Part III - Configuring OCSP for use with Enterprise CAs

    Chris here again. As promised I will be covering configuring an OCSP Responder to support Enterprise CA. I will also be covering validating your OCSP Configuration.

    Installing OCSP Responder Role

    The first step is to install the OCSP Responder Role.

    To install the OCSP Responder: Open a command prompt and type: servermanagercmd.exe –install ADCS-Online-Cert.

    Configuring the OCSP Responder

    First we will add a Revocation Configuration to the OCSP Responder.

    Right click on the Revocation Configuration and select Add Revocation Configuration from the context menu.

    image

    The Add Revocation Configuration wizard opens. Click Next to continue.

    image

    Give a Friendly Name to the Revocation Configuration, and click Next. It is a good idea to include the name of the CA for which you are setting up this Revocation Configuration, especially if this OCSP Responder will handle requests for multiple CAs.

    image

    On the Select CA Certificate page, you will need to select a CA certificate. This is where you determine the CA for which you will be providing revocation information.

    Select a certificate for an Existing enterprise CA, and click Next

    image

    Select Browse CA certificates published in Active Directory, and click Browse.

    image

    Select the appropriate CA, and click OK

    image

    Next you will need to select a certificate that will be used for signing OCSP responses. For a particular Revocation Configuration, the OCSP Signing certificate must be issued by the CA for which the OCSP Responder will answer revocation status requests.

    Select Automatically select a signing certificate. If you wish to automatically enroll for the OCSP Response Signing Certificate, make sure the Auto-Enroll for an OCSP signing certificate is checked. Select the certificate template that you configured for use with the OCSP Responder, then click Next.

    image

    On the Revocation Provider page, you can click Provider to select revocation providers. The Windows Server 2008 OCSP Responder can only use CRLs for revocation information. If you have the CDP Extension available in the signing certificate, the Revocation Providers will be populated from the information in the CDP Extension from the OCSP Response Signing Certificate.

    image

    You can add the repository locations for your CRLs and Delta CRLs if appropriate. By default these will be populated from information included in the CDP extension of the Signing certificate. After you have reviewed the configuration or made any changes, click OK.

    image

    That completes the initial Configuration of the OCSP Responder. If you would like to modify the configuration of the OCSP Responder, you can right click on the Revocation Configuration and select Properties from the context menu.

    image

    The Local CRL tab allows you to configure a Local CRL. You can add revocation information for certificates which you wish to consider revoked. It is recommended that you do not use this option, as it adds unnecessary complexity to the revocation configuration.

    image

    The Revocation Provider tab allows you to modify the location of the CRLs and Delta CRLs that will be used for providing revocation information.

    image

    Signing Tab

    In the signing tab you can:

    • Modify the hash algorithm used to sign responses.
    • Do not prompt for credentials for cryptographic operations. This setting may need to be disabled if you are using an HSM to protect the private key of the OCSP Signing certificate. Disabling this setting allows you to be prompted for the password that is associated with the operator card on the HSM.
    • Use renewed certificates for signing certificates. This option is enabled by default, when you use the OCSP Responder with an Enterprise CA and automatically renew certificates. If you use OCSP Responder with a standalone CA, the OCSP responder will use renewed signing certificates even if this setting is not enabled.
    • Enable NONCE extension support allows the user to attach the NONCE sent in the request with the OCSP response. If this setting is used, you will not be able to utilize cached responses.
    • Use any valid OCSP signing certificate. Not recommended if the OCSP Responder is supporting Vista clients since they do not support this option. This allows the OCSP responder to use any certificate that the OCSP Signing configured in the Extended Key Usage extension of the certificate. Vista clients will only accept OCSP responses that are signed by the same CA for which the OCSP Responder is providing revocation information.
    • All responses will included the following Online Responder identifies: This setting determines whether a Key Hash or Subject will be included in the response. RFC 2560 specifies the structure of the response. In section 4.2.1 of the RFC it is specified that the Responder ID field can either be populated with a Name or Key hash. This setting determines which is included in the response. The Key hash is a hash of the OCSP Responder’s public key. The Name is the distinguished name of the subject of the OCSP signing certificate.

    image

    Verify OCSP Configuration

    After configuring the OCSP Responder, you will want to verify that the OCSP responder is functioning properly. The easiest way to verify that the OCSP is functioning is to use the Certutil URL Retrieval tool.

    First request a certificate from the CA. Place a copy of that cert on the file system, and run the following command: certutil –URL <Certificate Name>. This will open the URL Retrieval Tool

    image

    Select OCSP, and click on the Retrieve button.

    image

    If the certificate is valid you will get the following response.

    image

    If the certificate is revoked, you will get the following response.

    image

    And if it fails, the status will be listed as Failed.

    image

    You can also use the PKIView tool to verify the configurations of the OCSP Responder.

    image

    Conclusion

    This concludes configuring an OCSP Responder to support an Enterprise CA. If you follow the steps listed here you now have your OCSP configured to support your Windows Server 2003 or Windows Server 2008 CA. In the next part of this series, I will be configuring an OCSP Responder to support Standalone CA.

    Implementing an OCSP responder: Part I Introducing OCSP
    Implementing an OCSP responder: Part II Preparing Certificate Authorities
    Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
    Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 
    Implementing an OCSP Responder: Part V High Availability
    Implementing an OCSP Responder: Part VI Configuring Custom OCSP URIs via Group Policy

    - Chris Delay

  • Debunking the Vista Remote Differential Compression Myth

    Ned here again. Have you ever visited Snopes.com? It’s a terrific urban legend reference where they research folklore. Snopes is the place you go to find out if eating Thanksgiving turkey makes you sleepy (it doesn’t), if Coca Cola can dissolve a tooth overnight (it can’t), or if a man really did live in a Paris airport for 8 years (he did!).

    Today I’m going to talk about another urban legend – that removing the Remote Differential Compression feature from Windows Vista will make your file copying faster over the network.

    Background on RDC

    Remote Differential Compression (RDC) is a Microsoft algorithm that was originally created for DFSR five years ago. RDC divides a file’s data into chunks by using signatures. When a file exists on two computers and the file is modified, only the differing chunks need to be sent to the other computer.

    image

    An application needs to be specifically written to support RDC. Windows Vista and Windows 7 include MSRDC.DLL to allow apps like Windows Live Messenger to use that functionality.

    image

    The feature can be turned on and off within the Control Panel “Program and Features” applet.

    image

    When turned on, the MSRDC.DLL will exist in the %SYSTEMROOT%\System32 directory. When it’s turned off, this DLL is removed.

    image

    The Myth

    Unfortunately, the Internet is full of people telling you that RDC will somehow make your network communication slower. I have no idea how this got started, but this nonsense has been reprinted on thousands of websites by people unfamiliar with the Scientific Method. Folks have actually convinced themselves that turning this feature on or off has some affect on file transfer speeds. While there are a variety of things you can do to speed up Vista file copy performance, this isn’t one of them.

    The Method

    So after hearing this baloney for the umpteenth time, I set out to debunk it once and for all:

    • I setup a Windows Server 2008 SP2 file server and a Windows Vista SP2 client as virtual guests inside a Hyper-V host.
    • Each guest got 1GB of RAM and a virtual gigabit NIC.
    • I dropped a 3GB ISO file onto a share on the file server.
    • I used robocopy.exe (which automatically reports bytes per second and megabytes per minute) to copy that ISO file five times with MSRDC.DLL installed, then five more times with MSRDC.DLL removed.
    • In between every copy, I rebooted both the client and server to ensure that there was no caching of the file that might artificially improve my results.

    The Results

    Here is what I found after the ten total passes, with and without RDC installed:

    Pass

     

    With MSRDC.DLL

     

    Without MSRDC.DLL

     

    1st pass

     

    1237.939

     

    1210.479

     

    2nd pass

     

    1186.415

     

    1330.882

     

    3rd pass

     

    1192.068

     

    1175.328

     

    4th pass

     

    1111.13

     

    1170.281

     

    5th pass

     

    1320.867

     

    1153.863

     

     

    image

    Hmmm… What if I sort my data points highest to lowest?

    image

    Interesting. Let’s look at the actual averages:

    With MSRDC.DLL installed: 1209.6838 MB/min

    With MSRDC.DLL removed: 1208.1666 MB/min

    Wait – so removing RDC actually made it slower? Not really – the variance there is well within any respectable margin of error. These results mean that the two sets of copies were 99.87% identical. Removing RDC did nothing at all. There are going to be various performance differences when copying a file, depending on what else happens on the network, what the computers are doing, and I think when people claim RDC removal made things ‘faster’, it’s because they are not testing repeatedly over time to see variance.

    Maybe you want more proof? Alright, let’s go to the debugger.

    • I installed Notmyfault on my Vista client and configured the computer for a complete memory dump
    • I started yet another file copy, then I listed out the process and service information with TASKLIST.EXE and TASKLIST.EXE /SVC
    • After a few seconds, I intentionally crash dumped (the so-called “Blue Screen of Death”) the Vista computer. I then cracked open the MEMORY.DMP file in the Windows debugger

    First, I listed loaded modules - all libraries and drivers loaded in the kernel memory space:

    kd> lm
    start             end                 module name
    00000000`75f50000 00000000`75f88000   odbcint    (deferred)            
    00000000`77aa0000 00000000`77b6d000   USER32     (deferred)            
    00000000`77b70000 00000000`77c9d000   kernel32   (deferred)            
    00000000`77ca0000 00000000`77e26000   ntdll      (export symbols)       ntdll.dll
    00000000`77e50000 00000000`77e54000   Normaliz   (deferred)            
    00000000`ffec0000 00000000`fff41000   Robocopy   (deferred)            
    000007fe`f4be0000 000007fe`f4d30000   MFC42u     (deferred)            
    000007fe`f9200000 000007fe`f9271000   ODBC32     (deferred)            
    000007fe`fbd90000 000007fe`fbe30000   COMCTL32   (deferred)            
    000007fe`fcc80000 000007fe`fce79000   comctl32_7fefcc80000   (deferred)            
    000007fe`fe420000 000007fe`fe4ac000   COMDLG32   (deferred)            
    000007fe`fe620000 000007fe`fe64d000   IMM32      (deferred)            
    000007fe`fe6f0000 000007fe`fe8c8000   ole32      (deferred)            
    000007fe`fe8d0000 000007fe`fe92f000   iertutil   (deferred)            
    000007fe`fe930000 000007fe`fe937000   NSI        (deferred)            
    000007fe`feb80000 000007fe`fecc3000   RPCRT4     (deferred)            
    000007fe`fecd0000 000007fe`fed6a000   USP10      (deferred)            
    000007fe`fed70000 000007fe`fedb4000   WS2_32     (deferred)            
    000007fe`fedc0000 000007fe`fee33000   SHLWAPI    (deferred)            
    000007fe`fee40000 000007fe`feedc000   msvcrt     (deferred)            
    000007fe`feee0000 000007fe`fefe2000   MSCTF      (deferred)            
    000007fe`feff0000 000007fe`feffd000   LPK        (deferred)            
    000007fe`ff000000 000007fe`ffc53000   SHELL32    (deferred)            
    000007fe`ffc60000 000007fe`ffd5e000   WININET    (deferred)            
    000007fe`ffd60000 000007fe`ffdc4000   GDI32      (deferred)            
    000007fe`ffdd0000 000007fe`ffea3000   OLEAUT32   (deferred)            
    000007fe`ffeb0000 000007fe`fffb8000   ADVAPI32   (deferred)            
    fffff800`01804000 fffff800`01d1c000   nt         (private)
    fffff800`01d1c000 fffff800`01d62000   hal        (deferred)            
    fffff960`000c0000 fffff960`00371000   win32k     (deferred)            
    fffff960`00480000 fffff960`0049e000   dxg        (deferred)            
    fffff960`00600000 fffff960`0060a000   TSDDD      (deferred)            
    fffff960`00820000 fffff960`0082b000   VMBusVideoD   (deferred)            
    fffffa60`00602000 fffffa60`0060c000   kdcom      (deferred)            
    fffffa60`0060c000 fffffa60`00647000   mcupdate_GenuineIntel   (deferred)            
    fffffa60`00647000 fffffa60`0065b000   PSHED      (deferred)            
    fffffa60`0065b000 fffffa60`006b8000   CLFS       (deferred)            
    fffffa60`006b8000 fffffa60`0076a000   CI         (deferred)            
    fffffa60`0076a000 fffffa60`007d0000   volmgrx    (deferred)            
    fffffa60`007d0000 fffffa60`007e4000   NDProxy    (deferred)            
    fffffa60`007e4000 fffffa60`007ef000   Msfs       (deferred)            
    fffffa60`007ef000 fffffa60`00800000   Npfs       (deferred)            
    fffffa60`00808000 fffffa60`008e2000   Wdf01000   (deferred)            
    fffffa60`008e2000 fffffa60`008f0000   WDFLDR     (deferred)            
    fffffa60`008f0000 fffffa60`00946000   acpi       (deferred)            
    fffffa60`00946000 fffffa60`0094f000   WMILIB     (deferred)            
    fffffa60`0094f000 fffffa60`00959000   msisadrv   (deferred)            
    fffffa60`00959000 fffffa60`00989000   pci        (deferred)            
    fffffa60`00989000 fffffa60`0099e000   partmgr    (deferred)            
    fffffa60`0099e000 fffffa60`009b2000   volmgr     (deferred)            
    fffffa60`009b2000 fffffa60`009ba000   intelide   (deferred)            
    fffffa60`009ba000 fffffa60`009ca000   PCIIDEX    (deferred)            
    fffffa60`009ca000 fffffa60`009fd000   netvsc60   (deferred)            
    fffffa60`00a00000 fffffa60`00a3d000   vmbus      (deferred)            
    fffffa60`00a3d000 fffffa60`00a51000   winhv      (deferred)            
    fffffa60`00a51000 fffffa60`00a64000   mountmgr   (deferred)            
    fffffa60`00a64000 fffffa60`00a6c000   atapi      (deferred)            
    fffffa60`00a6c000 fffffa60`00a90000   ataport    (deferred)            
    fffffa60`00a90000 fffffa60`00ad7000   fltmgr     (deferred)            
    fffffa60`00ad7000 fffffa60`00aeb000   fileinfo   (deferred)            
    fffffa60`00aeb000 fffffa60`00af8000   storvsc    (deferred)            
    fffffa60`00af8000 fffffa60`00b55000   storport   (deferred)            
    fffffa60`00b55000 fffffa60`00bdb000   ksecdd     (deferred)            
    fffffa60`00bdb000 fffffa60`00bee000   intelppm   (deferred)            
    fffffa60`00bee000 fffffa60`00bf7000   rdpencdd   (deferred)            
    fffffa60`00bf7000 fffffa60`00c00000   rasacd     (deferred)            
    fffffa60`00c00000 fffffa60`00c0e000   vga        (deferred)            
    fffffa60`00c0f000 fffffa60`00dd2000   ndis       (deferred)            
    fffffa60`00dd2000 fffffa60`00dee000   cdrom      (deferred)            
    fffffa60`00dee000 fffffa60`00df7000   Null       (deferred)            
    fffffa60`00df7000 fffffa60`00e00000   RDPCDD     (deferred)            
    fffffa60`00e00000 fffffa60`00e0a000   Fs_Rec     (deferred)            
    fffffa60`00e0c000 fffffa60`00e5c000   msrpc      (deferred)            
    fffffa60`00e5c000 fffffa60`00eb5000   NETIO      (deferred)            
    fffffa60`00eb5000 fffffa60`00ede000   fvevol     (deferred)            
    fffffa60`00ede000 fffffa60`00f0a000   CLASSPNP   (deferred)            
    fffffa60`00f29000 fffffa60`00f35000   tunnel     (deferred)            
    fffffa60`00f35000 fffffa60`00f4b000   i8042prt   (deferred)            
    fffffa60`00f4b000 fffffa60`00f59000   kbdclass   (deferred)            
    fffffa60`00f59000 fffffa60`00f65000   mouclass   (deferred)            
    fffffa60`00f65000 fffffa60`00f82000   serial     (deferred)            
    fffffa60`00f82000 fffffa60`00f8e000   serenum    (deferred)            
    fffffa60`00f8e000 fffffa60`00f9b000   fdc        (deferred)            
    fffffa60`00f9b000 fffffa60`00fad000   HIDCLASS   (deferred)            
    fffffa60`00fad000 fffffa60`00fb7000   VMBusVideoM   (deferred)            
    fffffa60`00fb7000 fffffa60`00fdc000   VIDEOPRT   (deferred)            
    fffffa60`00fdc000 fffffa60`00fec000   watchdog   (deferred)            
    fffffa60`00fec000 fffffa60`00ff5000   vms3cap    (deferred)            
    fffffa60`00ff5000 fffffa60`01000000   mouhid     (deferred)            
    fffffa60`01000000 fffffa60`01007b80   HIDPARSE   (deferred)            
    fffffa60`01008000 fffffa60`0117d000   tcpip      (deferred)            
    fffffa60`0117d000 fffffa60`011a9000   fwpkclnt   (deferred)            
    fffffa60`011a9000 fffffa60`011b9000   vmstorfl   (deferred)            
    fffffa60`011b9000 fffffa60`011e5000   ecache     (deferred)            
    fffffa60`011e5000 fffffa60`011ef000   crcdisk    (deferred)            
    fffffa60`01208000 fffffa60`01388000   Ntfs       (deferred)            
    fffffa60`01388000 fffffa60`013cc000   volsnap    (deferred)            
    fffffa60`013cc000 fffffa60`013d4000   spldr      (deferred)            
    fffffa60`013d4000 fffffa60`013e6000   mup        (deferred)            
    fffffa60`013e6000 fffffa60`013fa000   disk       (deferred)            
    fffffa60`013fa000 fffffa60`013ff500   VMBusHID   (deferred)            
    fffffa60`02200000 fffffa60`0220b000   flpydisk   (deferred)            
    fffffa60`0220f000 fffffa60`02248000   msiscsi    (deferred)            
    fffffa60`02248000 fffffa60`02255000   TDI        (deferred)            
    fffffa60`02255000 fffffa60`02278000   rasl2tp    (deferred)            
    fffffa60`02278000 fffffa60`02284000   ndistapi   (deferred)            
    fffffa60`02284000 fffffa60`022b5000   ndiswan    (deferred)            
    fffffa60`022b5000 fffffa60`022c5000   raspppoe   (deferred)            
    fffffa60`022c5000 fffffa60`022e3000   raspptp    (deferred)            
    fffffa60`022e3000 fffffa60`022fb000   rassstp    (deferred)            
    fffffa60`022fb000 fffffa60`02395000   rdpdr      (deferred)            
    fffffa60`02395000 fffffa60`023a8000   termdd     (deferred)            
    fffffa60`023a8000 fffffa60`023a9480   swenum     (deferred)            
    fffffa60`023aa000 fffffa60`023de000   ks         (deferred)            
    fffffa60`023de000 fffffa60`023e9000   mssmbios   (deferred)            
    fffffa60`023e9000 fffffa60`023f9000   umbus      (deferred)            
    fffffa60`02401000 fffffa60`0241e000   tdx        (deferred)            
    fffffa60`0241e000 fffffa60`02439000   smb        (deferred)            
    fffffa60`02439000 fffffa60`024a4000   afd        (deferred)            
    fffffa60`024a4000 fffffa60`024e8000   netbt      (deferred)            
    fffffa60`024e8000 fffffa60`02506000   pacer      (deferred)            
    fffffa60`02506000 fffffa60`02514000   nm3        (deferred)            
    fffffa60`02514000 fffffa60`02523000   netbios    (deferred)            
    fffffa60`02523000 fffffa60`0253e000   wanarp     (deferred)            
    fffffa60`0253e000 fffffa60`0258b000   rdbss      (deferred)            
    fffffa60`0258b000 fffffa60`02597000   nsiproxy   (deferred)            
    fffffa60`02597000 fffffa60`025c0000   srvnet     (deferred)            
    fffffa60`025c0000 fffffa60`025de000   bowser     (deferred)            
    fffffa60`02c08000 fffffa60`02c7e000   csc        (deferred)            
    fffffa60`02c7e000 fffffa60`02c9b000   dfsc       (deferred)            
    fffffa60`02c9b000 fffffa60`02cb7000   cdfs       (deferred)            
    fffffa60`02cb7000 fffffa60`02cc5000   crashdmp   (deferred)            
    fffffa60`02cc5000 fffffa60`02cd1000   dump_dumpata   (deferred)            
    fffffa60`02cd1000 fffffa60`02cd9000   dump_atapi   (deferred)            
    fffffa60`02cd9000 fffffa60`02cec000   dump_dumpfve   (deferred)            
    fffffa60`02cec000 fffffa60`02cf8000   Dxapi      (deferred)            
    fffffa60`02cf8000 fffffa60`02d1a000   luafv      (deferred)            
    fffffa60`02d1a000 fffffa60`02d2e000   lltdio     (deferred)            
    fffffa60`02d2e000 fffffa60`02d46000   rspndr     (deferred)            
    fffffa60`02d46000 fffffa60`02de5000   HTTP       (deferred)            
    fffffa60`02de5000 fffffa60`02dff000   mpsdrv     (deferred)            
    fffffa60`03805000 fffffa60`0382c000   mrxdav     (deferred)            
    fffffa60`0382c000 fffffa60`03855000   mrxsmb     (deferred)            
    fffffa60`03855000 fffffa60`0389e000   mrxsmb10   (deferred)            
    fffffa60`0389e000 fffffa60`038bd000   mrxsmb20   (deferred)            
    fffffa60`038bd000 fffffa60`038ef000   srv2       (deferred)            
    fffffa60`038ef000 fffffa60`03980000   srv        (deferred)            
    fffffa60`03c03000 fffffa60`03c9d000   spsys      (deferred)            
    fffffa60`03c9d000 fffffa60`03d53000   peauth     (deferred)            
    fffffa60`03d53000 fffffa60`03d5e000   secdrv     (deferred)            
    fffffa60`03d5e000 fffffa60`03d6e000   tcpipreg   (deferred)            
    fffffa60`03d6e000 fffffa60`03d75000   myfault    (deferred)            

    Note how MSRDC.DLL is not loaded in memory in the Kernel space. It’s still possible that a given process or service might have it loaded though, so then I listed all processes to see which ones would be interesting and likely to be involved in file copies. The TASKLIST output comes in handy here to see which PID is which hexadecimal CID value. In my case though I dumped them all just for exploratory purposes.

    kd> !process 0 0

    <snipped out some>

    PROCESS fffffa8009af9040
        SessionId: 1  Cid: 0790    Peb: 7fffffdf000  ParentCid: 0a08
        DirBase: 34e57000  ObjectTable: fffff880066f60d0  HandleCount:  65.
        Image: Robocopy.exe

    PROCESS fffffa800b9adc10
        SessionId: 0  Cid: 0340    Peb: 7fffffd5000  ParentCid: 0280
        DirBase: 1efc3000  ObjectTable: fffff8800634fb60  HandleCount: 522.
        Image: svchost.exe

     

     

     

     

    I know that the Workstation Service is responsible for SMB file copying, and the robocopy process is definitely doing work, so I examined those.

    kd> .process fffffa8009af9040
    Implicit process is now fffffa80`09af9040
    kd> !peb
    PEB at 000007fffffdf000
        InheritedAddressSpace:    No
        ReadImageFileExecOptions: No
        BeingDebugged:            No
        ImageBaseAddress:         00000000ffec0000
        Ldr                       0000000077db2960
        Ldr.Initialized:          Yes
        Ldr.InInitializationOrderModuleList: 00000000001226c0 . 00000000001373c0
        Ldr.InLoadOrderModuleList:           00000000001225d0 . 00000000001373a0
        Ldr.InMemoryOrderModuleList:         00000000001225e0 . 00000000001373b0
                Base TimeStamp                     Module
            ffec0000 479191ad Jan 19 00:59:09 2008 C:\Windows\system32\Robocopy.exe
            77ca0000 49e0421d Apr 11 03:09:17 2009 C:\Windows\system32\ntdll.dll
            77b70000 49e041d1 Apr 11 03:08:01 2009 C:\Windows\system32\kernel32.dll
         7feffeb0000 49e040cb Apr 11 03:03:39 2009 C:\Windows\system32\ADVAPI32.dll
         7fefeb80000 49e041ea Apr 11 03:08:26 2009 C:\Windows\system32\RPCRT4.dll
         7fef4be0000 49e04151 Apr 11 03:05:53 2009 C:\Windows\system32\MFC42u.dll
         7fefee40000 49e04189 Apr 11 03:06:49 2009 C:\Windows\system32\msvcrt.dll
            77aa0000 49e0420e Apr 11 03:09:02 2009 C:\Windows\system32\USER32.dll
         7feffd60000 49e04114 Apr 11 03:04:52 2009 C:\Windows\system32\GDI32.dll
         7fefe6f0000 49e041cf Apr 11 03:07:59 2009 C:\Windows\system32\ole32.dll
         7feffdd0000 49e041d2 Apr 11 03:08:02 2009 C:\Windows\system32\OLEAUT32.dll
         7feffc60000 49e04252 Apr 11 03:10:10 2009 C:\Windows\system32\WININET.dll
         7fefedc0000 49e041f4 Apr 11 03:08:36 2009 C:\Windows\system32\SHLWAPI.dll
            77e50000 4549b4d2 Nov 02 05:05:22 2006 C:\Windows\system32\Normaliz.dll
         7fefe8d0000 49e04146 Apr 11 03:05:42 2009 C:\Windows\system32\iertutil.dll
         7fefed70000 49e0422d Apr 11 03:09:33 2009 C:\Windows\system32\WS2_32.dll
         7fefe930000 4791adea Jan 19 02:59:38 2008 C:\Windows\system32\NSI.dll
         7fef9200000 49e041c1 Apr 11 03:07:45 2009 C:\Windows\system32\ODBC32.dll
         7fefbd90000 4791ac7c Jan 19 02:53:32 2008 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.6001.18000_none_40ba501d3c2b20ff\COMCTL32.dll
         7feff000000 49e041ef Apr 11 03:08:31 2009 C:\Windows\system32\SHELL32.dll
         7fefe420000 49e041e9 Apr 11 03:08:25 2009 C:\Windows\system32\COMDLG32.dll
         7fefe620000 49e0417d Apr 11 03:06:37 2009 C:\Windows\system32\IMM32.DLL
         7fefeee0000 49e04184 Apr 11 03:06:44 2009 C:\Windows\system32\MSCTF.dll
         7fefeff0000 4791ad25 Jan 19 02:56:21 2008 C:\Windows\system32\LPK.DLL
         7fefecd0000 49e04211 Apr 11 03:09:05 2009 C:\Windows\system32\USP10.dll
         7fefcc80000 49e041e9 Apr 11 03:08:25 2009 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18005_none_1509f8bef40ee4da\comctl32.dll
            75f50000 4549d310 Nov 02 07:14:24 2006 C:\Windows\system32\odbcint.dll

    kd> .PROCESS fffffa800b9adc10
    Implicit process is now fffffa80`0b9adc10
    kd> !peb
    PEB at 000007fffffd5000
        InheritedAddressSpace:    No
        ReadImageFileExecOptions: No
        BeingDebugged:            No
        ImageBaseAddress:         00000000ff820000
        Ldr                       0000000077db2960
        Ldr.Initialized:          Yes
        Ldr.InInitializationOrderModuleList: 00000000002125f0 . 0000000003a08ac0
        Ldr.InLoadOrderModuleList:           0000000000212500 . 0000000003a08aa0
        Ldr.InMemoryOrderModuleList:         0000000000212510 . 0000000003a08ab0
                Base TimeStamp                     Module
            ff820000 47919291 Jan 19 01:02:57 2008 C:\Windows\system32\svchost.exe
            77ca0000 49e0421d Apr 11 03:09:17 2009 C:\Windows\system32\ntdll.dll
            77b70000 49e041d1 Apr 11 03:08:01 2009 C:\Windows\system32\kernel32.dll
         7fefee40000 49e04189 Apr 11 03:06:49 2009 C:\Windows\system32\msvcrt.dll
         7feffeb0000 49e040cb Apr 11 03:03:39 2009 C:\Windows\system32\ADVAPI32.dll
         7fefeb80000 49e041ea Apr 11 03:08:26 2009 C:\Windows\system32\RPCRT4.dll
         7fefd440000 49e0422f Apr 11 03:09:35 2009 C:\Windows\system32\NTMARTA.DLL
            77aa0000 49e0420e Apr 11 03:09:02 2009 C:\Windows\system32\USER32.dll
         7feffd60000 49e04114 Apr 11 03:04:52 2009 C:\Windows\system32\GDI32.dll
         7fefe940000 49e0427e Apr 11 03:10:54 2009 C:\Windows\system32\WLDAP32.dll
         7fefed70000 49e0422d Apr 11 03:09:33 2009 C:\Windows\system32\WS2_32.dll
         7fefe930000 4791adea Jan 19 02:59:38 2008 C:\Windows\system32\NSI.dll
            77e40000 47919b74 Jan 19 01:40:52 2008 C:\Windows\system32\PSAPI.DLL
         7fefdce0000 49e041e3 Apr 11 03:08:19 2009 C:\Windows\system32\SAMLIB.dll
         7fefe6f0000 49e041cf Apr 11 03:07:59 2009 C:\Windows\system32\ole32.dll
         7fefe620000 49e0417d Apr 11 03:06:37 2009 C:\Windows\system32\IMM32.DLL
         7fefeee0000 49e04184 Apr 11 03:06:44 2009 C:\Windows\system32\MSCTF.dll
         7fefeff0000 4791ad25 Jan 19 02:56:21 2008 C:\Windows\system32\LPK.DLL
         7fefecd0000 49e04211 Apr 11 03:09:05 2009 C:\Windows\system32\USP10.dll
         7fefc1c0000 49e0419d Apr 11 03:07:09 2009 c:\windows\system32\es.dll
         7feffdd0000 49e041d2 Apr 11 03:08:02 2009 C:\Windows\system32\OLEAUT32.dll
         7fefc000000 49e041dd Apr 11 03:08:13 2009 c:\windows\system32\PROPSYS.dll
         7fefd510000 49e041ed Apr 11 03:08:29 2009 C:\Windows\system32\rsaenh.dll
         7fefe650000 4791acc9 Jan 19 02:54:49 2008 C:\Windows\system32\CLBCatQ.DLL
         7fefc4b0000 4791adeb Jan 19 02:59:39 2008 c:\windows\system32\nsisvc.dll
         7fefe250000 49e04210 Apr 11 03:09:04 2009 C:\Windows\system32\secur32.dll
         7fefdb10000 49e04202 Apr 11 03:08:50 2009 C:\Windows\system32\CRYPT32.dll
         7fefdcc0000 4791ad5c Jan 19 02:57:16 2008 C:\Windows\system32\MSASN1.dll
         7fefe270000 49e04210 Apr 11 03:09:04 2009 C:\Windows\system32\USERENV.dll
         7fefd8f0000 4791adc3 Jan 19 02:58:59 2008 C:\Windows\system32\credssp.dll
         7fefd4b0000 49e041f1 Apr 11 03:08:33 2009 C:\Windows\system32\schannel.dll
         7fefdfc0000 49e041a5 Apr 11 03:07:17 2009 C:\Windows\system32\NETAPI32.dll
         7fefbc50000 49e04225 Apr 11 03:09:25 2009 c:\windows\system32\webclnt.dll
         7fefba50000 49e04251 Apr 11 03:10:09 2009 c:\windows\system32\WINHTTP.dll
         7fefedc0000 49e041f4 Apr 11 03:08:36 2009 C:\Windows\system32\SHLWAPI.dll
         7fefe4b0000 49e04209 Apr 11 03:08:57 2009 C:\Windows\system32\urlmon.dll
         7fefe8d0000 49e04146 Apr 11 03:05:42 2009 C:\Windows\system32\iertutil.dll
         7fefcc80000 49e041e9 Apr 11 03:08:25 2009 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18005_none_1509f8bef40ee4da\comctl32.dll
         7feff000000 49e041ef Apr 11 03:08:31 2009 C:\Windows\system32\shell32.dll
         7feffc60000 49e04252 Apr 11 03:10:10 2009 C:\Windows\system32\WinInet.dll
            77e50000 4549b4d2 Nov 02 05:05:22 2006 C:\Windows\system32\Normaliz.dll
         7fefbc10000 4791ae1c Jan 19 03:00:28 2008 c:\windows\system32\wkssvc.dll
         7fefda40000 49e04193 Apr 11 03:06:59 2009 c:\windows\system32\IPHLPAPI.DLL
         7fefd9f0000 49e040f3 Apr 11 03:04:19 2009 c:\windows\system32\dhcpcsvc.DLL
         7fefdd00000 49e04119 Apr 11 03:04:57 2009 c:\windows\system32\DNSAPI.dll
         7fefd9e0000 4791ae08 Jan 19 03:00:08 2008 c:\windows\system32\WINNSI.DLL
         7fefd9b0000 49e040f4 Apr 11 03:04:20 2009 c:\windows\system32\dhcpcsvc6.DLL
         7fefdc90000 4791adef Jan 19 02:59:43 2008 c:\windows\system32\NTDSAPI.dll
         7fefd5a0000 4791adf5 Jan 19 02:59:49 2008 c:\windows\system32\WINBRAND.dll
         7fefb7d0000 4549d27e Nov 02 07:11:58 2006 c:\windows\system32\fdrespub.dll
         7fefb500000 49e0423a Apr 11 03:09:46 2009 c:\windows\system32\wsdapi.dll
         7fefb810000 4791ad11 Jan 19 02:56:01 2008 c:\windows\system32\HTTPAPI.dll
         7fefd3a0000 4791ae1a Jan 19 03:00:26 2008 c:\windows\system32\WINTRUST.dll
         7fefe400000 4791ad46 Jan 19 02:56:54 2008 C:\Windows\system32\imagehlp.dll
         7fefcfc0000 4791addb Jan 19 02:59:23 2008 c:\windows\system32\XmlLite.dll
         7fefd290000 4791ace8 Jan 19 02:55:20 2008 c:\windows\system32\FirewallAPI.dll
         7fefd820000 49e04210 Apr 11 03:09:04 2009 c:\windows\system32\VERSION.dll
         7fefb280000 49e0411b Apr 11 03:04:59 2009 C:\Windows\system32\FunDisc.dll
         7fefc840000 4791ac8a Jan 19 02:53:46 2008 C:\Windows\system32\ATL.DLL
         7fefe9a0000 49e041ed Apr 11 03:08:29 2009 C:\Windows\system32\SETUPAPI.dll
         7fefd790000 49e0418f Apr 11 03:06:55 2009 C:\Windows\system32\mswsock.dll
         7fefd400000 4791aeae Jan 19 03:02:54 2008 C:\Windows\System32\wshtcpip.dll
         7fefd810000 4791aea8 Jan 19 03:02:48 2008 C:\Windows\System32\wship6.dll
         7fefacc0000 49e04191 Apr 11 03:06:57 2009 C:\Windows\System32\msxml3.dll
         7fefb140000 4791ae0e Jan 19 03:00:14 2008 c:\windows\system32\ssdpsrv.dll
         7fefafa0000 49e0420a Apr 11 03:08:58 2009 c:\windows\system32\w32time.dll
         7fefdd40000 4791adc8 Jan 19 02:59:04 2008 c:\windows\system32\cryptdll.dll
         7fefd3e0000 49e04118 Apr 11 03:04:56 2009 C:\Windows\system32\GPAPI.dll
         7fefdae0000 49e041da Apr 11 03:08:10 2009 C:\Windows\system32\slc.dll
         7fefb100000 49ee93d7 Apr 21 23:49:43 2009 C:\Windows\System32\vmictimeprovider.dll
         7fefa300000 4791ad84 Jan 19 02:57:56 2008 c:\windows\system32\netprofm.dll
         7fefc900000 4791ad8c Jan 19 02:58:04 2008 c:\windows\system32\nlaapi.dll
         7fefa2a0000 4791adbc Jan 19 02:58:52 2008 c:\windows\system32\upnphost.dll
         7fefb3f0000 4549d324 Nov 02 07:14:44 2006 c:\windows\system32\SSDPAPI.dll
         7fefaf90000 4549d36c Nov 02 07:15:56 2006 C:\Windows\System32\npmproxy.dll
         7fefe070000 4791adb4 Jan 19 02:58:44 2008 C:\Windows\system32\SXS.DLL
         7fefce90000 4791acf3 Jan 19 02:55:31 2008 c:\windows\system32\fdphost.dll
         7fef69c0000 49e04124 Apr 11 03:05:08 2009 C:\Windows\system32\fdwsd.dll
         7fef6980000 4791ad25 Jan 19 02:56:21 2008 C:\Windows\system32\MLANG.dll
         7fef6960000 49e04121 Apr 11 03:05:05 2009 C:\Windows\system32\fdssdp.dll
         7fefd020000 49e0411f Apr 11 03:05:03 2009 C:\Windows\system32\fdproxy.dll
         7fefb840000 4791ad5c Jan 19 02:57:16 2008 C:\Windows\system32\napinsp.dll
         7fef97d0000 4791adb8 Jan 19 02:58:48 2008 C:\Windows\system32\pnrpnsp.dll
         7fefb860000 4791ae09 Jan 19 03:00:09 2008 C:\Windows\System32\winrnr.dll
         7fef9660000 4791ad9a Jan 19 02:58:18 2008 C:\Windows\system32\rasadhlp.dll

     

    Note how neither process has MSRDC.DLL loaded either. It’s simply not being used, and a module that is not being used cannot possibly affect anyone. Remember, an application has to be coded to use RDC. Nothing in the Kernel, in Robocopy, or in the Workstation service uses RDC at all in Vista or Win7.

    Still don’t believe me? Here is the Microsoft Remote File Systems development team stating it as well.

    Changes that can truly improve file copy performance

    By now you want me to get to the helpful part. Here’s a short list of some things that can improve your file copy network performance on Windows Vista and Windows Server 2008:

    • Install Service Pack 2
    • Install the latest NIC drivers from your vendor.
    • Try disabling Receive Side Scaling, Chimney Offload, and NetDMA support, then testing like I did above to see if the results are measurably different after many copies of the same file. Note that this just disables the Windows implementation of those components – your vendor may also support them through their NIC configuration and it will need to be turned off there as well. While these components are intended to help performance, mileage can vary based on how good your hardware and vendor drivers are.
    • Use robocopy.exe rather than Explorer – the price of the friendly shell showing progress and browsing folders is slightly slower performance.

    Wrapup

    It’s amazing that a component that was designed to speed up network file performance can somehow be vilified as a cause of bad performance; especially when it’s not even being used. I welcome people following my steps and telling me what you find out.

    Don’t believe everything you read on the Internet. Unless I wrote it. :-)

    - Ned ‘Rick Rolled’ Pyle

  • Migrating from PolicyMaker to Group Policy Preferences with GPPMIG

    Mike here. PolicyMaker customers rejoice—Microsoft has a way for you to migrate from PolicyMaker 2.x to the new Group Policy preferences released with Windows Server 2008 and included in the Remote Server Administration Tools for Windows Vista Service Pack 1 or higher.

    Download GPPMIG: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=35791cb6-710b-48c4-aaa1-90db170bcf2a

    PolicyMaker to Preferences… how to get there

    If you’ve been using PolicyMaker then you already know how to use Group Policy Preferences. It is all managed using the Group Policy Management Console included with Windows Server 2008 or, using Windows Vista Service Pack1 by installing the Remote Server Administration Tools. However, Group Policy preferences cannot process PolicyMaker data and vice versa. Therefore, you need to have a strategy to migrate from PolicyMaker to Group Policy Preferences. Hopefully, this should help. Everything discussed below is also included in the GPPMIG installer as the ‘GPPMIG Migration Guide'.

    Prerequisites

    I want to take a few minutes to discuss some of the prerequisites before we jump right into the migration strategy. We have two categories Management and Client.

    Management

    Policymaker’s management looks and feels the same as managing other Group Policy setting. The same look and feel returns using Preferences. One thing to consider is-- each instance (PolicyMaker or Group Policy Preferences) cannot edit the others data. For this reason, you may need to leave one or more Windows XP computer, with the PolicyMaker administrative tools installed, until you’ve completed your migration. If your migration follows a staged approach, then you may encounter a small period of time where you may need to manage using both Windows Vista and Windows XP. Or, you may be the weekend warrior type and have your migration complete from Friday to Monday. The choice and freedom are there, but the requirement remains—PolicyMaker administrative additions can only edit PolicyMaker items. Server 2008 and the RSAT tools can only edit Preferences. Read Microsoft Knowledgebase article 941314, Description of Windows Server 2008 Remote Server Administration Tools for Windows Vista Service Pack 1 for more information.

    Client

    The critical component that actual makes PolicyMaker and/or Preferences work is the client side extensions (CSEs), which you must install on the client computer. The CSEs make normal Group Policy processing PolicyMaker/Preferences aware. The same rules apply to the client portion—PolicyMaker CSEs only process PolicyMaker data and Preference CSEs only process Preference data. Also, installing the Group Policy Preference CSEs automatically removes PolicyMaker CSEs. The new Group Policy preference client side extensions installs on

    • Windows Vista RTM and Service Pack 1
    • Windows Server 2003 Service Pack 1
    • Windows XP Service Pack 2

    Both Windows Server 2003 and Windows XP require the installation of XmlLite prior to installing the CSEs. Preference CSEs are included in Windows Server 2008. Read Microsoft Knowledgebase article 943729, Information about new Group Policy preferences in Windows Server 2008 for more information.

    Testing

    It goes without saying—you can never test enough and this scenario is not any different. Make sure you have backups… and they actually work. If you are going to use GPMC to backup your GPOs, then remember to use the correct version. GPMC backups are not interchangeable. If you backup with pre-Server 2008 GPMC, then you must restore with the same version. Back up some of your most complex or important GPOs and then important them into isolated test GPOs in a test OU with a single user and computer. Run through your entire migration strategy—noting what works and what does not— refining the plan with each pass. All efforts spent in planning usually pay off during implementation.

    Group Policy Preference Migration utility

    Now that we have the planning stuff out of the way—on to the good stuff. GPPMIG is a console application developed with version 3.0 of the .NET framework. Use GPPMIG to migrate PolicyMaker items to Group Policy Preference items into the same or a different Group Policy object. GPPMIG does not migrate PolicyMaker Application or Mail Profile data as Group Policy Preferences do not included client-side extensions for these items.

    What it does

    Let us take a few moments to discuss how GPPMIG works. For starters, GPPMIG always uses the domain of the currently logged on user. You’ll want to remember this so you can log on with domain administrator account for the domain GPOs you want to migrate. And, you must be a domain administrator as GPPMIG write to SYSVOL and Active Directory. One last point is that GPPMIG always connects to the PDC of the user domain—for reading and writing to Active Directory and SYSVOL. So, you’ll want to run GPPMIG from a computer close (same subnet) as the PDC emulator.

    With GPPMIG, you can target a single GPO to migrate or, you can choose to migrate all GPOs. GPPMIG performs a paged LDAP query to the PDC to retrieves a list of all the Group Policy objects in the user’s domain. GPPMIG then filters out any GPO in the list that is not configured for PolicyMaker items. Then, GPPMIG iterates through each GPO in the final list, looking for PolicyMaker specific client side extensions in each GPO. The entire GPO is evaluated before moving to the next. If a PolicyMaker setting is found, then GPPMIG ensures there is not an equivalent Group Policy Preference configuration, as it will not migrate PolicyMaker items into existing Group Policy Preference items. When GPPMIG completes its search for PolicyMaker items in the GPO, it then updates the Group Policy object to included Group Policy Preference client side extensions and then increases the version number for the user, computer, or both depending on what PolicyMaker items it migrated. In no way does a migration alter any PolicyMaker items for the GPO. All PolicyMaker items remain configured and available in the GPO. GPPMIG creates a migration log in the directory from which it ran.

    Usage

    You can use GPPMIG to migrate to Group Policy Preferences in staged approach or, you can create brand new GPOs to hold your new Group Policy Preference items and migrate to those new GPOs. The staged approach is a planned migration strategy and is the approach I’ll document here. After reading this, you should be able to alternate this strategy to best suit the needs of your environment. Generally, you’ll migrate from PolicyMaker to Group Policy Preferences in three stages (after you’ve done your testing).

    • Stage 1— Identify GPOs containing PolicyMaker items and use GPMC 1.x to back up those GPOs
    • Stage 2— Migrate PolicyMaker items to Group Policy Preference items in the same or a new Group Policy object. Then, deploy the Group Policy Preference CSEs to your client computers.
    • Stage 3— Confirm Group Policy Preference items migrated and are successfully applying to user and computers. Use GPMC 6.0.0.1 to backup your GPOs (to a different back up location then Stage 1. Then remove PolicyMaker items from GPOs, if applicable

    Download GPPMIG: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=35791cb6-710b-48c4-aaa1-90db170bcf2a

    Commands

    GPPMIG contains four basic commands:

    • Whatif — display all the Group Policy objects that contain PolicyMaker items
    • Migrate— migrates PolicyMaker items to Group Policy Preference items in the same GPO
    • MigrateTo— migrates PolicyMaker items to Group Policy Preference items to a different GPO
    • Remove— removes PolicyMaker items from a GPO

    Stage 1 – Identify PolicyMaker GPOs

    Begin your migration process by identifying GPOs containing PolicyMaker items. You can do this by using the –whatif command. Use the –all command afterwards to search all the GPOs in the user’s domain or, you can use the –name command and provided the display name of the GPO. Use GPMC to backup all of the GPOs identified to have PolicyMaker items.

    Stage 2 – Migrate PolicyMaker Data to Group Policy Preferences

    Next, you’ll want to migrate PolicyMaker items to Group Policy Preference items. You have a choice to migrate the setting within the same or to a different Group Policy object.

    clip_image001Note
    The migration does not modify PolicyMaker items, regardless of the migration action you choose.

    Use the –migrate command to migrate PolicyMaker items to Group Policy Preference items within the same GPO. Use the following syntax:

    Gppmig –migrate –name:gpo_name

    Alternatively, you replace the –name argument with –all to migrate all the GPOs in the users domain that contain PolicyMaker items.

    clip_image003

    Figure 1 GPPMIG In-place Migration

    You may prefer to keep PolicyMaker GPOs separate from Group Policy Preference GPOs. You use the –migrateTo command to accomplish this task

    clip_image004Important
    You must create the target GPO before using the -migrateTo command. GPPMIG does create the target Group Policy object.

    clip_image006

    Figure 2 - GPPMIG Source-target Migration

    The –migrateTo command requires two additional arguments: -source: and –target: follows by the display name of the Group Policy object. Enclose the name of the GPO in quotes if the name contains spaces. Also, the –migrateTo command does not support the –all argument.

    Deploy GPP Client

    You’re now ready to deploy the Group Policy Preference client-side extensions after you’ve migrated all of your GPOs to include Group Policy Preference items. The migration does not modify any PolicyMaker items; so clients with the PolicyMaker CSE and the Group Policy preference CSEs process the same data

    clip_image001[1]Note
    GPPMIG does not migrate Application or Mail PolicyMaker items. Therefore, Group Policy Preference CSEs do not apply these items to users or computers. Leave the PolicyMaker CSE installed on computers that require these items and do not install the Group Policy Preference CSEs as the installation removes PolicyMaker CSEs).

    You can apply Group Policy Preferences to several Microsoft operating systems. The minimum operating system requirements are:

    • Windows Vista RTM or Windows Vista Service Pack 1 (32 or 64-bit)
    • Windows Server 2003 Service Pack 1 or later (32 or 64-bit)
    • Windows XP Service Pack 2 or later (32 or 64-bit)

    Group Policy Preference client-side extensions are included in Windows Server 2008. You can use the links above to download the client-side extension installation packages. Or, you can download the extensions as an optional update from Windows Update.

    clip_image004[1]Important
    Remember-- installing Preference client-side extensions removes PolicyMaker Client Side Extensions.

    Stage 3

    The last stage in the migration process involves verifying your items migrated and apply correctly. Use GPMC to view the Group Policy object to which you migrated your items. Click the Settings tab to show the Preference items included in the GPO.

    clip_image008

    Figure 3 GPMC Settings Tab

    Next, you'll want to apply the Group Policy object to your client computers. For in-place migrations, you'll want to apply the GPO to computers using PolicyMaker CSEs and computers using Preference CSEs. Also verify user PolicyMaker and Preference items apply to the appropriate user. GPOs that are targets of in-place migrations should apply items to both (PolicyMaker and Preferences). Source-target migrations migrate the PolicyMaker items to Preference items in the newly created GPO. This allows you to keep your existing PolicyMaker GPOs separate from your Preference GPOs. You apply GPOs containing Preference items to computers are users using the Group Policy Preference CSEs.

    Use the Resultant Set of Policy (RSOP) management console to confirm PolicyMaker items are applying to computers or users. Use the Group Policy Results feature within GPMC to confirm Preference items are applying to computers or users.

    clip_image010

    Figure 4 GPMC Group Policy Results for testuser

    The actual migration from PolicyMaker to Group Policy Preferences is complete. Computers running either Preferences or PolicyMaker should be applying their respective items. Source-target migrations contain both PolicyMaker and Preference items. After you've transitioned your client to use the Group Policy Preference CSEs, you'll want to remove the PolicyMaker data, which remains in the GPO. You can use GPPMIG with the -remove option to remove overlapping PolicyMaker and Preference items.

    clip_image012

    Figure 5 Removing PolicyMaker settings

    clip_image001[2]Note
    GPPMIG does not remove PolicyMaker Application and Mail items from the Group Policy object.

    Source-target Migrations do not included PolicyMaker items. Therefore, once you've completed transitioning your client computers to use Preference CSEs, you can delete the source version of the GPO, which contains only PolicyMaker items.

    Conclusion

    You should consider backing up your Group Policy objects after you've completed your migration and cleanup of Group Policy objects. Use the Group Policy Management Console included in Windows Server 2008 and the Remote Server Administration tools to backup all of your Group Policy objects before you proceed with any further changes.

    - Mike Stephens

  • Recovering from Unsupported One-Way Replication in DFSR Windows Server 2003 R2 and Windows Server 2008.

    Warren here again. The purpose of this blog post is to outline the proper steps to move from an unsupported one-way replication deployment when using DFSR from Windows 2003 R2 and Windows 2008 to a supported configuration of two-way replication.

    Before we get started, here is some good news. Starting with Windows 2008 R2 we will now have Read Only replicas support in DFSR. Details of this feature can be found here:

    https://blogs.technet.com/filecab/archive/2009/01/21/read-only-replicated-folders-on-windows-server-2008-r2.aspx

    Recovering from One Way Replication

    First let’s define some terminology so the rest of this post will make more sense.

    • Upstream server – For this article “Upstream” server will refer to the DFSR server that is able to replicate changes to its partner.
    • Downstream server – For this article the “Downstream server” will refer to the DFSR server that is not allowed to replicate changes to its partner.

    Note: In practice any server in a multi-master replication implementation can be Upstream or Downstream. It all depends on who has the changes “Upstream” and where the change is getting replicated to “Downstream”.

    • RG – Replication Group - a set of servers, known as members, that participates in the replication of one or more replicated folders
    • RF – Replicated Folder - defines a set of files and folders to be kept in sync across multiple servers in a replication group

    See these links for more information on RG’s and RF’s

    http://msdn.microsoft.com/en-us/library/bb540026(VS.85).aspx

    http://technet.microsoft.com/en-us/library/cc759803.aspx

    What Not To Do

    1. You do not want to simply enable replication from the Downstream server to the Upstream server. That is a BAD idea. The reasons why are listed in the section below “Why One-way Replication is not Supported”. You may end up with very old data in your Replicated Folder and some upset users wondering why they are looking at last year’s report.
    2. Do not remove and re-add the Downstream server to the Replication Group again. It is a common misconception that doing this will trigger DFSR to perform an Initial Sync and Initial Replication.

    What To Do

    To successfully recover from a one way replication deployment we must force the downstream server to perform an Initial Synchronization and Initial Replication of the RF so that all queued changes on the Downstream server are discarded.

    I will provide two methods for you to consider. These would be the ones I would use based on key considerations:

    1. How divergent is the data?
    2. How slow are the links?

    Method 1 will generate some file replication traffic as the downstream member is most likely out of date for at least some of its data. If the data is fairly consistent then I would use this method

    Method 2 will generate the least amount of file replication as you will be pre-staging a recent backup of the data taken from the Upstream server on the Downstream. I would use this method if the data is very different or the degree of divergence is unknown but suspected to be high. Also if the links are slow Method 2 is a good choice.

    If you use method 2 make sure you read Ned’s blog post on the proper method of pre-staging files. If you pre-stage the files incorrectly you will generate more replication traffic than if you had just used Method 1.

    http://blogs.technet.com/askds/archive/2008/02/12/get-out-and-push-getting-the-most-out-of-dfsr-pre-staging.aspx

    Whatever you do, always backup the data on your Upstream and Downstream server before making any changes to your one way replication configuration. Backing up your data is a matter of best practice so you should be doing this nightly already. As with any setup of replication, the work should be done off hours or on a weekend when possible to minimize user interruption.

    Patching

    Make sure your DFSR servers patch levels are up to date per the KB Articles linked below before implementing any other changes. Do not skip this point.

    2003 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;958802

    2008 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;968429

    Important Note: DFSR stores its configuration in AD. When changes are made to the DFSR configuration the update will take place on the DC that the DFSR server is connected to. Those changes are then replicated to all DC’s in the domain. DFSR will pick up the change on its own during its next poll of AD.

    In the methods mentioned below all changes to the DFSR configuration will be made on the Upstream server. We will then force AD replication and finally force a poll of AD on the DFSR servers. The steps to do this is listed below and the referenced in both Methods 1 and 2.

    Forcing AD Synchronization

    a. To find out what DC a DFSR server is connected to, use WMIC.

    WMIC /namespace:\\root\microsoftdfs path DfsrReplicationGroupConfig get LastChangeSource

    image

    b. To force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain run this command on the DC returned in step A. i.e <dc name> = the DC returned in step A.

    repadmin /syncall /d /e /P < dc name> <Naming Context>

    image

    c. On the Upstream and Downstream servers run this command. “DFSRDIAG Pollad”.

    image

    Tip: If you want to remote the pollad command you can by specifying the target server with the “/mem:” switch.

    image

    Method 1 – Disabling the Downstream server’s membership in the RF to force Initial Sync and Initial Replication when the membership is re-enabled

    1. Get a full backup of the Replicated Folder(s) from the Upstream server.

    2. If there are changes you want to keep on the Downstream server, back them up now.

    Note: Any files that are different or unique on the downstream will be moved to the ConflictAndDeleted or pre-existing directories. The data in ConflictAndDeleted will be permanently removed if the quota is reached. The quota by default is 660 MB. Make sure you backup the Downstream if there is any data there you want to keep. See http://technet.microsoft.com/en-us/library/cc782648.aspx

    3. On the Upstream sever open the DFSR management snap-in and highlight the Replication Group that has the affected Replicated Folder. Make sure you are on the Memberships Tab. Right click the Downstream server and select Disable.

    image

    Figure 1 - Disabling the Downstream servers membership in the Replication Group

    You will get prompted for verification that you want to disable the membership of the server. Depending on if the RF is published in a DFS Namespace or not you will get a different set of prompts.

    If your RF is not published you will see only the prompt in Figure 2

    Select yes.

    image

    Figure 2 - Disable Membership prompt when the Replicated Folder is not published in a DFS namespace

    If your RF is published in a DFS Namespace you see the prompts in Figure 3 and 4.

    Click yes and Ok.

    image

    Figure 3 First popup you will get disabling membership on an RF that is published via a DFS namespace

    image

    Figure 4 Second popup that you get disabling membership on a RF that is Published via a DFS Namespace

    4. On the Upstream server use the DFS Management console to enable the connection from the Downstream server to the Upstream server. This is done on the Connections Tab. If there is no connection from Downstream to Upstream create it at this time.

    image

    Figure 5 - Enabling the connection from Downstream to Upstream

    5. On the Upstream find out what DC it is connected to -Step A in Forcing AD Synchronization.

    6. Force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain. Step B in Forcing AD Synchronization.

    7. Force the Upstream and Downstream servers to poll AD. Step C in Forcing AD Synchronization.

    Once the downstream server detects the change to its membership the downstream server will then log events 4114 and 4008. Once you confirm these events are logged you can proceed to step 8. Event 4114 informs the admin that the data in the in the RF will be seen as pre-existing. The data will be treated like any other pre-staged data during Initial Sync and Replication when its membership is re-enabled.

    image

    Figure 6 Event ID 4008 logged

    image 

    Figure 7 Event ID 4114

     

     

     

     

     

     

     

     

     

     

    8. Enable the membership of the Downstream server in the Replicated Folder using the DFS Management snap-in. This is located on the Memberships Tab.

    image

    Figure 8 - Enabling the membership of the Downstream server in the Replication Group.

    Depending if your RF is published in a DFS Namespace or not you will a get set of different prompts.

    If the RF is not published in DFS you will get prompted with the dialog box in figure 9. Verify the path and click OK.

    image

    Figure 9 Enabling Membership on a server where the RF is not published via DFS

    If your RF is published in a DFS namespace you will get the popup in figure 10. You will not be able to click OK until you set the share permissions and share name by clicking on the Edit button. Clicking in the Edit button will bring up the popup in figure 11. Set the perms and share names as needed (defaults are fine if suitable). The extra prompts are presented when the RF is published because the member is being added again as a target for the folder in the DFS namespace.

    image

    Figure 10 First popup presented when enabling membership on an RF that is published in DFS.

    image

    Figure 11 Second popup displayed when enabling membership on an RF that is published in DFS

    9. Repeat steps 5-7. After those steps are done the Downstream server will log Event 4102 when Initial Replication begins and 4104 when it is complete. You will also more than likely see 4412 events due to the servers having different versions of some files. If the data is very different on the servers you will see a large amount of these.

    image

    Figure 12 Event ID 4102

    image

    Figure 13 Event ID 4412 “Conflict Event”

    image

    Figure 14 Event 4104

    10. Once you see event ID 4104 on the Downstream server for the replicated folder(s), you are finished with setting up two-way replication.

    Method 2 – Total Recreation of the RG using pre-seeded data on the Downstream server.

    1. On the Upstream server get a full Backup of the replicated folder.

    2. Ship the Backup to the Downstream server’s site.

    3. On the Upstream server delete the Replication Group.

    4. On the Upstream server find out what DC is connected to. – Step A in Forcing AD Synchronization.

    5. Force a synchronization of AD so the changes to DFSR are replicated to all DC’s in the domain. – Step B in Forcing AD Synchronization.

    6. On the Upstream and Downstream servers force them to PollAD. – Step C in Forcing AD Synchronization.

    7. On both the Upstream and Downstream server s you will log events 4010 and 3006 noting that the RF and RG have been removed from the configuration:

    image 

    Figure 15 Event ID 4010 logged when a RF is removed from a RG

    image

    Figure 16 Event ID 3006 Logged when a RG is removed

    8. Pre-seed the backup on the downstream server. See Ned’s blog post on how to do this correctly: http://blogs.technet.com/askds/archive/2008/02/12/get-out-and-push-getting-the-most-out-of-dfsr-pre-staging.aspx. In my experience most pre-seeding attempts that fail are due to mismatched NTFS permissions. The entire directory tree must have matching ACLs or the files will be different and file replication will occur. See Ned’s post for options on how to get ACLS to match. (I like icacls.exe but there are more options)

    9. On the Upstream server recreate the Replication Group and Replicated Folder(s), specifying that the Upstream server is the Primary server for the content. Make sure to set your staging areas to the largest size possible up to the size of the RF if possible.

    10. Repeat Steps 4-6. The upstream server will log event 4112. The Downstream server will log 4102 when initial replication begins and 4104 when it is finished:

    image

    Figure 17 Event ID 4112 logged on the Primary Member of a new RF.

    image

    Figure 18 Event ID 4102 on the downstream when the RF is initialized

    image

    Figure 19 Event 4104 on the downstream when Intial Replication is finished.

    11. Once you see the event ID 4104 on the downstream server Initial Replication is done and you are finished with your task.

    Why One-way Replication is not Recommended or Supported

    Here is a snippet from the blog post by DFSR PM Mahesh Unnikrishnan covering the reasons why it is not recommended nor supported to use one way replication in DFSR in Windows 2003 R2 and Windows 2008. The full post can be found here:

    https://blogs.technet.com/filecab/archive/2007/08/16/using-one-way-connections-in-dfs-replication.aspx

    “We recommend that customers avoid configuring such one way connections to the extent possible since:

    a)       The DFS Replication service’s conflict resolution algorithms are severely hampered if the outbound connection from a member server is deleted (or disabled). Therefore, scenarios where the DFS Replication service is unable to over-write undesired updates occurring on the ‘read-only’ member server with the authoritative contents of the hub/datacenter server may arise.

    b)       Accidental deletions on the ‘read-only’ server (in this case, site server ‘design.contoso.com’) could cause issues with the replication updates being trapped on that server. Further, as described above, updates from the authoritative server can potentially not be applied since the parent folder could have been deleted locally. Therefore, with time it is possible to see substantial divergence in the contents of the replicated folders across all replication member servers.

    c)       Problems with the deployment are difficult to detect without regular and meticulous monitoring. There might be a lot of false positives in the health report and system eventlogs owing to the fact that the replication topology is being set up to do something DFSR wasn’t designed to handle. Mining through these false positives and monitoring servers can be a challenge.

    d)       Administrators need to develop their own scripts to identify which files are backlogged on the ‘read-only’ member (in this case site server ‘design.contoso.com’) and replicate authoritative content back to that ‘read-only’ site server. This can be quite tricky to get right and might need a lot of very close monitoring (perhaps, at times on a per-file basis). Microsoft does not supply any tools for this purpose.

    e)       There is a risk of administrators inadvertently creating the missing connection and causing backlogs to flow to and corrupt the contents of an authoritative server. With these changes getting replicated out further from the authoritative server, the contents of the replicated folder could get out of sync and corrupt on all replication member servers very quickly.

    “Please note that configuring one way connections is not a configuration supported by Microsoft Product Support Services.”

    Reducing Unwanted Changes

    To reduce unwanted changes at certain servers you have a few items in your arsenal. These work for UNC connections.

    1. If it is not necessary to share the data on some servers, don’t share it. DFSR does not require that a replicated folder be shared.

    2. Set Share permissions (not NTFS permissions) to read only.

    3. If you never want anyone accessing the data through the DFS Namespace, disable the referral for that target or delete the server as a folder target.

    Hopefully you will never find yourself in the situation where you need to use the information in this blog post.

    - Warren “Don’t Call Me Warren” Williams

  • The Case of the Random DFS Access Denial

    Hi, this is Bobby from the Microsoft Directory Services support team. Today we are going to discuss some confusing behavior that I’ve observed regarding inconsistent DFS access. While the root of this is issue is easy to remediate, it can be difficult to find the offending servers.

    Symptoms

    When first presented with this issue, I was asked to help determine the nature of users getting prompted to connect to a DFS root. DFS roots are defined in “How DFS Works as

    The starting point of the DFS namespace. The root is often used to refer to the namespace as a whole. A root maps to one or more root targets, each of which corresponds to a shared folder on a separate server. The DFS root must reside on an NTFS volume. A DFS root has one of the following formats: \\ServerName\RootName or \\DomainName\RootName.

    The dialog was as follows:

    image

    Prior to calling Microsoft support, the customer was able to successfully connect to the NETLOGON and SYSVOL share of the domain without issue (\\contoso.com\sysvol and \\contoso.com\netlogon) Rebooting the client machine or having the client logoff would at times alleviate the problem. After a random period of time, users could access the DFS root, would later lose access, and users who did not work before would start working.

    Investigating the issue, it was noticed that the "Distributed File System" MMC would change as time passed. For reference, I have included a picture of what it should look like when all is ok.

    image

    Using the "Distributed File System" MMC to check the status of the root would provide differing results at different times.

    image

    In this picture, it appears that three of the four root targets are inaccessible. When the MMC was checked later (or the user logged off and then back on) the results changed:

    image

    When using the “DFS Management” MMC from 2003 R2, The display is different, but the behavior is the same:

    image

    Initial Troubleshooting

    When we began troubleshooting the issue, we started to look at the output of “dir” and dfsutil /pktinfo. This was to test the access to the DFS root and connectivity to the root shares.

    image

    At this point, each target appeared accessible. This was an unexpected result. I would have expected that we would get access denied to all of the targets. If we could reach each target, why were getting random “access is denied” when attempting to get a directory listing of the root?

    We used the dfsutil /pktflush command to purge the referral cache thus forcing a request for new referrals. Then we ran the dir commands again that returned different results:

    image

    Using Netmon, we took a trace to see what we can find.

    The first thing we attempted was to request a DFS referral for \\contoso.com\root

    image

    We were returned four targets for the root.

    image

    This was represented in the pktinfo as follows:

    image

    At this point we attempted to create a connection to the first server returned in the referral.

    image

    However we were returned: Status_access_denied.

    Normal troubleshooting would be to attempt to connect to the share \\2003-02\root. In this case, we executed dir /b \\2003-02\root.

    When we attempt to do this, we received more unexpected results:

    image

    I would have actually expected this to get an “access is denied” too.

    Looking further in the network trace, we noticed that we received a new referral when we attempted to connect to 2003-02 server directly. To illustrate this, I have several screenshots of the referral network traffic.

    image

    The DFS service returned the following (note that ‘TargetPath: Index:1’ is not 2003-02 but another server):

    image

    Again the pktinfo:

    When connecting to the path \\2003-02\root, we were instead accessing the root share on 2003-01, as detailed in the trace:

    image

    Cause and Remediation

    When connecting to a root DFS share, the DFS service returns referrals for all of the participants on the root target set. So when we executed the command dir \\2003-dc1\root, the DFS service on 2003-dc1 then responded with a new DFS root referral for the path \\2003-dc1\root.  This behavior accounts for the entries in the referral cache for paths to each root (i.e. \\2003-dc1\root, \\2003-01\root, \\2003-02\root, \\2003-03\root). DFS referrals are retuned in a random order for servers in the site of the client. This will cause differencing results for each time the client requests a referral.

    The root cause of the issue was actually incorrect share permissions on the “root” share of 2003-02. This can be observed locally using net share sharename

    image

    Comparing this to the same output of 2003-01, we see the problem:

    image

    In small environments, connecting to the server and examining the share permissions is an acceptable method, however in large environments this can prove cumbersome. I prefer to use subinacl to verify the permissions remotely. (Subinacl can be downloaded here: http://www.microsoft.com/downloads/details.aspx?FamilyID=E8BA3E56-D8FE-4A91-93CF-ED6985E3927B&displaylang=en)

    image

    To fix this we could connect to the server and add the permissions. Or it could be altered remotely with the following:. The syntax is : subinacl /share \\2003-02\root /grant=everyone=r

    Note: Read is the default share permission in 2003 and later. Your permissions may be different based upon your business needs.

    Using DFSGUI.MSC to check the status of the DFS root targets has been deprecated and is no longer available in 2008. This however does not change the behavior of the referrals that are returned.

    Using DFSdiag.exe from Windows 2008 does not return any additional information as to the root cause.

    Further reading

    For more information about the technologies discussed here, please refer to “How DFS Works” http://technet.microsoft.com/en-us/library/cc782417(WS.10).aspx

    - Bobby ‘Bucket Head’ McMillan
  • Implementing an OCSP responder: Part IV - Configuring OCSP for use with Standalone CAs

    Chris here again. In part I of this series we covered the basics of how OCSP works. We also covered the underlying reasons for deploying an OCSP Responder. In Part II we covered configuring the Certificate Authorities for whom which the OCSP Responder will check revocation status for on behalf of the clients. In Part III we covered configuring and OCSP Responder to support an Enterprise CAs. You may use Standalone CAs in your environment. In this blog post, I will be covering deploying a Revocation Configuration to support a Standalone CA.

    Enterprise CAs are very tightly integrated with Active Directory. As such the certificates for the Root CA and for intermediate CAs are published to Active Directory. These certificates are automatically placed in the appropriate certificate stores on the clients. If you publish the Root CA certificate that the issuing CA chains up to; in Active Directory the clients will have that Root CA certificate published to the Trusted Root Certification Authorities container in the user and machine store. If you have not, or do not plan to deploy the Root CA certificate through Active Directory and Group Policy you will need to manually publish the Root Certificates in the Trusted Root Certification Authority store.

    Installing OCSP Responder Role

    The first step is to install the OCSP Responder Role.

    To install the OCSP Responder: Open a command prompt and type: servermanagercmd.exe –install ADCS-Online-Cert

    Requesting and Installing the OCSP Responder Signing Certificate

    The next step is to request the OCSP Response Signing Certificate from the Standalone CA. Since a Standalone CA does not have certificate templates we must manually request the attributes we would like in the certificate. To do this we use a utility called certreq.exe. More information for Certreq is available here: http://technet.microsoft.com/en-us/library/cc736326.aspx.

    To use certreq we must first generate a configuration file. FIgure 1 shows a sample configuration file. The key items that must be included is the OCSP Signing OID, and the OCSP No Revocation Check Extension, otherwise known as the id-pkix-ocsp-nocheck extension.

    image

    Let us take a look at this configuration file.

    · First we have [NewRequest] which is a required section indicating that this is for a new certificate request.

    · Then we have the subject in X.500 format. You can also use the ldap format which is derived from X.500. For example: CN=FCOCSP01,DC=Fourthcoffe,DC=Com. Alternatively, you could use just the common name, such as CN=FCOCSP01.

    · PrivateKeyArchive=False since we will not be archiving the private key.

    · Exportable=True which gives us the option to export the private key if so desired.

    · UserProtected=False which disables strong key protection.

    · MachineKeySet =True which is used to indicte that the resulting certificate will be stored in the machine store.

    · ProviderName=”Microsoft Enhanced Cryptographic Provider v1.0” specifies the Cryptographic Service Provider (CSP) that will be used.

    · UseExistingKey Set=False indicates that this request is for a new certificate, with a new key pair.

    · RequestType=CMC tells certreq to generate the request in CMC format.

    · Then we specify the new section [EnhancedKeyUsageExtension] which indicates what extensions should be placed in the EKU Extension in the certificate. Under that extension we specify that this certificate can be used for OCSP Signing by specifying the OCSP Signing OID (OID=”1.3.6.1.5.5.7.3.9).

    · We then start a new section called [Extensions] and specify that the id-pkix-ocsp-nocheck extension should be included in the certificate.

    Below are the steps for generating the request and installing the signing certificate:

    1. First we use certreq to generate the request file. We specify the configuration file and the output request file. The key pair for this certificate is generated at the same time the request file is created by Certreq.

    image

    2. Next, we must submit the request to the CA. Copy the request file over to the Standalone CA. From the Certification Authority MMC, right click on the CA Name, and select All Tasks from the context menu, and then Submit New Request.

    image

    3. Browse to the request file, and select Open.

    4. The request will then show up in Pending Requests. Right click on the request, and select All Tasks from the context menu, then select Issue.

    image

    5. You will now find the requested Certificate under Issued Certificates. Double click on the certificate to view its properties.

    image

    6. Verify the certificate. Key things to look for here are the presence of the OCSP No Revocation Checking Extension. And that OCSP Signing is specified in the Enhanced Key Usage (EKU) Extension.

    image

    Exporting the Certificate from the CA

    1. First select Copy to File from the Details Tab of the Certificate Properties. This will open the Certificate Export Wizard.

    2. Click Next at the Welcome Screen.

    3. Select DER encoded binary x.509 (.CER), and click Next.

    4. Browse to the location where you which to save the resulting certificate, and give the certificate a name, and click on Save.

    5. Click Finish at the Completing the Certificate Export Wizard screen.

    6. You will be prompted that The export was successful. Click OK.

    Installing the OCSP Response Signing Certificate

    Copy the resulting certificate to the OCSP Server. Open up a command prompt. Navigate to the location where you saved the certificate file, and run certreq –accept <Certificate Name>, to complete the installation of the certificate.

    image

    Configuring Private Key Permissions

    The Online Responder Service runs under the Network Service account. By default the Network Service account does not have access to private keys of certificates located in the Local Computer Personal store. To give the Network Service access, perform the following steps:

    1. Open up the Certificates MMC targeted for the Local Computer.

    2. Right click on the certificate, then select “All Tasks” from the context menu, and then select Manage Private Keys….

    image

    3. Click Add on the Permissions dialog box.

    image

    4. Type Network Service,and then click Check Names to resolve the name. Then click OK.

    image

    5. The Network Service only needs read permissions to the Private Key, so deselect the Allow privilege for Full Control, and verify the Allow privilege is granted for Read, and click OK.

    image

     

    Now that we have installed the OCSP Response Signing certificate, and configured Private Key permissions, we must now configure the Revocation Configuration for the CA, on the OCSP Responder. Open the OCSP Management Console. Follow the following steps to configure the Revocation Configuration:

    1. Right click on Revocation Configuration, and select Add Revocation Configuration from the context menu.

    image

    2. This will start the Add Revocation Configuration wizard. Click Next, when presented with the Getting started with adding a revocation configuration screen.

    image

    3. On the Name the Revocation Configuration screen, give a name to the configuration, and click Next. Note: It is a good idea to name the configuration for the CA server, in case this Responder will be used for multiple CAs.

    image

    4. On the Select CA Certificate Location screen, Select a certificate from the Local certificate store, and click Next.

    image

    5. On the Choose CA Certificate screen, click Browse.

    image

    6. Select the CA certificate, for the CA you are configuring on the OCSP Responder, and click OK.

    image

    7. You will then be returned to the Choose CA Certificate screen. The CA that you selected will be displayed. Click Next to continue.

    image

    8. You will now need to select a signing certificate, on the Select Signing Certificate screen. Select Manually select a signing certificate, and click Next.

    image

    9. You will then be returned to the Revocation Provider screen, click Finish to complete the wizard.

    Assigning the Signing Certificate

    After completing the Wizard, you will notice under the “Revocation Configuration Status” portion of the “Online Responder Configuration” page that the OCSP Configuration that you just added has an error indicating “Bad Signing certificate on Array controller. No need to panic at this point. This error is generated because we have not assigned the OCSP Response Signing certificate yet.

    image

    Now let us go ahead and assign the Signing certificate.

    1. In the OCSP MMC, expand Array Configuration, and click on the name of the OCSP Server. Then in the center pane of the console, select the appropriate Revocation Configuration, then right click on that revocation configuration, and elect Assign Signing Certificate from the context menu.

    image

    2. You will then be prompted select the Signing certificate. Select the appropriate Signing certificate, and click OK.

    image

    At this point you will now see some warnings. If you look under the Revocation Configuration Status for the Revocation Configuration you are configuring, you will notice this error:

    image

    Also, on the Online Responder Configuration page you will notice this error:

    image

    This is due to the fact that the Revocation Provider has not yet been verified. To verify the Revocation Provider, right click on Array Configuration, and select Refresh Revocation Data.

    image

    Once the Revocation Provider has been verified, you should see this under Revocation Configuration Status for the Revocation Configuration you are configuring.

    image

    And that OCSP Signing is specified in the Enhanced Key Usage (EKU) Extension.

    image

    Verify OCSP Configuration

    To verify your ocsp configuration please follow the Verify OCSP Configuration section in Part III of this series.

    Conclusion

    This concludes Part IV of this Series. I hope you enjoyed the first four parts of the series and find them useful. I plan to cover other PKI topics in the near future.

    Implementing an OCSP responder: Part I Introducing OCSP
    Implementing an OCSP responder: Part II Preparing Certificate Authorities
    Implementing an OCSP responder: Part III Configuring OCSP for use with Enterprise CAs
    Implementing an OCSP responder: Part IV Configuring OCSP for use with Standalone CAs 
    Implementing an OCSP Responder: Part V High Availability
    Implementing an OCSP Responder: Part VI Configuring Custom OCSP URIs via Group Policy

    - Chris Delay