Blog - Title

November, 2010

  • iPad / iPhone Certificate Issuance

    Hey all, Rob here again. It’s been a while since I have written a blog post, and this one was too interesting to pass up.

    I recently worked a case around deploying certificates to Apple iPhones and iPads to secure their network communications. The investigation uncovered that Apple devices can get certificates via the Simple Certificate Enrollment Protocol (SCEP), also known in the Microsoft world as Network Device Enrollment Service (NDES) in Windows Server 2008/R2.

    Extreme Disclaimer Warning!!!

    Microsoft obviously does not directly support Apple products of any kind. Your first stop when troubleshooting or configuring any Apple product is:

    http://www.apple.com/support/

    Now that that is out of the way, let’s continue.

    I was unable to find any true corporate support information on Apple’s webpage for the iPad or iPhone, only sales goo. If someone has a proper support link and phone number please send it along so I can update this post. Obviously, we’d rather have you talking to Apple about the iPhone or iPad as they are the authority on how those products should consume certificates.

    Strangely, Apple doesn’t appear to document a step-by-step process for certificate enrollment, so on behalf of some Microsoft customers we had to figure it out. Bear in mind, there may be some changes to this article later.

    Most of this blog is going to be covering the setup and configuration options with NDES to support the solution. If you have worked with MSCEP in the past, not much has changed other than some new registry keys to manage SCEP certificate enrollment. Enrolling for certificates against the old Windows Server 2003 SCEP-Add On utility does not work with Apple devices so Windows Server 2008 or later is required.

    NDES Requirements:

    • Only available on the Enterprise Edition of the Windows Server 2008 or Windows Server 2008 R2 operating systems.
    • Can be installed on the same server as the CA, or on another member server. If you install it on another member server you can configure NDES to use a Windows Server 2003 CA.
    • Requires the installation of the Certification Authority Web Enrollment role service on the NDES Server.

    Installation:

    The Installation of NDES is straight forward, however the steps below assume that you are installing the NDES role for an Enterprise Certification Authority rather than for a Standalone CA. If you are installing this for a Standalone CA certain settings should be skipped. I would encourage you to review the NDES whitepaper for more information.

    1. Launch Server Manager.

    2. Click on Add Roles.

    3. Click the Next button.

    4. Check Active Directory Certificate Services.

    5. Click the Next button twice.

    6. If you are installing the NDES Server on a separate server from the CA, uncheck Certification Authority.

    7. Check Certification Authority Web Enrollment, and Network Device Enrollment Service.

    NOTE: If you see a dialog box about adding required role services for Web Server (IIS), click the Add Required Role Services button.

    8. Click the Next button.

    9. If you are not installing the role on a CA, you will be prompted with the screen shown below. You will need to select the Enterprise CA that should be used for the CA Web Enrollment pages. Click the browse button, and select the appropriate CA. If you want to use CA Web Enrollment Pages on a non-CA, see this blog about web enrollment proxy.

    clip_image002

    10. Click the Next button.

    11. Provide a user account under which to run the IIS application pool account for the NDES web application. It is strongly recommended that you create a domain based service account. This account must be given the Enroll permissions on the certificate template(s) that NDES will use. This gives you the ability to lock down the certificate template so that only devices that use NDES can enrollment for certificates based on this template.

    NOTE: The service account used MUST be added to the IIS_USRS group before attempting to use the account in the wizard.

    clip_image004

    12. Click the Next button.

    13. If you are installing NDES on a server other than the CA, you will be presented with the below screen to select the CA that the NDES web application will submit the requests it receives. Click the Browse button, and select the appropriate CA.

    clip_image006

    14. The next step is to provide information for the Registration Authority certificate to be issued to the web application pool account that we defined at step 11.

    NOTE: The RA Name can be anything you like. The default is the computer’s name concatenated with ”–MSCEP-RA”. This becomes the subject line of the RA certificates.

    clip_image008

    15. Click the Next button.

    16. Select the Signature and Encryption key CSPs to be used, as well as the Key length for the Registration Authority certificates. The default will work fine. If you later decide that you want to change this, review the following blog by Jonathan Stephens.

    17. Click the Next button twice.

    18. Click the Install button.

    NDES Configuration settings:

    NDES configuration settings are stored in the registry. I cover some of the more commonly modified registry keys; for a complete listing of configuration settings please read the NDES Whitepaper.

    The base registry key location NDES reads is:

    HKEY_LOCAL_MACHINE\Software\Microsoft\Cryptography\MSCEP

    All the registry values referenced below are set in this registry key.

    Template Settings

    Use these settings to customize the certificate templates used by NDES.

    SignatureTemplate (REG_SZ)
    EncryptionTemplate (REG_SZ)
    GeneralPurposeTemplate (REG_SZ)

    These three registry keys hold the LDAP name of the template that should be issued for each type of key that the SCEP client could possibly request. There are three types of keys that can be specificed.

    SignatureTemplate: The private key can only be used for creating a digital signature. In the certificate template configuration, this is denoted by the Purpose, Signature, on the Request Handling tab.
    EncryptionTemplate: The private key can be used for encryption. In the certificate template configuration, this is denoted by the Purpose, Encryption, on the Request Handling tab.
    GeneralPurposeTemplate: The private key can be used for both encryption and for creating a digital signature. In the certificate template configuration, this is denoted by the Purpose, Signature and encryption, on the Request Handling tab.

    Here is a screen shot of a certificate template to show where the template name is that needs to be populated in the registry values. In the below figure it is IPSecIntermediateOffline (the default template used by NDES).

    clip_image009

    NOTE: If you decide to use a custom certificate template there are more requirements:

    • The NDES application pool identity needs enroll permissions on the template; this is set on the Security tab when looking at the properties of the template.
    • The template must be valid for computer and not user accounts. You can find out the template type by looking at the properties of the template and clicking on the Extensions tab. Then select the extension Certificate Template Information and you will see Subject type: Computer.

      clip_image011
    • Template Subject Name should be set to Supply in the request. This can be seen by click on the Subject Name tab.

      clip_image013

    Now, let’s continue to look at the NDES configuration settings.

    Password Settings

    Use these settings to configure some of the password behavior in NDES.

    Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Cryptography\MSCEP
    Value: PasswordValidity
    Type: REG_DWORD
    Data: Default 60 (decimal)

    PasswordValidity sets the amount of time (in minutes) for which the NDES admin-supplied password is valid. The default value is 60 minutes, but most admins change this value to something that accommodates the time it takes to communicate the password to the device owner. The device owner enters this password on the device in order to enroll for a certificate.

    A good value might be 0x78h (120 decimal). This will give the owner of the device 2 hours to get through the iPhone configuration utility and set the challenge password. If the validity period expires, and the device owner has failed to obtain a certificate, then the SCEP Admin will need to generate a new challenge for the user.

    Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Cryptography\MSCEP\PasswordMax
    Value: PasswordMax
    Type: REG_DWORD
    Data: Default 5

    PasswordMax sets the number of passwords that the service will track once the NDES admin starts generating passwords. This means that the NDES Admin can get X unique passwords generated at one time. Once the number has been reached the NDES admin will not be able to generate any further passwords until the old ones have been utilized by a device or the password validity has expired.

    You can change the behavior of NDES to force the service to use only one password for all client certificate enrollments. This is used with the UseSinglePassword registry value added in the following hotfix:

    959193 Two improvements are available that shorten the time that is required to manage SCEP certificates by using the Network Device Enrollment Service in Windows Server 2008
    http://support.microsoft.com/kb/959193

    IIS configuration change:

    By default, IIS 7/7.5 security is too restrictive to permit these Apple devices to enroll via SCEP. With the out-of-the-box settings enrollment will fail with the following error in the Application event log:

    Log Name: Application
    Source: Microsoft-Windows-NetworkDeviceEnrollmentService
    Date: {DATE}
    Event ID: 11
    Task Category: None
    Level: Error
    Keywords: Classic
    User: N/A
    Computer: {COMPUTERNAME}

    Description:

    The Network Device Enrollment Service received an http message without the "Operation" tag, or with an invalid "Operation" tag.

    The IIS logs will show the following line when the iPad device attempts to send its certificate enrollment to the NDES server:

    2010-11-04 12:43:38 10.28.40.27 GET /certsrv/mscep/mscep.dll operation=PKIOperation&message=MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJGSIb3DQEHAaCAJIAEggSTMIAG%0 . . . {Shortened for the blog} . . . EMPlcwhmd8c1XAAAAAAAAA%3D%3D%0A 80 - 10.188.117.101 Settings/1.0+CFNetwork/467.12+Darwin/10.3.1 404 15 0 812

    This is a 404.15 (Request Filtering: Denied because query string too long) error and it means that the amount of data being sent in the HTTP URL is larger than what is allowed by default. In the scenario above, the iPad was sending a string over 2700 characters, but the default size allowed by the request filtering is 1024. This is so in order to mitigate against buffer overrun attacks. To change the value you will use the following IIS appcmd.exe command:

    %systemroot%\system32\inetsrv\appcmd.exe set config
      /section:system.webServer/security/requestFiltering /requestLimits.maxQueryString:"3072"
      /commit:apphost

    The command above has been wrapped for clarity.

    Generating the enrollment challenge password:

    Alright, now that you have gone through the effort of configuring NDES you can generate the challenge so that you can add it to the iPhone / iPad configuration.

    1. Launch Internet Explorer and go to:

    http://<NDES Server’s DNS FQDN>/CertSrv/MSCEP_Admin/.

    So you would type something like:

    http://2k8r2-mem1.fabrikam.com/CertSrv/MSCEP_Admin.

    2. You will then see the following screen with the provided challenge:

    clip_image015

    Now you are ready to put the challenge password into the iPhone configuration utility.

    iPhone / iPad configuration:

    Alright, now we get to the “fun” section. As stated earlier, we do not support any of the applications or features shown in this section. If you try things the exact way that I show in this blog and it fails or if the utility is totally different a year from now, or whatever: don’t call us, call Apple. This blog post is about giving our best effort to be helpful, not for Microsoft to become Apple’s support and documentation channel.

    Download and install the iPhone Configuration Utility 3.1 for Windows (never mind the name, it is for iPads also). Now you have an application where you can configure the SCEP settings. After this you will have to deploy the configuration settings to the iPhone / iPad device. Don’t ask us how to do this; Like Melinda, I don’t own any Apple devices for a repro and my customers had to figure it out. Or, step up to the Genius Bar.

    Below is the configuration utility screen with highlighting of the SCEP configuration settings.

    clip_image017

    Here are the settings you should use:

    URL: http://yourNDESServerName.domain.com/certsrv/mscep/mscep.dll

    The URL field should be the DNS name of the NDES server. So if the NDES server name is 2k8r2-mem1.fabrikam.com then the URL should be: http://2k8r2-mem1.fabrikam.com/certsrv/mscep/mscep.dll

    NAME: YOUR CA-NAME

    This should be the CA’s name from which you are requesting a certificate. Keep in mind that most CA names are really friendly names and usually are different than the CA’s computer name. For example: Fabrikam Issuing CA1.

    Subject: O=Fabrikam,OU=IT,CN=Robs IPad

    This is the subject field of the certificate issued to the iPhone/iPad device. It’s is backwards from the usual canonical form you are familiar with but that is the way Apple wants it.

    Subject Alternative Name Type:

    Leave this blank, unless you need some kind of Subject Alternative Name (SAN) on the issued certificate. If you plan on using a subject alternative name, then you will need to run the following commands on the CA before issuing certificates.

    CertUtil -SetReg Policy\EditFlags +EDITF_ATTRIBUTESUBJECTALTNAME2
    net stop certsvc & net start certsvc

    NOTE: These instructions are counter to Microsoft's security best practices with regard to allowing SANs in certificates. Administrators are normally discouraged from enabling the EDITF_ATTRIBUTESUBJECTALTNAME2 EditFlag on an Enterprise CA because of the risk of impersonation attacks. This flag allows a user to specify arbitrary names in a certificate request; an ability that can be abused by persons with malicious intent. In this case, however, the certificate request is being generated by a third party configuration utility, so an Administrator will no control over how the SAN information is included in the request. In order to get this solution to work, the decision was made to accept the possible risk in favor of functionality. In your own enterprise, the results of such a choice may be different.

    The actual risk only exists with those certificate templates that are configured to accept subject information in the request. If the certificate subject is built using information from Active Directory, the subject name information in the request is ignored. The risk can be further mitigated by modifying the permissions on the templates used by NDES so that only the NDES service account is permitted to enroll. The EditFlag is global to the CA, however, so any certificate templates enabled on the CA and configured to accept subject information in the request represent a potential vector of attack.

    For more information on Microsoft's security best practices for allowing SANs in certificates, review this information.

    Challenge:

    This is the password that is given to you from the NDES administrator. If you are the NDES administrator this is the password that you get from visiting the site: http://yourNDESServerName.domain.com/certsrv/mscep_admin/

    Key Size:

    This is the key size of the certificate you are requesting. This value has to be set to the value set in the certificate template.

    Use as digital signature and Use for key encipherment checkboxes:

    Checking / unchecking of these values determines which certificate template is specified by NDES when sending the request to the CA. These are directly related to the registry value settings of SignatureTemplate, EncryptionTemplate, and GeneralPurposeTemplate. What boxes to check depends on the application that needs the certificate.

    Fingerprint:

    Leave this blank.

    Final Thoughts

    Hopefully between this blog and the configuration utility you will be able to successfully issue certificate to the iPhone or iPad. Please keep in mind that we have not discussed anything advanced with NDES, nor have we talked about how to secure this powerful service. Please review the NDES Whitepaper to learn more about these topics.

    Oh….and one last thing. If you have problems or issues with the iPad or iPhone, call Apple.

    Rob “I know the turtle neck comments are coming” Greene

    Gratuitous Windows Phone 7 Plug

  • Friday Mail Sack: General Lejeune Edition

    Hello everybody, Ned here again to share some conversations. This week I talk some SMB security, domain renames, file compression, and DFSR DFSR DFSR!

    Question

    I successfully renamed my domain some years ago and but I find some registry values that reflect the old name. Those values are also present in my test domain where I tested the rename before I renamed the production environment. They are values in HKEY_LOCAL_MACHINE\System\currentcontrolset\Services\NTDS\Parameters, such as "Configuration NC" and "Src Srv objectGuid". Do you know why those keys are not cleaned out or changed by rendom.exe? And how do I tell if a domain has been renamed?

    Answer

    Those value names are not vital and it’s up to LSASS.EXE and DCPROMO.EXE (not rendom.exe) to populate them; there will be a few others, depending on the DC being first created or later added, such as “Src Root Domain Srv” and “Root Domain”. They are used during the DC promotion process but after that they are no longer used. There are a few others like this in DCs – for instance, in the NetLogon key you will often find an abandoned AutoSiteName registry value that never changes to match reality. It’s not great fit and finish, but nothing fatal here. Keep in mind that domain rename came many years after AD was created so it’s unlikely the owners of NTDS ever expected these values to change outside DCPROMO.

    If you want to update them for consistency that’s perfectly ok. After all, some application might decide (incorrectly) that it should use these values to make some decisions rather than correctly querying through APIs. If you want to be ultra-paranoid, you can demote each server in turn, then promote them back up and that will change those values for you. But that is crazy with a capital K.

    As for your other question: as new DC’s are created they have the updated info, so it’s not possible to rely on these registry values to tell if a domain was renamed. If you look at the oldest DCs and see that they have SPNs registered for another domain, that might hint at it. The problem is that DC attrition may eventually remove those traces, and it still doesn’t emphatically mean a rename operation happened – it only shows that someone registered some other domain as SPNs for some reason.

    The best way I can think of is to look at the version of the msDS-UpdateScript attribute on the “cn=partitions,cn=configuration,dc=<domain>object. The only operation that’s supposed to change that attribute is a domain rename and it’s protected from direct alteration by administrators. If it’s version is higher than 1, a rename has likely been done.

    clip_image002

    Question

    KB 951003 talks about how to turn off DFSR file staging compression for certain kinds of files. Does it make sense to also add the Office 2007/2010 .docx, .pptx, and .xlsx suffixes to this list because these file formats are also ZIP files?

    Answer

    It’s an excellent question and I agree: adding those to the exclusion list would generally be a good idea. For instance, here I zipped a large-ish DOCX file that did not include any pictures:

    clip_image002[6]

    The savings with conventional WinZip against the default ZIP compression level of DOCX was only 1%. DFSR will not do any better and you will waste a (small) amount of time, disk, and CPU trying. It's such a good idea that it became the default in Windows Server 2008 R2. :)

    Question

    Can you could share some background information about the "Server SPN target name validation level" GPO setting? With our current information, setting the "Accept if provided by client" seemed to be a reasonable thing to incrementally increase our security configuration on DCs. Unfortunately we immediately hit a compatibility wall with some third party file appliances not working.

    Answer

    We bury this explanation as deep as possible; this setting was added to all supported versions of Windows (XP and later) with KB 973811 but only Vista and later know about it via group policy (security policy). What you are really modifying is the DWORD registry setting:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
    SmbServerNameHardeningLevel

    This article explains the setting and its rules (see the bottom section “More information about this update”, which naturally states nothing about the group policy):

    2345886  Description of the update that implements Extended Protection for Authentication in the Server service - http://support.microsoft.com/default.aspx?scid=kb;en-US;2345886

    If a 3rd party has not updated their products to act more closely like Windows for things like Extended Protection or they are not following RFC2743 or if they are based on Win2000 or older OS emulation, or a host of other issues described in that article, they will fail using settings of 1 or 2. A network capture may show the issue in this case as to see why its failing or if you need to specify SrvAllowedServerNames entries – but it may also be that your appliance will not work until you contact that vendor to see how it must be reconfigured, updated, or that it’s a known issue and the appliance is incompatible.

    Question

    Can I cluster DFSR on Windows Server 2003 R2 or Windows Server 2008 non-R2? [asked by many people and seen in many support cases]

    Answer

    NOOOOOOOOOOOOOOOOOOO!

    Question

    Are you sure I cannot cluster it?

    Answer

    Yes, I am sure. People don’t seem to notice that when they cluster DFSR on Win2003 R2 it explodes their database and never replicates again after a node failover. Not to mention that we obviously spent millions of dollars to create a clustered DFSR in Windows Server 2008 R2 – why bother if it already worked? Plus we say you cannot cluster except on Win 2008 R2 all over the place.

    Question

    We successfully tested DFSR replication of user data across sites in our network. We now want to completely remove DFSR from both of the servers used in testing as they are going to be repurposed and I need to ensure I get back all the disk space, CPU, and memory. How can I make sure it’s totally removed?

    Answer

    Good question, we don’t document this (who would ever want to remove DFSR after all?!?! :-P ). This assumes you already removed Replication Groups or removed the members from replication, obviously:

    clip_image002[10] 

    clip_image002[12]

    To fully remove DFSR, you need to do the following:

    1. Uninstall the DFSR role via Server Manager (on Win2008/R2) or Add/Remove Windows Components (Win2003):

    image

    image 

    2. Delete the folder(s) that you were replicating.

    3. If using Win2008/R2, give yourself access to the hidden operating system folder “System Volume Information” on the C: drive with this command (that should be run as an elevated Admin user; if on Win2003 the Admin will already have permissions):

    icacls "c:\system volume information" /grant Administrators:F

    4. Repeat step 3 on any drives that had replicated folders, changing the path to that drive.

    5. In that CMD prompt (don’t use Windows Explorer) you can then delete the “DFSR” folder that was left behind on those drives. That contains all the staged, pre-existing, and conflicted files in 2008/R2. It also contains the DFSR database and XML files in all versions of Windows. For example:

    Rd “c:\system volume information\dfsr” /s /q

    6. Repeat as needed on any other drives.

    7. Done.

    The reason you need to use a CMD prompt on Win2008/R2 is based on how Explorer special-cases the SVI folder and prevents anyone (even admins with full control permissions) from deleting its contents. You have to want it real bad, basically. Win2003 just lets you delete it arbitrarily because he is a punk.

     

    Other stuff

    Thanks to everyone who sent along nice emails about the Think Positive post, there were quite a few of you and I may not have responded to everyone. If you want to give those pleasant folks in Melbourne a little jolt of cash to keep their campaign going, click here.

    They also sell the posters, but I will be sticking with what I already have up in my cubicle. I can only be so positive before I have to return to my roots…

    big_trouble_in_little_chinammgbu 
    No one at DePaul gets this anymore

     

    Finally, to all the former and present US Marines, happy 235th birthday from an old leatherneck. In keeping with the tradition started in 1921, here is the 13th Commandant’s message:

    On November 10, 1775, a Corps of Marines was created by  a resolution of Continental
    Congress. Since that date many thousand men have borne the name "Marine". In memory of them it is
    fitting that we who are Marines should commemorate the birthday of our corps by calling to mind the
    glories of its long and illustrious history.

    The record of our corps is one which will bear comparison with that of the most famous
    military organizations in the world's history. During 90 of the 146 years of its existence the
    Marine Corps has been in action against the Nation's foes. From the Battle of Trenton to the
    Argonne, Marines have won foremost honors in war, and in the long eras of tranquility at home,
    generation after generation of Marines have grown gray in war in both hemispheres and in every
    corner of the seven seas, that our country and its citizens might enjoy peace and security.

    In every battle and skirmish since the birth of our corps, Marines have acquitted themselves
    with the greatest distinction, winning new honors on each occasion until the term "Marine" has come
    to signify all that is highest in military efficiency and soldierly virtue.

    This high name of distinction and soldierly repute we who are Marines today have received
    from those who preceded us in the corps. With it we have also received from them the eternal spirit
    which has animated our corps from generation to generation and has been the distinguishing mark of
    the Marines in every age. So long as that spirit continues to flourish Marines will be found equal
    to every emergency in the future as they have been in the past, and the men of our Nation will
    regard us as worthy successors to the long line of illustrious men who have served as "Soldiers of
    the Sea" since the founding of the Corps.

    Have a nice weekend folks,

    - Ned “0341” Pyle

  • Yeowza, 0x1f4 articles!

    And I just noticed that the last post put us to our 500th article. That’s a post every 205,200 seconds! Wait, that doesn’t sound impressive. That’s a post every 2.3 days! OK, not quite Raymond Chen or Keith Combs, but it will have to do.

    You have kept us going with your comments, questions, and readership – thanks very much.

    - Ned “User profile cannot be loaded” Pyle

  • New Directory Services KB Articles

    Hi everyone, we have a few new and updated articles from last week that are Directory Services related. 

    Content ID

    Title

    951581

    LDAP queries are executed more slowly than expected in the AD or LDS/ADAM directory service and Event ID 1644 may be logged

    933430

    Clients cannot make connections if you require client certificates on a Web site or if you use IAS in Windows Server 2003

    979682

    Microsoft Security Advisory: Vulnerability in Windows Kernel could allow elevation of privilege

    2300745

    A memory leak occurs in Svchost.exe when you apply or update Group Policy preference settings in Windows Server 2008

    975363

    A time-out error occurs when many NTLM authentication requests are sent from a computer that is running Windows Server 2008 R2, Windows 7, Windows Server 2008, or Windows Vista in a high latency network

  • New ADFS Content on TechNet Wiki (11/16/2010)

    Hello everyone!

    Adam has published a new round of content for Active Directory Federation Services (ADFS) to the TechNet Wiki. These articles include troubleshooting information and how-tos to assist you when you are evaluating, implementing, or troubleshooting ADFS.

    AD FS 2.0 - How to change the Federation Service Name 

    AD FS 2.0 - How to capture a log during installation (AdfsSetup.exe) 

    AD FS 2.0 setup fails to install PowerShell feature on Windows Server 2008 

    Windows Identity Foundation (WIF) - A potentially dangerous Request.Form value was detected from the client (wresult="<t:RequestSecurityTo..."). 

    Windows Identity Foundation (WIF) - How to protect static content with the Federation Authentication Module (FAM) 

    AD FS 2.0 - Federation Server Proxy servers fail to authenticate users - Events 248 and 996 logged 

    AD FS 2.0 - How to change the local authentication type

     -- Jonathan "Ned's Blog Monkey" Stephens

     

  • Reading LDAP SSL Network Traffic with NetMon 3.4 and NMDecrypt

    Hi folks, Ned here again. Today I show you how to decrypt LDAP traffic protected by SSL by using Network Monitor and its handy add-on NetMon Decryption Expert. This is useful when you need to see what an application is asking your domain controllers, especially when that app has lousy logging. Since the traffic is all encrypted on the wire, ordinary network captures are useless.

    I’m not going to explain LDAP and SSL as we already have some great articles by James, Mike, Rob, Randy, and Dave here:

    http://blogs.technet.com/b/askds/archive/tags/ldap/

    This is also possible to do in WireShark but naturally, you’re on your own there. Let them write some darned blog posts for once.

    I like screenshots!

    What you need

    First download your tools:

    Install these on the computer that’s talking LDAP; this could be the DC or a member server or a client or whatever. If you are troubleshooting a non-Windows OS then the DC is your only choice, obviously.

    Note: NMDecrypt has its own support channels with a discussion group and issue tracker. They also have the people you can contact. They are probably not AskDS readers so using our Comments section is not the best use of your time.

    Getting the data

    It goes without saying that this is all being done in your test lab, mmmmmkay. Don’t have a test lab? Come back later when you do.

    1. Fire up NetMon, pick your network(s), and start capturing without filters.

    2. Make the application start sending encrypted LDAP traffic. Naturally, you won’t be able to easily capture an LDAP application running on a DC itself, so use at least two computers to test. In my examples, I use LDP.EXE because it’s the closest thing to a “pure” LDAP client in Windows; PowerShell, ADSIEDIT, and other tools usually go through various levels of abstraction. LDP also obeys orders unlike many third party apps. You can download LDP with the Win2003 Support Tools. It’s included in Win2008/2008 R2/Vista RSAT/7 RSAT.

    So in my example, I start up LDP.EXE and connect with port 636 and SSL set. Then I bind with my credentials and navigate (i.e. LDAP query) through a few levels until I return data for a user named Alice Scott.

    image 

    image

    image

    image

    image

    In my example, this returned lots of unreadable data:

    image

    3. Filter your capture display by the IP address of the computer sending LDAP traffic and by “TLS”. This allows us to see the SSL handshake process, including the “Server Hello”:

    image

    The “Server Hello” is the response frame that tells the application which certificate is being used by LDAP to create the SSL-encrypted session. Examine the frame’s “Cert” field, and then drill down to the “SerialNumber” and “IdCeSubjectAltName” fields. These give important certificate uniqueness info that we can use later to identify which certificate was being used on the DC. Below I have an example – you will have completely different values obviously:

    image

    image

    image

    image

    4. Now decrypt the traffic and this requires an exported PFX file copy of the certificate that was used on the DC to encrypt this traffic. Things can get gummy here so pay close attention.

    A. Start MMC.EXE on that DC and add the Certificates snap-in for the Computer store:

    image

    B. Examine the certificates. There may be a bunch or only one. You need to open each certificate and look at the details tab for the “Serial Number” and “Subject Alternative Name” fields. The certificate that matches your network capture data is the one you used.

    image

    image

    image

    C. Now right-click that certificate and select All Tasks then Export. This is where things get sticky – if the certificate has “yes, export the private key” grayed out, you cannot proceed. So the screenshot below is bad news:

    image

    When you decrypt the network capture later you will need a server authentication certificate PFX file that includes the private key info. So if the certificate being used doesn’t allow this, you will need to issue a new certificate that does allow private key export.

    To fix this you change whatever server authentication certificate template that is being used for issuing to your DC to allow private key exports. In my example I have altered the built-in Domain Controller Authentication template to do this – I am doing this out of convenience in a lab, not need. You may decide to do something totally different. For instance, creating a special certificate for this troubleshooting is a good idea; only that DC could be allowed to enroll to minimize your risk, based on the template security. This can all be done through the Certificate Templates snap-in in an MMC console on your certificate authority.

    image

    If you are using certs issues by a third party that you have no control over, stop here: you cannot decrypt these captures. And you paid for that privilege… :-/

    Having ironed all that out, request or auto-enroll that certificate to this DC so that it can be used for decryption. The old server authentication certificate will need to be deleted or it will simply keep being used. If it is being used for other things or if its deletion might break something, you need to stop now and take stock of this entire exercise. But since you’re in a test lab (right?!?!) it’s probably fine to zap it.

    D. If you had to get a new certificate, you need to repeat all the steps leading up to here and confirm that your new certificate was used. At this point you can now export the certificate with the private key in a PFX format. Make sure you put the file somewhere safe and that it is named appropriately. You must set a password and it should be a good one.

    image

    image

    image

    5. Make sure you have the “All Traffic” node selected in the Network Connections pane – not “My Traffic” or any conversations. Now you decrypt the traffic with NMDecrypt. Click:

    Experts –> NMDecrypt –> Run Expert.

    image

    6. NMDecrypt makes you save a copy of your capture. Select your saved PFX file by browsing the “server Certificate Path” and enter the password. Specify an output capture file in the “decrypted file path” field. You don’t have to specify a log unless you get errors with decryption.

    image

    7. Click “Start” and the new CAP file will be created with all the SSL/TLS traffic decrypted and readable. This capture stands on its own so you can give it to others without the need to provide any certificates. To see the good stuff, use the following display filter in the capture:

    DecryptedPayloadHeader

    image

    Now everything is readable (the filter removes a lot of the chaff created by the expert to render the translated frames). That previous capture was just a series of “Encrypted over SSL” message. In my example now, I can see all this attribute data being returned for the Users container and Alice Scott when I was querying with LDP. Sweet!

    image

    image

    Naturally, everything I talked about today works for SSL-encrypted traffic in general. HTTPS web traffic can be analyzed with the same technique, for instance.

    Don’t forget to delete that exported certificate when you’re done!

    Common Mistakes

    • Error “SSL Frames are not found in the current capture file

    This means that you did not select the “All Traffic” node when you started the expert. It must be selected.

    image

    • Error “Invalid Key Exchange data or Invalid Certificate

    This means you exported the wrong certificate and tried to decrypt with it. Go back to step 3.

    Important Side Note

    If you are allergic to network captures and the LDAP is coming from a Windows Vista or later computer, you can use ETW to leverage client tracing to see traffic before decryption or after decryption. Rather than reinvent those steps I recommend you examine these excellent write-ups:

    Until next time.

    - Ned “sizzle” Pyle

  • Happy 25th Birthday Windows (tomorrow)

    On November 20th, 1985 Windows 1.0 launched. It fit on just two double-sided floppy disks and needed 256K of RAM – although you needed 512K if you wanted to actually run more than one program at a time.

    Within a year it had some incredible new features like postscript printer drivers and MS-DOS 3.2 support! And check out this beeeeaaaauuuutiful grill:

    Windows1_0
    Man, that looked sweet

    It wasn’t until Windows 3.0 that things really started to take off and NT where we had an architectural version you would recognize today – but this is where it all began. So play a game of Reversi and let your thoughts return to the years of 16-bit tiled shell cooperative multi-tasking.

    Then you can use your Windows Phone 7 to watch high definition movies streamed from NetFlix while you wait for your boss to email you some stupid presentation while cruising at 35,000 feet.

    Happy birthday Windows!

     

    - Ned “I was still a Tandy guy at this point” Pyle

    PS: Windows also has the same birthday as my brother. Almost 30, Nick! :)

  • Common DFSR Configuration Mistakes and Oversights

    Greetings everyone. Warren here again with a blog post that is a collection of common DFSR issues that I have encountered over the past few years. The goal of this blog post is to list these common DFSR configuration mistakes that have caused problems to prevent you making the same mistake. Knowing what not to do is as important as knowing what to do. Many of these points are addressed in other content so links are provided for more in-depth reading.

     

    • Too small of a Staging Area Quota

    Are you seeing a lot of event ID’s 4202 and 4204? If so, your staging area is not sized correctly. The downside to an improperly sized staging area is that replication performance will be negatively affected as the service has to spend time cleaning up the staging area instead of replicating files.

    DFSR servers are more efficient with a full staging area for at least these two reasons:

    1. It is much more efficient to stage a file once and send it to all downstream partners than to stage a file, replicate the file, purge for each downstream partner.
    2. If at least one member is running Enterprise Edition the servers can take advantage of Cross File RDC

    An improperly sized staging area can also cause a replication “loop” to occur. This condition happens when a file get replicated to the downstream server and is present in the staging however the file is purges by the staging area cleanup process before the file can be installed into the Replicated Folder. The purged file will be replicated again to the server that just purged it from its staging as it never got to install the file. This process will keep repeating until the file gets installed.

    Don’t ignore staging area events.

    See this blog post for the method to use to determine your minimum staging area size

    See the section “Increase Staging Quota” here

    See “Remote Differential Compression details” here for information on Cross File RDC

    • Improper or Untested Pre-seeding procedure

    Pre-seeding is the act of copying the data that will be replicated to a new replica member before they are added to the Replicated Folder with the goal of reducing the amount of time it takes to complete initial replication. Most failed pre-seeding cases I have worked on failed in 3 ways.

    1. ACL mismatch between source and target.
    2. Changes were made to the files after they were copied to the new member
    3. No testing was done to verify the pre-seeding process they were using worked as expected.

    In short the files must be copied in a certain way, you cannot change the files after they are staged and you must test your process.

    Click here to read Mr. Pyle’s blog on how to properly pre-seed your DFSR servers

    • High backlogs for extended periods of time

    Besides the fact that having high backlogs for extended periods of time means your data is out of sync, it can lead to unwanted conflict resolution where a file with older content wins in a conflict resolution scenario. The most common way I have seen this condition hit is when rolling out new RF’s . Instead of doing a phased rollout some admins will add 20 new RF’s from 20 different branch offices at once overloading their hub server. Stagger your rollouts so that initial replication is finished in a reasonable amount of time.

    • DFSR used as a backup solution

    Believe it or not some admins run DFSR without backing up the replicated content offline. DFSR was not designed as a backup solution. One of DFSR’s design goals is to be part of an enterprise backup strategy in that it gets your geographically distributed data to a centralized site for backup, restore and archival. Multiple members do offer protection from server failure; however, this does not protect you from accidental deletions. To be fully protected you must backup your data.

    • One way Replication – Using it, or Fixing One way replication the wrong way

    In an attempt to prevent unwanted updates from occurring on servers where they know the data will never be changed (or they don’t want changes made there) many customers have configured one-way replication by removing outbound connections from replica members. One-way replication is not supported on any version of DFSR until Windows Server 2008 R2. On Windows 2008 R2 one-way replication is supported provided you configure Read-Only replicated folders. Using Read –Only DFSR members allows you to accomplish the goal of one-way replication, which is preventing unwanted changes to replicated content. If you must use one-way replication with DFSR then you must use Windows 2008 R2 and mark the members as read-only where changes to content should not occur.

    Click here and here to learn about Read-Only DFSR replicas.

    Another common problem that occurs is when an Admin discovers that one way replication is not supported they go about fixing it the wrong way. Simply enabling two-way replication again can have undesirable results. See the blog post below on how to recover from one-way replication.

    Click here to learn about fixing one-way replication.

    • Hub Server - Single Point of Failure or Overworked Hub Servers

    I have seen many deployments with just one hub server. When that hub server fails the entire deployment is at risk. If you’re using Windows Server 2003 or 2008 you should have at least 2 hub servers so that if one fails the other can handle the load while the other is repaired with little to no impact to the end users. Starting with Windows Server 2008 R2 you can deploy DFSR on a Windows Failover Cluster which gives you high availability with half of the storage requirement.

    Other times admins will have too many branch office servers replicating with a single hub server. This can lead to delays in replication. Knowing how many branch office servers a single hub server can service is a matter of monitoring your backlogs. There is no magic formula as each environment is unique and there are many variables.

    Read the “Topology Tuning” section here for ideas on deploying hub servers.

    Click here to learn how to setup DFSR a Windows Server 2008 Fail-Over Cluster.

    • Too many Replicated Folders in a single Jet Database

    DFSR maintain one Jet database per volume. As a result placing all of your RFs on the same volume puts them all in the same Jet Database. If that Jet database has a problem that requires repair or recovery of the database all of the RF’s on that drive are affected. It is better to spread the RFs out using as many drives as possible to provide maximum uptime for the data.

    • Bargain Basement iSCSI deployments

    I have seen more than one DFSR iSCSI deployment where the cheapest hardware available was used. Usually if you are using DFSR it is for some mission critical purpose such as data redundancy, backup consolidation, pushing out applications and OS upgrades on a schedule. Depending on low-end hardware with little to no vendor support is not a good plan. If the data is important to your business then the hardware that runs the OS and replication engine is important to your business.

    • Did not maintain the DFSR service at the current patch level

    DFSR is actively maintained by Microsoft and has updates released for it as needed. You should update DFSR when a new release is available during your normal patch cycle. Make sure your servers are up to date per the KB articles listed below.

    Windows 2003 R2 DFSR Hotfixes

    Windows 2008 and Windows 2008 R2 DFSR Hotfixes

    You will notice that updates are listed for NTFS.SYS and other files besides DFSR.EXE/DFSRS.EXE. For replication always make sure DFSR and NTFS are at least at the highest version listed. The other patches listed mostly deal with UI issues that you will at minimum want installed on the systems you perform DFSR configuration tasks on.

    Proactively patching the DFSR servers is advisable even if everything is running normally as it will prevent you from getting effected by a known issue.

    • Did not maintain NIC Drivers

    DFSR will only work as well as the network you provide for it. Running drivers that are 5 years old is not smart. Yes, I have talked with more than a few customers with NIC drivers that old who’s DFSR replication issue was resolved by updating the NIC driver.

    • Did not monitor DFSR

    Despite the fact that the data DFSR is moving around is usually mission critical many Admins have no idea what DFSR is doing until they discover a problem. Savvy admins have created their own backlog scripts to monitor their servers but a lot of customers just “hoped for the best”. The DFSR Management Pack has been out for almost a year now (and other versions for much longer). Deploy it and use it so you can detect problems and respond before they become a nightmare. If you can’t use the DFSR Ops Management pack at a minimum write a script to track your backlogs on a daily basis so that you know that DFSR is replicating files or not.

    Click here for information on the Ops Manager DFSR Management Pack.

    Updated Jan 19, 2011:

    • Making changes to disk storage without first backing up the data

      If you must replace or add hard drive space to your DFSR server it is critical that you have a current backup of the data in case something goes wrong. Any number if things can go wrong the most common being conflict event s created due to accidental changes to a parent folder or unintentional deletion of a parent folder that is replicated to all partners. You must backup your data before starting the changes and maintain the backup until the project is completed. 
       
    • Stopping the DFSR service to temporarily prevent replication

      Sometimes there is a need to temporarily stop replication. The proper way to accomplish this is to set the schedule to no replication for the Replication Group in question. The DFSR service must be running to be able to read updates to the USN Journal. Do not stop the DFSR service for long periods of time (days, weeks) as doing so may cause a journal wrap to occur (if many files are modified, added, or deleted in the meantime). DFSR will recover from the journal wrap but in large deployments this will take a long time and replication will not occur or will happen very slowly during the journal wrap recovery. You may also see very high backlogs until the journal wrap recovery completes.

    I hope this list has been a help to you. Happy replicating.

    Warren “wide net” Williams