Blog - Title

August, 2010

  • The Case of the Enormous CA Database

    Hello, faithful readers! Jonathan here again. Today I want to talk a little about Certification Authority monitoring and maintenance. This topic was brought to my attention by a recent case that I had where a customer’s CA database had grown to rather elephantine proportions over the course of many months quite unbeknownst to the administrators. In fact, the problem didn’t come to anyone’s attention until the CA database had consumed nearly all of the 55 GB partition on which it resided. How many of you may be in this same situation and be completely unaware of it? Hmmm? Well, in this post, I’ll first go over the details of the issue and the steps we took to resolve the immediate crisis. In the second part, I’ll cover some processes and tools you can put in place to both maintain your CA database and also alert you to possible problems that may increase its size.

    The Issue

    Once upon a time, Roger contacted Microsoft Support and reported that he had a problem. His Windows Server 2003 Enterprise CA database, which had been given its own partition, had grown to over 50 GB in size, and was still growing. The partition itself was only 55 GB in size, so Roger asked if there is any way to compact the CA database before the CA failed due to a lack of disk space.

    Actually, compacting the CA database is a simple process, and while this isn’t a terribly common request we’re pretty familiar with the steps. What made this case so unusual was the sheer size of the database file. Previously, the largest CA database I’d ever seen was only about 21 GB, and this one was over twice that size! But no matter. The principles are the same regardless, and so we went to it.

    Compacting the CA Database

    Compacting a CA database is essentially a two-step process. The first step is to delete any unnecessary rows from the CA database. This will leave behind what we call white space in the database file that can be reused by the CA for any new records that it adds. If we just removed the unneeded records the size of the database file would not be reduced, but we could be confident that the database file would grow no larger in size.

    If the database file were smaller, this might be an acceptable solution. In this case, the size of the database file relative to the size of the partition on which it resided mandated that we also compact the database file itself.

    If you are familiar with compacting the Active Directory database on a domain controller, then you will realize that this process is identical. A new database file is created and all the active records are copied from the old database file to the new database file, thus removing any of the white space. When finished, the old database file is deleted and the new file is renamed in place with the name of the old file. While actually performing the compaction, Certificate Services must be disabled.

    At the end of this process, we should have a significantly smaller database file, and with appropriate monitoring and maintenance in the future we can ensure that it never reaches such difficult to manage proportions again.

    What to Delete?

    What rows can we safely delete from the CA database? First, you need to have a basic understanding of what exactly is stored in the CA database. When a new certificate request is submitted to the CA a new row is created in the database. As that request is processed by the CA the various fields in that row are updated and the status of each request at a particular point in time describes at what point in the process the request is. What are the possible states for each row?

    • Pending - A pending request is basically on hold until an Administrator manually approves the request. When approved, the request is re-submitted to the CA to be processed. On a Standalone CA, all certificate requests are pended by default. On an Enterprise CA, certificate requests are pended if the option to require CA Manager approval is selected in the certificate template.
    • Failed - A failed request is one that has been denied by the CA because the request isn’t suitable per the CA’s policy, or there was an error encountered while generating the certificate. One example of such an error is if the certificate template is configured to require key archival, but no Key Recovery Agents are configured on the CA. Such a request will fail.
    • Issued - The request has been processed successfully and the certificate has been issued.
    • Revoked - The certificate request has been processed and the certificate issued, but the administrator has revoked the certificate.

    In addition, issued and revoked certificates can either be time valid or expired.

    These states, and whether or not a certificate is expired, need to be taken into account when considering which rows to delete. For example, you do not want to delete the row for a time valid, issued certificate, and in fact, you won’t be able to. You won’t be able to delete the row for a time valid, revoked certificate either because this information is necessary in order for the CA to periodically build its certificate revocation list (CRL).

    Once a certificate has expired, however, then Certificate Services will allow you to delete its row. Expired certificates are no longer valid on their face, so there is no need to retain any revocation status. On the other hand, if you’ve enabled key archival then you may have private keys stored in the database row as well, and if you delete the row you’d never be able to recover those private keys.

    That leaves failed and pending requests. These rows are just requests; there are no issued certificates associated with them. In addition, while technically a failed request can be resubmitted to the CA by the Administrator, unless the cause of the original failure is addressed there is little purpose in doing so. In practice, you can safely delete failed requests. Any pending requests should probably be examined by an Administrator before you delete them. A pending request means that someone out there has an outstanding certificate request for which they are patiently waiting on an answer. The Administrator should go through and either issue or deny any pending requests to clear that queue, rather than just deleting the records.

    In this customer’s case, we decided to delete all the failed requests. But first, we had to determine exactly why the database had grown to such huge proportions.

    Fix the Root Problems, First

    Before you start deleting the failed requests from the database, you should ensure that you have addressed any configuration issues that led to these failures to begin with. Remember, Roger reported that the database was continuing to grow in size. It would make little sense to start deleting failed requests -- a process that requires that the CA be up and running -- if there are new requests being submitted to the CA and subsequently failing. The rows you delete could just be replaced by more failed rows and you’ll have gained nothing.

    In this particular case, we found that there were indeed many request failures still being reported by the CA. These had to be addressed before we could actually do anything about the size of the CA database. When we checked the application log, we saw that Certificate Services was recording event ID 53 warnings and event ID 22 errors for multiple users. Let’s look at these events.

    Event ID 53

    Event ID 53 is a warning event indicating that the submitted request was denied, and containing information about why it was denied. This is a generic event whose detailed message takes the form of:

    Certificate Services denied request %1 because %2. The request was for %3. Additional information: %4

    Where:

    %1: Request ID
    %2: Reason request was denied
    %3: Account from which the request was submitted
    %4: Additional information

    In this particular case, the actual event looked like this:

    Event Type:   Warning

    Event Source: CertSvc

    Event Category:      None

    Event ID:     53

    Date:         <date>

    Time:         <time>

    User:         N/A

    Computer:     <CA server>

    Description:

    Certificate Services denied request 22632 because The EMail name is unavailable and cannot be added to the Subject or Subject Alternate name. 0x80094812 (-2146875374).  The request was for CORP02\jackburton.  Additional information: Denied by Policy Module

    This event means that the certificate template is configured to include the user’s email address in the Subject field, the Subject Alternative Name extension, or both, and that this particular user does not have an email address configured. When we looked at the users for which this event was being recorded, they were all either service accounts or test users. These are accounts for which there would probably be no email address configured under normal circumstances. Contributing to the problem was the fact that user autoenrollment had been enabled at the domain level by policy, and the Domain Users group had permissions to autoenroll for this particular template.

    In general, one probably shouldn’t configure autoenrollment for service accounts or test accounts without specific reasons. In this case, simple User certificates intended for “real” users certainly don’t apply to these types of accounts. The suggestion in this case would be to create a separate OU wherein user autoenrollment is disabled by policy, and then place all service and test accounts in that OU. Another option is to create a group for all service and test accounts, and then deny that group Autoenroll permissions on the template. Either way, these particular users won’t attempt to autoenroll for the certificates intended for your users which will eliminate these events.

    For information on troubleshooting other possible causes of these warning events, check out this link.

    Event ID 22

    Event ID 22 is an error event indicating that the CA was unable to process the request due to an internal failure. Fortunately, this event also tells you what the failure was. This is a generic event whose detailed message takes the form of:

    Certificate Services could not process request %1 due to an error: %2. The request was for %3. Additional information: %4

    Where:

    %1: Request ID
    %2: The internal error
    %3: Account from which the request was submitted
    %4: Additional information

    In this particular case, the actual event looked like this:

    Event Type:   Error

    Event Source: CertSvc

    Event Category:      None

    Event ID:     22

    Date:         <date>

    Time:         <time>

    User:         N/A

    Computer:     <CA server>

    Description:

    Certificate Services could not process request 22631 due to an error: Cannot archive private key.  The certification authority is not configured for key archival. 0x8009400a (-2146877430).  The request was for CORP02\david.lo.pan.  Additional information: Error Archiving Private Key

    This event means that the certificate template is configured for key archival but the CA is not. A CA will not accept the user’s encrypted private key in the request if there are no valid Key Recovery Agent (KRA) configured. The fix for this is pretty simple for our current purposes; disable key archival in the template. If you actually need to archive keys for this particular template then you should set that up before you start removing failed requests from your database. Here are some links to more information on that topic:

    Key Archival and Recovery in Windows Server 2003
    Key Archival and Recovery in Windows Server 2008 and Windows Server 2008 R2

    Template, Template, Where’s the Template?

    What’s the fastest way to determine which template is actually associated with each of these events? You can find that by looking at the failed request entry in the Certification Authority MMC snap-in (certsrv.msc). If you have more than a couple hundred failed requests, however, find the one you actually want can be difficult. This is where filtering the view comes in handy.

    1. In the Certification Authority MMC snap-in, right-click on Failed Requests, select View, then select Filter….

    clip_image001

    2. In the Filter dialog box, click Add….

    clip_image002

    3. In the New Restriction dialog box, set the Request ID to the value that you see in the event, and click Ok.

    clip_image003

    4. In the Filter dialog box, click Ok.

    clip_image004

    5. Now you should see just the failed request designated in the event. Right-click on it, select All Tasks, and then select View Attributes/Extensions….

    clip_image005

    6. In the properties for this request, click on the Extensions tab. In the list of extensions, locate Certificate Template Information. The template name will be show in the extension details.

    clip_image006

    This is the name of the template whose settings you should review and correct, if necessary.

    Once the root problems causing the failed requests have been resolved, monitor the Application event log to ensure that Certificate Services is not logging any more failed requests. Some failed requests in a large environment are expected. That’s just the CA doing its job. What you’re trying to eliminate are the large bulk of the failures caused by certificate template and CA misconfiguration. Once this is complete, you’re ready to start deleting rows from the database.

    Deleting the Failed Requests

    The next step in this process is to actually delete the rows using our trusty command line utility certutil.exe. The -deleterow verb, introduced in Windows Server 2003, can be used to delete rows from the CA database. You just provide it with the type of records you want deleted and a past date (if you use a date equal to the current date or later, the command will fail). Certutil.exe will then delete the rows of that type where the date the request was submitted to the CA (or the date of expiration, for issued certificates) is earlier than the date you provide. The supported types of records are:

    Name

    Description

    Type of date

    Request

    Failed and pending requests

    Submission date

    Cert

    Expired and revoked certificates

    Expiration date

    Ext

    Extension table

    N/A

    Attrib

    Attribute table

    N/A

    CRL

    CRL table

    Expiration date

     

     

     

     

    For example, if you want to delete all failed and pending requests submitted by January 22, 2001, the command is:

    C:\>Certutil -deleterow 1/22/2001 Request

    The only problem with this approach is that certutil.exe will only delete about 2,000 - 3,000 records at a time before failing due to exhaustion of the version store. Luckily, we can wrap this command in a simple batch file that runs the command over and over until all the designated records have been removed.

    @echo off

    :Top

    Certutil -deleterow 8/31/2010 Request

    If %ERRORLEVEL% EQU -939523027 goto Top

    This batch file runs certutil.exe with the -deleterow verb. If the command fails with the specific error code indicating that the version store has been exhausted, the batch file simply loops and the command is executed again. Eventually, the certutil.exe command will exit with an ERRORLEVEL value of 0, indicating success. The script will then exit.

    Every time the command executes, it will display how many records were deleted. You may therefore want to pipe the output of the command to a text file from which you can total up these values and determine how many records in total were deleted.

    In Roger’s case, the total number of deleted records came to about 7.8 million rows. Yes…that is 7.8 million failed requests. The script above ran for the better part of a week, but the CA was up and running the entire time so there was no outage. Indeed, the CA must be up and running for the certutil.exe command to work as certutil.exe communicates with the ICertAdmin COM interface of Certificate Services.

    That is not to say that one should not take precautions ahead of time. We increased the base CRL publication interval to seven days and published a new base CRL immediately before starting to delete the rows. We also disabled delta CRLs temporarily while the script was running. We did this so that even if something unexpected happen, clients would still be able to check the revocation status of certificates issued by the CA for an extended period, giving us the luxury of time to take any necessary remediation steps. As expected, however, none were required.

    And Finally, Compaction

    The final step in this process is compacting the CA database file to remove all the white space resulting from deleting the failed requests from the database. This process is identical to defragmenting and compacting Active Directory’s ntds.dit file, as the Certificate Services uses the same underlying database technology as Active Directory -- the Extensible Storage Engine (ESE).

    Just as with AD, you must have free space on the partition equal to or greater than the database file size. As you’ll recall, we certainly didn’t have that in this case what with a database of 50 GB on a 55 GB partition. What do you do in this case? Move the database and log files to a partition with enough free space, of course.

    Fortunately, Roger’s backing store was on a Storage Area Network (SAN), so it was trivial to slice off a new 150 GB partition and move the database and log files to the new, larger partition. We didn’t even have to modify the CA configuration as Roger’s storage admins were able to just swap drive letters since the only thing on the original partition was the CertLog folder containing the CA database and log files. Good planning, that.

    With enough free space now available, all is ready to compact the database. Well…almost. You should first take the precaution of backing up the CA database prior to starting just in case something goes wrong. The added benefit to backing up the CA database is that you’ll truncate the database log files. In Roger’s case, after deleting 7.8 million records there were several hundred megabytes of log files. To back up just the CA database, run the following command:

    C:\>Certutil -backupDB backupDirectory

    The backup directory will be created for you if it does not already exist, but if it does exist, it must be empty. Once you have the backup, copy it somewhere safe. And now we’re finally ready to proceed.

    To compact the CA database, stop and then disable Certificate Services. The CA cannot be online during this process. Next, run the following command:

    C:\>Esentutl /d Path\CaDatabase.edb

    Esentutl.exe will take care of the rest. In the background, esentutl.exe will create a temporary database file and copy all the active records from the current database file to the new one. When the process is complete, the original database file will be deleted and the temporary file renamed to match the original. The only difference is that the database file should be much smaller.

    How much smaller? Try 2.8 GB. That’s right. By deleting 7.8 million records and compacting the database, we recovered over 47 GB of disk space. Your own mileage may vary, though, as it depends on the number of failed requests in your own database. To finish, we just copied the now much smaller database and log files to the original drive and then re-enabled and restarted Certificate Services.

    While very time consuming, simply due to the sheer number of failed requests in the database, overall the operation went off without a hitch. And everyone lived happily ever after.

    Preventative Maintenance and Monitoring

    Now that the CA database is back down to its fighting weight, how do you make sure you keep it that way? There are actually several things you can do, including regular maintenance and, if you have the capability, closer monitoring of the CA itself.

    Maintenance

    You’ll remember that it was not necessary to take the CA offline while deleting the failed requests. We did take precautions by modifying the CRL publication interval but fortunately that turned out to be unnecessary. Since no outage is required to remove failed requests from the CA database, it should be pretty simple to get approval to add it to your regular maintenance cycle. (You do have one, right?) Every quarter or so, run the script to delete the failed requests. You can do it more or less often as is appropriate for your own environment.

    You don’t have to compact the CA database each time. Remember, the white space will simply be reused by the CA for processing new requests. Over time, you may find that you reach a sort of equilibrium, especially if you also have the freedom to delete expired certificates as well (i.e., no Key Archival), where the CA database just doesn’t get any bigger. Rows are deleted and new rows are created in roughly equal numbers, and the space within the database file is reused over and over -- a state of happy homeostasis.

    If you want, you can even use scheduled tasks to automatically perform this maintenance every three months. The batch file above can be modified to run using VBScript or even PowerShell. Simply add some code to email yourself a report when the deletion process is finished; there are plenty of code samples available on the web for sending email using both VBScript and PowerShell. Bing it!

    Monitoring

    In addition to this maintenance, you can also use almost any monitoring or management software to watch for certain key events on the CA. Those key events? I already covered two of them above -- event IDs 53 and 22. For a complete list of events recorded by Certificate Services, look here.

    If you have Microsoft Operations Manager (MOM) 2005 or System Center Operations Manager (SCOM) 2007 deployed, and you have Windows Server 2008 or Windows Server 2008 R2 CAs, then you can download the appropriate management pack to assist you with your monitoring.

    MOM 2005: Windows Server 2008 Active Directory Certificate Services Management Pack for Microsoft OpsMgr 2005
    SCOM 2007 SP1: Active Directory Certificate Services Monitoring Management Pack

    The management packs encompass event monitoring and prescriptive guidance and troubleshooting steps to make managing your PKI much simpler. These management packs are only supported for CAs running on Windows Server 2008 or higher, so this is yet one more reason to upgrade those CAs.

    Conclusion

    Like any other infrastructure service in your enterprise environment, the Windows CA does require some maintenance and monitoring to maintain its viability over time. If you don’t pay attention to it, you may find yourself in a situation similar to Roger’s, not noticing the problem until it is almost too late to do anything to prevent an outage. With proper monitoring, you can become aware of any serious problems almost as soon as they begin, and with regular maintenance you prevent such problems from ever occurring. I hope you find the information in this post useful.

    Jonathan “Pork Chop Express” Stephens

  • New Directory Services Content 8/22-8/28

    KB Articles

    There are several articles below that are related to support of Single Label Domains, Disjointed Namespaces and Discontiguous Namespaces.  These will soon be linked to the DNS Namespace Planning Solution Center.  More on that here.

    Article ID

    Title

    2273047

    User Account Control (UAC) and Windows Explorer

    2360265

    KRB_AP_ERR_BAD_INTEGRITY error when server tries to delegate in mixed Read-Only DC and Windows Server 2003 DC environment

    2328240

    Event ID 4107 or 11 is logged in the Application Log in Windows Vista or Windows Server 2008 and later

    2030310

    TerminalServices-Licensing 4105 – The Terminal Services license server cannot update the license attributes for user “<UserName>” in Active Directory Domain “<DomainName>”

    2269838

    Microsoft Exchange compatibility with Single Label Domains, Disjointed Namespaces, and Discontiguous Namespaces

    2379064

    Microsoft Biztalk Server compatibility with Single Label Domains, Disjointed Namespaces, and Discontiguous Namespaces

    2379369

    Microsoft Office Communications Server compatibility with Single Label Domains, Disjointed Namespaces, and Discontiguous Namespaces

    2379367

    Microsoft Forefront compatibility with Single Label Domains, Disjointed Namespaces, and Discontiguous Namespaces

    2379371

    Microsoft Office Outlook compatibility with Single Label Domains, Disjointed Namespaces, and Discontiguous Namespaces

    2379373

    Microsoft Office SharePoint compatibility with Single Label Domains, Disjointed Namespaces, and Discontiguous Namespaces

    2379375

    Microsoft SQL Server compatibility with Single Label Domains, Disjointed Namespaces, and Discontiguous Namespaces

    2379380

    Microsoft System Center product compatibility with Single Label Domains, Disjointed Namespaces, and Discontiguous Namespaces

    Blogs

    Title

    Moving Your Organization from a Single Microsoft CA to a Microsoft Recommended PKI

    Forcing Afterhours User Logoffs

    Don't mess about with USMT's included manifests!

    ACT: Suppressing Elevation Prompts for Legacy Applications

    Use the DirectorySearcher .NET Class and PowerShell to Search Active Directory

    Use the PowerShell [adsiSearcher] Type Accelerator to Search Active Directory

    Query Active Directory and Ping Each Computer in the Domain by Using PowerShell

    Query Active Directory with PowerShell and Run WMI Commands

  • No more yellow banner (and the gallery of things that are not work)

    Well, my wife complained one too many times about the old blog image. Now it’s a nice soothing blue with a more worldwide bend, plus some updated “artwork” courtesy of Visio and MS Paint (sorry for cutting you off Australia & New Zealand, sacrifices had to be made to the pixel gods). I also updated our About page to reflect modern times, such as how we’re support for ADFS and AppLocker. Obviously, no mail sack this week; things were just too busy and a lot of questions were repeats. I do have a new DFSR series in the works that I believe many of you will find useful, look for that to start soon. I believe Jonathan has some more PKI things underway as well.

    Enough of work. Here’s some stuff that has nothing to do with pleasing your boss:

    image

    image[40]

     

     

     

     

     

    image[51]

    image[46]

    image[53]

    Ned “back on duty” Pyle

  • How Not to Interview

    Our excellent colleague Keith Combs found this movie created by Microsoft Learning.  It is both hilarious and sometimes frighteningly true. Really…

    To avoid being in a movie yourself:

    Then come and get it.

    - Ned “I was the jacket guy in my interview” Pyle

  • Don’t mess about with USMT’s included manifests!

    Ned here. Today I talk about the importance of the included USMT component manifests and how things can get gross when they are not available to Scanstate and Loadstate.

    Here’s you

    1. You are using USMT 4.0 to migrate from XP to Windows 7.
    2. You run scanstate on XP and it appears to work fine (no unexpected errors to console or logs).
    3. You run loadstate on Win7 and it appears to work fine (no unexpected errors to console or logs).
    4. A zillion system settings are not migrated, so the user does not get their mapped printers and drives, shell settings, RAS settings, favorites, regional settings, etc; all the stuff here
    • Accessibility settings
    • Address book
    • Command-prompt settings
    • Desktop wallpaper
    • EFS files
    • Favorites
    • Folder options
    • Fonts
    • Group membership.
    • Windows Internet Explorer® settings
    • Microsoft® Open Database Connectivity (ODBC) settings
    • Mouse and keyboard settings
    • Network drive mapping
    • Network printer mapping
    • Offline files
    • Phone and modem options
    • RAS connection and phone book (.pbk) files
    • Regional settings
    • Remote Access
    • Taskbar settings
    • Windows Mail.
    • Microsoft Outlook Express Mail (.dbx) files are migrated from Windows XP.
    • Windows Media Player
    • Windows Rights Management

        5.   Everyone runs around screaming.

    What’s Up?

    The problem is that USMT cannot locate the folder containing the migration manifest files. There are two of these included with USMT 4:

    DlManifests
    ReplacementManifests

    clip_image002

    Here’s what will happen if they are not present:

    • If running scanstate and loadstate with a full migration on XP and going to Vista/Windows 7 and you were missing the DLManifests folder, none of these settings I described earlier will migrate. Your log will show:

    [0x000000] Downlevel Manifests folder is not present. System component settings will not be gathered.

    • If running scanstate /genconfig without the DlManifests or ReplacementManifests folders, your config.xml file will not have any WindowsComponents settings in it at all and will only contain the ErrorControl entries, like so:

    <?xml version="1.0" encoding="UTF-8"?>

    <Configuration>

      <Applications/>

      <Documents/>

      <WindowsComponents/>

      <Policies>

        <ErrorControl>

          <!--   Example:

     

              <fileError>

                <nonFatal errorCode="33">* [*]</nonFatal>

                <fatal errorCode="any">C:\Users\* [*]</fatal>

              </fileError>

              <registryError>

                <nonFatal errorCode="5">* [*]</nonFatal>

              </registryError>

          -->

        </ErrorControl>

        <HardLinkStoreControl>

          <!--   Example:

     

              <fileLocked>

                <createHardLink>c:\Users\* [*]</createHardLink>

                <errorHardLink>C:\* [*]</errorHardLink>

              </fileLocked>

          -->

        </HardLinkStoreControl>

      </Policies>

      <ProfileControl>

        <!--   Example (local group mapping):

     

              <localGroups>

                <mappings>

                  <changeGroup from="Administrators" to="Users" appliesTo="MigratedUsers">

                    <include>

                      <pattern>DomainName1\Username</pattern>

                    </include>

                    <exclude>

                      <pattern>DomainName2\Username</pattern>

                    </exclude>

                  </changeGroup>

                </mappings>

              </localGroups>

             

          -->

        <!--   Example (domain and user mapping):

     

              <domains>

                <domain from="Domain1" to="Domain2"/>

              </domains>

             

              <users>

                <user from="Domain1\User1" to="Domain2\User2"/>

              </users>

             

          -->

      </ProfileControl>

    </Configuration>

    • If running scanstate and loadstate from Vista/7 to Vista/7, you may see various settings not work or other niggles not behaving as expected. Your log will show:

    [0x000000] The ReplacementManifests folder used to service system component manifests is not present. OS settings migration will be done with system component manifests installed onto the system.

    This is because these manifests are included by the component owners to handle special scenarios within migration, so you are falling back to using the manifests included in the OS for upgrades (which uses some of the same underlying USMT engine code). Your mileage may vary. A lot.

    Nevermind that, what do I do?

    The folders are often missing due to administrative misadventure – i.e. someone forgot to copy them and only copied the base files included in the USMT main folder. If you make sure they are present in the USMT working folder, you will be good to go. They were included for good reason reason and USMT is only tested with those files being included, so definitely do not intentionally discard them.

    There are also a few reasons why they might not accidentally be found, based on how USMT was shelled. Both SCCM and MDT have bugs on this that are described and worked around:

    • MDT bug with workaround - 
    • SCCM bug with workaround

    I’m not sure if those folks have plans to fix these permanently or what, the KB’s don’t indicate.

    Note: Micheal Niehaus let me know offline that MDT is fixed in MDT 2010 Update 1 and the KB just needs to be update to reflect this. Thanks Michael. :-)

    That’s all for now.

    - Ned “precious fontses!!!” Pyle

  • Forcing Afterhours User Logoffs

    Mike here and today I want to answer a common customer request—how to force users to logoff at the end of the day. The scenario requires a bit of an explanation, so let’s get started.

    Let’s recognize the value of forcing users to logoff at the end of their work day, rather than simply allowing them to lock their computer. Locking their computer leaves many processes running. Running processes keep files open. Open files may introduce problems with synchronizing user data with Offline Files, home folders and distributing user content to other replica targets. Also, roaming user profiles are updated only at logoff (with the exception of Windows 7 background upload of the ntuser.dat, which must be turned on through policy). Allowing users to remain logged on after hours provides little benefit (aside from people like Ned, who does not sleep for fear of clowns may eat him).

    clip_image002
    Everybody floats down here…

    We force an after hour logoff using two Group Policy Preference Scheduled Task items. We’ll configure the items from a Windows Server 2008 R2 computer. Our targeted client computers are Windows 7 and Windows Vista. The typical business work day begins around 8am and ends between 5 and 6 pm. For this scenario, we’ll presume our workday ends at 5 pm. Our first scheduled task notifies the user the computer will shut down in 15 minutes. The second scheduled task actually shutdowns the computer.

    Notify the user

    We use the first scheduled task to notify the user they will be logged off in 15 minutes. This gives the user a reasonable amount of time to save their work. Ideally, users will save their work and logoff or shut down the computer within this allow time (once they understand their computer will log them off regardless). Our Group Policy Preference items target users; so, we’ll open GPMC and create a new Scheduled Task (Windows Vista or later) preference item.

    clip_image003

    We use the Update action for the Preference item and name the item DisplayLogoffMessage. The Update action creates the new scheduled task if it does not exist, or updates an existing task with the current configuration. Under the Security option select %LogonDomain\LogonUser% and select Run only when user is logged on.

    clip_image004

    Next, we need to configure when the event triggers. For this scenario, we want the event to trigger daily, at 5 pm. Also, ensure the status for the task is set to Enabled. Next, we’ll configure the action that occurs when the event triggers.

    clip_image005

    Select Display a message for the action. Type Afterhours Logoff in the Title box. In the Message box, type Windows will logoff your session in 15 minutes. Please save your work. Click OK.

    Force the logoff

    We’ve notified the user. Now we need actually force the logoff. We’ll use a new Schedule Task (Windows Vista or later) preference item.

    clip_image006

    We’ll configure the General tab similar to the previous preference item. We’ll use Update for the Action. The Name and Description can vary; however, understand that name is the criterion used to determine if the scheduled task exists on the applying computer. The only change we’ll make in the Triggers configuration is the time. We should configure this preference item should start at 5:15 pm.

    clip_image008

    The Action for our new preference item is going to Start a program. The program we’ll use is LOGOFF.EXE, which is included with Windows and resides in the System folder. We represent this by using a Group Policy Preference variable. In the Program/script: box, type %SystemDir%\logoff.exe. The LOGOFF.EXE program does not require any arguments.

    We should have two Scheduled Task Preference item. The DisplayLogoffMessage should be ordered first and the Force_afterhours_logoff should be second. The only remaining configuration is to link the Group Policy object hosting these preference items to a point in Active Directory so it applies to user objects.

    clip_image010

    On to the client

    Users on Windows 7 computers will process the above settings without any additional configuration. However, Windows Vista computers, including those running Service Pack 1 need the latest Group Policy Preference Client Side Extension (http://support.microsoft.com/kb/974266).

    At 5 pm, the scheduled task triggers Windows to display a message to the user.

    clip_image012

    Fifteen minutes after the message, Windows will then end all the running applications and log off the user.

    clip_image014

    This is actually the hardest part of the scenario. However, there is one additional configuration we must perform on the user account to complete the solution.

    We need to configure Logon Hours for the user. The Logon Hours should be configured to prevent the user from logging on the computer after we’ve forcefully logged them out. In this scenario, we forcefully log off the user at 5:15 pm; however, we’ve configured their user account so their logon hours deny them from logging on past 5 pm. Windows prevents the logon and displays a message to the user explaining they are not allowed to logon at this time.

    clip_image016

    Conclusion

    The scenario explains how to administratively force a user session logoff to your environment. If users are members of the local Administrators group, then all bets are off. The only way to prevent an administrator from doing something is not to make them an administrator.

    Alternatively, you can slightly modify this scenario to force a computer shutdown rather than a user logoff. Windows includes SHUTDOWN.EXE, with a variety of command arguments. This may be the most optimal form of power management because a powered down computer uses the least amount of energy. Also, forcing shutdowns will force users to save their work before leaving, which helps with making sure centralized backups have the most current and accurate user data.

    Mike “Nice Marmot” Stephens

  • New Directory Services Content 8/15-8/21

    KB Articles

    Only one new KB article of interest this week:

    Number

    Title

    2385108

    Accessing a folder under a mapped network drive to a Distributed File System Namespace (DFSN) folder target may fail on Windows Vista and later with error “Access is Denied”

    Blogs

    Title

    Friday Mail Sack: Scooter Edition

    Fine-Grained Password Policy and “Urgent Replication”

    Best Practice: Roaming Profiles and Folder Redirection (a.k.a. User State Virtualization)

    How to Change a User’s Active Directory Password with PowerShell

    Use PowerShell to Add Domain Users to a Local Group

    Using PowerShell to determine your elevation status (UAC)

    Generating a report on Distribution Groups and their Membership

    Using the Remote Desktop Services BPA to analyze a Remote Desktop Gateway implementation

    Two Very Important Attributes with Active Directory Recycle Bin

    PowerShell Cookbook V2 Now Available

  • Moving Your Organization from a Single Microsoft CA to a Microsoft Recommended PKI

    Hi, folks! Jonathan here again, and today I want to talk about what appears to be an increasingly common topic: migrating from a single Windows Certification Authority (CA) to a multi-tier hierarchy. I’m going to assume that you already have a basic understanding of Public Key Infrastructure (PKI) concepts, i. e., you know what a root CA is versus an issuing CA, and you understand that Microsoft CAs come in two flavors – Standalone and Enterprise. If you don’t know those things then I recommend that you take a look at this before proceeding.

    It seems that many organizations had installed a single Windows CA in order to support whatever major project that may have required it. Perhaps they were rolling out System Center Configuration Manager (SCCM), or wireless, or some other certificate consuming technology and one small line item in the project’s plan was Install a CA. Over time, though, this single CA began to see a lot of use as it was leveraged more and more for purposes other than originally conceived. Suddenly, there is a need for a proper Public Key Infrastructure (PKI) and administrators are facing some thorny questions:

    1. Can I install multiple PKIs in my forest without them interfering with each other?
    2. How do I set up my new PKI properly so that it is scalable and manageable?
    3. How do I get rid of my old CA without causing an interruption in my business?

    I’m here to tell you that you aren’t alone. There are many organizations in the same situation, and there are good answers to each of these questions. More importantly, I’m going to share those answers with you. Let’s get started, shall we?

    Important Note: This blog post does not address the private key archival scenario. Stay tuned for a future blog post on migrating archived private keys from one CA to another.

    Multiple PKIs In The Forest? Isn’t That Like Two Cats Fighting Over the Same Mouse?

    Uh….no.

    (You know, I actually considered asking Ned to find some Office clip art that showed two cats fighting over a mouse, and then thought, “What if he found it?!” I decided I didn’t really want to know and bagged the idea.)

    To be clear, there is absolutely no issue with installing multiple Windows root CAs in the same forest. You can deploy your new PKI and keep it from issuing certificates to your users or computers until you are good and ready for it to do so. And while you’re doing all this, the old CA will continue to chug along oblivious to the fact that it will soon be removed with extreme prejudice.

    Each Windows CA you install requires some objects created for it in Active Directory. If the CA is installed on a domain member these objects are created automatically. If, on the other hand, you install the CA on a workgroup computer that is disconnected from the network, you’ll have to create these objects yourself.

    Regardless, all of these objects exist under the following container in Active Directory:

    CN=Public Key Services, CN=Services, CN=Configuration, DC=<forestRootPartition>

    As you can see, these objects are located in the Configuration partition of Active Directory which explains why you have to be an Enterprise Admin in order to install a CA in the forest. The Public Key Services Container holds the following objects:

    CN=AIA Container

    AIA stands for Authority Information Access, and this container is the place where each CA will publish its own certificate for applications and services to find if needed. The AIA container holds certificationAuthority objects, one for each CA. The name of the object matches the canonical name of the CA itself.

    CN=CDP Container

    CDP stands for CRL Distribution Point (and CRL stands for Certificate Revocation List). This container is where each CA publishes its list of revoked certificates to Active Directory. In this container, you’ll find another container object whose common name matches the host name of the server on which Certificate Services is installed – one for each Windows CA in your forest. Within each server container is a cRLDistributionPoint object named for the CA itself. The actual CRL for the CA is published to this object.

    CN=Certificate Templates Container

    The Certificate Templates container holds a list of pKICertificateTemplate objects, each one representing one of the templates you see in the Certificate Templates MMC snap-in. Certificate templates are shared objects, meaning they can be used by any Enterprise CA in the forest. There is no CA-specific information stored on these objects.

    CN=Certification Authorities Container

    The Certification Authorities container holds a list of certificationAuthority objects representing each root CA trusted by the Enterprise. Any root CA certificate published here is distributed to each and every member of the forest as a trusted root. A Windows root CA installed on a domain server will publish its certificate here. If you install a root CA on a workgroup server you’ll have to publish the certificate here manually.

    CN=Enrollment Services

    The Enrollment Services container holds a list of pKIEnrollmentService objects, each one representing an Enterprise CA installed in the forest. The pKIEnrollmentService object is used by Windows clients to locate a CA capable of issuing certificates based on a particular template. When you add a certificate template to a CA via the Certification Authority snap-in, that CA’s pKIEnrollmentService object is updated to reflect the change.

    Other Container

    There are few other objects and containers in the Public Key Services container, but they are beyond the scope of this post. If you’re really interested in the nitty-gritty details, post a comment and I’ll address them in a future post.

    To summarize, let’s look at a visual of each of these objects and containers and see how they fit together. I’ve diagrammed out an environment with three CAs. One is the Old And Busted CA, which has been tottering along for years ever since Bob the network admin put it up to issue certificates for wireless authentication.

    Now that Bob has moved onto new and exciting opportunities in the field of food preparation and grease trap maintenance after that unfortunate incident with the misconfigured VLANs, his successor, Mike, has decided to deploy a new, enterprise-worthy PKI.

    To that end, Mike has deployed the New Hotness Root CA, along with the More New Hotness Issuing CA. The New Hotness Root CA is an offline Standalone root, meaning it is running the Windows CA in Standalone mode on a workgroup server disconnected from the network. The New Hotness Issuing CA, however, is an online issuing CA. It’s running in Enterprise mode on a domain server.

    Let’s see what the AD objects for these CAs look like:

    clip_image001

    Figure 1: Sample PKI AD objects

    We’ve come an awful long way to emphasize one simple point. As you can see, each PKI-related object in Active Directory is uniquely named, either for the CA itself or the server on which the CA is installed. Because of this, you can install a (uniquely named) CA on every server in your environment and not run into the sort of conflict that some customers fear when I talk to them about this topic. You could also press your tongue against a metal pole in the dead of winter. Of course, it would hurt, and you’d look silly, but you could do it. Same concept applies here.

    So what’s the non-silly approach?

    The Non-Silly Approach

    If you need to migrate your organization from the Old And Busted CA to the New Hotness PKI, then the very first thing you should do is deploy the new PKI. This requires proper planning, of course; select your platform, locate your servers, that sort of thing. I encourage you to use a Windows Server 2008 R2 platform. WS08R2 CAs are supported with a minimum schema version of 30 which means you do not need to upgrade your Windows Server 2003 domain controllers. More details are here.

    Once your planning is complete, deploy your new PKI. Actual step-by-step guidance is beyond the scope of this blog post, but it is pretty well covered elsewhere. You should first take a look at the Best Practices for Implementing a Microsoft Windows Server 2003 Public Key Infrastructure. Yes, I realize this was for Windows Server 2003, but the concepts are identical for Windows Server 2008 and higher, and the scripts included in the Best Practices Guide are just as useful for the later platforms. It is also true that the Guide describes setting up a three tiered hierarchy, but again, you can easily adapt the prescriptive guidance to a two tiered hierarchy. If you want help with that then you should take a look at this post.

    The major benefit to using Windows Server 2008 or higher is a neat little addition to the CAPolicy.INF file. When you install a new Enterprise CA it is preconfigured with a set of default certificate templates for which it is ready immediately to start issuing certificates. You don’t really want the CA to issue any certificates until you’re good and ready for it to do so. If the Enterprise CA wasn’t configured with any templates by default then it wouldn’t issue any certificates after the CA starts up. When you were ready to switch over to the new PKI, you’d just configure the issuing CA with the appropriate templates. It turns out that as of Windows Server 2008 you can install an Enterprise issuing CA so that the default certificate templates were not automatically configured on the CA. You accomplish this by adding a line to the CAPolicy.inf file:

    LoadDefaultTemplates=False

    Now, if at this point you’re wondering, “What is a CAPolicy.INF file, and how is it involved in setting up a CA,” then guess what? That is your clue that you need to read the Best Practices Guide linked above. It’s all in there, including samples.

    “Oh…but the samples are for Windows Server 2003,” you say, accusingly. Relax; here’s a blog post I wrote earlier fully documenting the Windows Server 2008 R2 CAPolicy.INF syntax. Again, the concepts and broad strokes are all the same; just some minor details have changed. Use my earlier post to supplement the Best Practices Guide and you’ll be golden.

    I Have My New PKI, So Now What?

    So you have your new PKI installed and you’re ready to migrate your organization over to it. How does one do that without impacting one’s organization too severely?

    The first thing you’ll want to do is prevent the old CA from issuing any new certificates. You just uninstall it, of course, but that could cause considerable problems. What do you think would happen if that CA’s published CRL expired and it wasn’t around to publish a new one? Depending on the application using those certificates, they’d all fail to validate and become useless. Wireless clients would fail to connect, smart card users would fail to authenticate, and all sorts of other bad things would occur. The goal is to prevent any career limiting outages so you shouldn’t just uninstall that CA.

    No, you should instead remove all the templates from the Certificate Templates folder using the Certification Authority MMC snap-in on the old CA. If an Enterprise CA isn’t configured with any templates it can’t issue any new certificates. On the other hand, it is still quite capable of refreshing its CRL, and this is exactly the behavior you want. Conversely, you’ll want to add those same templates you removed from the Old And Busted CA into the Certificate Templates folder on the New Hotness Issuing CA.

    If you modify the contents of the Certificate Templates folder for a particular CA, that CA’s pKIEnrollmentService object must be updated in Active Directory. That means that you will have some latency as the changes replicate amongst your domain controllers. It is possible that some user in an outlying site will attempt to enroll for a certificate against the Old And Busted CA and that request will fail because the Old And Busted CA knows immediately that it should not issue any certificates. Given time, though, that error condition will fade as all domain controllers get the new changes. If you’re extremely sensitive to that kind of failure, however, then just add your templates to the New Hotness Issuing CA first, wait a day (or whatever your end-to-end replication latency is) and then remove those templates from the Old And Busted CA. In the long run, it won’t matter if the Old And Busted CA issues a few last minute certificates.

    At this point all certificate requests within your organization will be processed by the New Hotness Issuing CA, but what about all those certificates issued by the Old And Busted CA that are still in use? Do you have to manually go to each user and computer and request new certificates? Well…it depends on how the certificates were originally requested.

    Manually Requested

    If a certificate has been manually requested then, yes, in all likelihood you’ll need to manually update those certificates. I’m referring here to those certificates requested using the Certificates MMC snap-in, or through the Web Enrollment Pages. Unfortunately, there’s no automatic management for certificates requested manually. In reality, though, refreshing these certificates probably means changing some application or service so it knows to use the new certificate. I refer here specifically to Server Authentication certificates in IIS, OCS, SCCM, etc. Not only do you need to change the certificate, but you also need to reconfigure the application so it will use the new certificate. Given this situation, it makes sense to make your necessary changes gradually. Presumably, there is already a procedure in place for updating the certificates used by these applications I mentioned, among others I didn’t, as the current certificates expire. As time passes and each of these older, expiring certificates are replaced by new certificates issued by the new CA, you will gradually wean your organization off of the Old And Busted CA and onto the New Hotness Issuing CA. Once that is complete you can safely decommission the old CA.

    And it isn’t as though you don’t have a deadline. As soon as the Old And Busted CA certificate itself has expired you’ll know that any certificate ever issued by that CA has also expired. The Microsoft CA enforces such validity period nesting of certificates. Hopefully, though, that means that all those certificates have already been replaced, and you can finally decommission the old CA.

    Automatically Enrolled

    Certificate Autoenrollment was introduced in Windows XP, and it allows the administrator to assign certificates based on a particular template to any number of forest users or computers. Triggered by the application of Group Policy, this component can enroll for certificates and renew them when they get old. Using Autoenrollment, once can easily deploy thousands of certificates very, very quickly. Surely, then, there must be an automated way to replace all those certificates issued by the previous CA?

    As a matter of fact, there is.

    As described above, the new PKI is up and ready to start issuing digital certificates. The old CA is still up and running, but all the templates have been removed from the Certificate Templates folder so it is no longer issuing any certificates. But you still have literally thousands of automatically enrolled certificates outstanding that need to be replaced. What do you do?

    In the Certificates Templates MMC snap-in, you’ll see a list of all the templates available in your enterprise. To force all holders of a particular certificate to automatically enroll for a replacement, all you need to do is right-click on the template and select Reenroll All Certificate Holders from the context menu.

    clip_image002

    What this actually does is increment the major version number of the certificate template in question. This change is detected by the Autoenrollment component on each Windows workstation and server prompting them to enroll for the updated template, replacing any certificate they may already have. Automatically enrolled user certificates are updated in the exact same fashion.

    Now, how long it takes for each certificate holder to actually finish enrolling will depend how many there are and how they connect to the network. For workstations that are connected directly to the network, user and computer certificates will be updated at the next Autoenrollment pulse.

    Note: For computers, the autoenrollment pulse fires at computer startup and every eight hours thereafter. For users, the autoenrollment pulse fires at user logon and every eight hours thereafter. You can manually trigger an autoenrollment pulse by running certutil -pulse from the command line. Certutil.exe is installed with the Windows Server 2003 Administrative Tools Pack on Windows XP, but it is installed by default on the other currently supported versions of Windows.

    For computers that only connect by VPN it may take longer for certificates to be updated. Unfortunately, there is no blinking light that says all the certificate holders have been reenrolled, so monitoring progress can be difficult. There are ways it could be done -- monitoring the certificates issued by the CA, using a script to check workstations and servers and verify that the certificates are issued from the new CA, etc. -- but they require some brain and brow work from the Administrator.

    There is one requirement for this reenrollment strategy to work. In the group policy setting where you enable Autoenrollment, you must have the following option selected: Update certificates that use certificate templates.

    clip_image003

    If this policy option is not enabled then your autoenrolled certificates will not be automatically refreshed.

    Remember, there are two autoenrollment policies -- one for the User Configuration and one for the Computer Configuration. This option must be selected in both locations in order to allow the Administrator to force both computers and users to reenroll for an updated template.

    But I Have to Get Rid of the Old CA!

    As I’ve said earlier, once you’ve configured the Old And Busted CA so that it will no longer issue certificates you shouldn’t need to touch it again until all the certificates issued by that CA have expired. As long as the CA continues to publish a revocation list, all the certificates issued by that CA will remain valid until they can be replaced. But what if you want to decommission the Old And Busted CA immediately? How could make sure that your outstanding certificates would remain viable until you can replace them with new certificates? Well, there is a way.

    All X.509 digital certificates have a validity period, a defined interval time with fixed start and end dates between which the certificate is considered valid unless it has been revoked. Once the certificate is expired there is no need to check with a certificate revocation list (CRL) -- the certificate is invalid regardless of its revocation status. Revocation lists also have a validity period during which time it is considered an authoritative list of revoked certificates. Once the CRL has expired it can no longer be used to check for revocation status; a client must retrieve a new CRL.

    You can use this to your advantage by extending the validity period of the Old And Busted CA’s CRL in the CA configuration to match (or exceed) the remaining lifetime of the CA certificate. For example, if the Old And Busted CA’s certificate will be valid for the next 4 years, 3 months, and 10 days, then you can set the publication interval for the CA’s CRL to 5 years and immediately publish it. The newly published CRL will remain valid for the next five years, and as long as you leave that CRL published in the defined CRL distribution points -- Active Directory and/or HTTP -- clients will continue to use it for checking revocation status. You no longer need the actual CA itself so you can uninstall it.

    One drawback to this, however, is that you won’t be able to easily add any certificates to the revocation list. If you need to revoke a certificate after you’ve decommissioned the CA, then you’ll need to use the command line utility certutil.exe.

    Certutil.exe -resign “Old And Busted CA.crl” +<serialNumber>

    Of course, this requires that you keep the private keys associated with the CA, so you’d better back up the CA’s keys before you uninstall the role.

    Conclusion

    Wow…we’ve covered a lot of information here, so I’ll try to boil all of it down to the most important points. First, yes you can have multiple root CAs and even multiple PKIs in a single Active Directory forest. Because of the way the objects are representing those CAs are named and stored, you couldn’t possibly experience a conflict unless you tried to give more than one CA the same CA name.

    Second, once the new PKI is built you’ll want to configure your old CA so that it no longer issues certificates. That job will now belong to the issuing CA in your new PKI.

    Third, the ease with which you can replace all the certificates issued by the old CA with certificates issued by your new CA will depend mainly on how the certificates were first deployed. If all of your old certificates were requested manually then you will need to replace them in the same way. The easiest way to do that is replace them all gradually as they expired. On the other hand, if your old certificates were deployed via autoenrollment then you can trigger all of your autoenrollment clients to replace the old certificates with new ones from the new PKI. You can do this through the Certificate Templates MMC snap-in.

    And finally, what do you do with the old CA? Well, if you don’t need the equipment you can just keep it around until it either expires or all the old certificates have been replaced. If, however, you want to get rid of it immediately you can extend the lifetime of the old CA’s CRL to match the remaining validity period of the CA certificate. Just publish a new CRL and it’ll be good until all outstanding certificates have expired. Just keep in mind that this route will limit your ability to revoke those old certificates.

    If you think I missed something, or you want me to clarify a certain point, please feel free to post in the comments below.

    Jonathan “Man in Black” Stephens

    PS: Don’t ever challenge my Office clip art skills again, Jonathan.

    image

    - Ned