Windows PKI blog

News and information for public key infrastructure (PKI) and Active Directory Certificate Services (AD CS) professionals

Disaster Recovery Procedures for Active Directory Certificate Services (ADCS)

Disaster Recovery Procedures for Active Directory Certificate Services (ADCS)

  • Comments 28
  • Likes

 

Introduction:

When designing a public key infrastructure (PKI) for your organization, you must develop an effective disaster recovery plan to ensure that, in the event of failure of the computer hosting Certificate Services, you can recover in a timely manner with little effect on your organization.

Common Reasons that Make a Disaster Recovery Plan Necessary Include: 

Failed services. If Certificate Services fails to start on the certification authority (CA) computer, no certificate can be issued and certificate revocation lists (CRLs) cannot be published. Your disaster plan for recovery should include performing either System State or manual CA backups and testing recovery (on a different system) on a regular basis.

Hardware failure. Disaster plan options for recovering after hardware failure include: 

·         Maintaining duplicate hardware (such as spare motherboards or spare computers);

·         Implementing fault-tolerant RAID 1 or RAID 5 volumes to prevent CA failure due to a single disk failure.

Network infrastructure failure. Disaster recovery plans must account for network infrastructure failures. If an application implements CRL checking and network infrastructure failure prevents the application from accessing the most recent version of the CRL, an application will not validate the certificates presentedto the application. Your disaster recovery should include methods of diagnosing network infrastructure failures and developing methods of publishing CRL information that are redundant to protect against network failure. 

Developing Required Documentation

 

One of the most important tasks during the design and deployment of a PKI is to ensure that your network and configuration documentation is updated continually. When you undergo disaster recovery, this documentation is the most important source of information regarding the previous Certificate Services configuration.

You should maintain the following documentation to ensure that you can apply all required configuration of Certificate Services successfully. The backup and recovery procedure for each of these items is explained later in this document. 

 All certificate template definitions. In the worst case, you might have to rebuild Active Directory, which requires the redefinition of all certificate templates. By documenting the individual settings for each certificate template on a tab-by-tab basis, you can easily re-create each certificate template.

All certificate templates published at the CA. You can create a custom script file that implements the certutil –SetCAtemplates +<TemplateName> to publish certificate templates and certutil –SetCAtemplates –<TemplateName> to remove certificate templates from the CA.

 All permissions and user rights assignments. CA permissions define which users or groups hold the CA administrator and certificate manager Common Criteria roles, which groups or users can read the CA configuration, and which groups or users can request certificates from the CA. In addition, the Local Security Policy or domain-based Group Policy objects (GPOs) applied to the CA’s computer account defines the user rights assigned to the computer account, including the Common Criteria backup operators and auditor role holders.

 All names used for the CA. Includes the CA’s logical name, the NetBIOS name of the computer hosting Certificate Services, and the domain or workgroup membership. The certificate information is based on the CA’s specific names and must be restored exactly.

 All specific settings in the properties of the CA in the Certification Authority console. Be sure to identify which certificates are designated for key recovery, if implemented, as well as certificate manager restrictions.

 Any post- or preinstallation script files used to configure the CA. For example, if you run a batch file consisting of certutil commands that define the CA’s registry settings, you should store a copy of the batch file for documentation and recovery purposes. Likewise, you should keep a copy of a batch file that publishes the CA’s CRL on an externally accessible Web server.

The CA data paths. When you restore the CA, the previous file locations for the CA database, CA log files, and CA configuration information must be maintained to match the restored registry values.

The CRL and Authority Information Access (AIA) publication points. Once the CA is restored, you must publish an updated CRL and, possibly, an updated CA certificate to the designated publication points. Ensure that no previous publication points are omitted.

The cryptographic service provider (CSP) used to protect the CA’s privatekey. The same CSP must be used to restore the previous key pair for theCA. The CSP might require additional software.

 The key length of the CA’s certificate. If you are reinstalling the CA orrenewing the CA certificate, you should maintain the same key length as originallydeployed.

 The logical disk-partitioning scheme for the CA computer. When you restore Certificate Services configuration, the disk volumes must implement the same drive letters. Disk volumes can be different sizes or implement different RAID levels, but the drive letters and locations must remain the same for theCA database, CA logs, CA configuration folder (if implemented), and operating system.

 A copy of the CAPolicy.inf file deployed in the %windir% of the CA computer. The CAPolicy.inf must be in place when renewing the CA’s certificate.

Disaster Recovery Procedures: 

There are two methods to backup and restore the Certification Authority. The methods are:

1-      System State Backup

2-      Certutil command line in combination of registry export

 Update: It just came to my attention that System State Backup in Windows 2008 and 2008 R2 will not backup the private key of the CA. The private key will be stored in hidden folder structure "%systemdrive\ProgramData\Microsoft\Crypto\Keys" which will be linked and accessible via "%systemdrive%\users\all users\microsoft\crypto\keys". %systemdrive%\ProgramData\Microsoft\Crypto\Keys" is not included in System State backup as it's not in system writers metadata and so will be empty when doing a System State restore.

 

If you prefer to have System State Backup, then you should consider applying the following hotfix http://support.microsoft.com/kb/2603469 on your CAs running Windows Server 2008 or 2008 R2 to backup the Private Key. 

Advantages and Disadvantages of Each Procedure: 

Each method has advantages and disadvantages. The main advantage of System State backup is simplicity, where the administrator has to join an identical piece of hardware to the domain where the CA existed and restore System State Backup. The main disadvantage of System State is dependence on identical hardware

The Certutil command in combination with the registry export allows the administrator to restore the Certification Authority to any server – hardware agnostic. The main disadvantage of the Certutil command is the amount of steps required to perform the restore.

 

The document hereafter will only discuss the backup and restore methods using the Certutil command. In addition the document assumes Web Enrollment Pages are installed at the Certification Authority.

 


Back Up the Certification Authority: 

Any proper backup of a CA should include the Certificate Security Protocol, Templates published at the CA, Private Key, Certificate Database and logs in addition to the configuration of the CA stored in HKLM\System\CurrentControlSet\Services\Certsvc\Configuration. The script below combines all of these steps

1-      Log on as user who has CA administrator rights.

2-      Create a folder under %Homedrive% called Backup.

3-      Create a new text document under C:\scripts

4-      Paste the following text: 

Echo Backup Certification Authority, Certificates, Templates and CSP

c:

cd \scripts

Echo Y| del C:\backup\database

rd C:\backup\database

Echo Y| del c:\backup

Echo Backing up the Certification Authority and Certificates

certutil -backup -p Password c:\backup

Echo Backing up the registry keys

reg export HKLM\System\CurrentControlSet\Services\CertSvc\Configuration c:\backup\regkey.reg

 Certutil –getreg CA\CSP > C:\Backup\CSP.txt

Echo Documenting all certificate templates published at the CA

Certutil –catemplates > C:\Backup\CATemplates.txt

*Note* You need to enter a valid Password in the script where noted. The immediate line following the Certification Authority backup will back up the registry

5-      Save the file as “BackupCertificates.cmd”

6-      Schedule a task to run every day using an administrative account.

7-      Schedule your regular backup software job to backup the System State and the C:\Backup folder every day or copy the folder to a safe location.

Steps to Restore the Certification Authority:

Restoring the CA will require using the backup files taken from the Certification Authority, in addition to rebuilding a new server. The steps required are:

 1-      Extending the life of the CRL file

2-      Decommission the Old Certification Authority

3-      Install Active Directory Certificate Services (ADCS) at the new server

4-      Restore the Certification Authority Configuration

5-      Restore the Database and Templates to the Certification Authority 

Extending the Life of the CRL file: 

This step is necessary to ensure clients’ revocation files are processed in a timely manner,

1-      Log on to a any machine in your domain as an administrator

2-    Restore the CA's Private key to the machine

3-      Obtain a CRL or certificate issued by the CA being tested. The CRL or certificate must correspond to the CA key and certificate being tested where you are restoring multiple keys.

4-      Extend the life of the CRL by running Certutil –sign <CRLFileName.crl>  ++dd, and when prompted , select the CA certificate (imported in the previous procedure) as the signing certificate.

 

Example:

 

                Certutil -sign Contoso-Issuing-CA.crl ++03

 5-      Publish the CRL file to all distribution points as follows:

a.       Copy the CRL file to the http distribution points

b.      Log on to any machine in the domain as an enterprise admin and run the Certutil –f –dspublish <CRLFileName.crl>

 You must now clean the keys from the test system.  To clean the keys from the system

1.       Log on as a member of local Administrators and delete the user profile of the administrator account using Advanced Properties in My Computer.

2.       Delete the administrator account.

3.       Securely erase unallocated areas of the disk to permanently remove traces of the keys by running the following command.

                 Cipher /W:%AllUsersProfile%

 Note:  Specifying %allusersprofile% as the path ensures that the cipher.exe command operates on the drive holding the user profiles. It clears the whole drive, not just the indicated path, hence making the machine unusable.

Decommission the Old Certification Authority:

This procedure is explained in details in a support article. Refer to http://support.microsoft.com/kb/889250 for the steps required to decommissions the old Certification Authority

Install Active Directory Certificate Services at the New Server:

The new server must have the same computer name as the old server. Furthermore, it should have the same Operating System of the failed server

1-      Partition the server with the same volume names

2-      Copy or restore the files from the Backup folder. You should have the database, PKCS12 (*.P12) file, the registry, CATemplates.txt, and CSP.txt  to the new server.

3-      In Server Manager, click Add Roles.

4-      In the Select Role Services window, select Certification Authority and Certification Authority Web Enrollment – if installed previously , and then click Next

5-      In the Specify CA Type dialog box, click the appropriate CA type based on the failed server CA type.

6-      Click Use custom settings to generate the key pair and CA certificate, and then click Next.

7-      In the Set Up Private Key windows, select Use existing private key and then select the option select a certificate and use its associated private key. Click Next

8-      In the Select Existing Certificate, choose the Import option and browse to the PKCS12 file in the backup folder, type the password you used during the backup, and click OK, then click Install

9-      Follow the setup screens until the CA is restored

Restore the Certification Authority Configuration:

1-      Stop the Certificate Services service.

2-      Locate the registry file that you restored, and then double-click it to import the registry settings. If the path that is shown in the registry export from the old CA differs from the new path, you must adjust your registry export accordingly. By default, the new path is C:\Windows in Windows Server 2003.

Restore the Database and Templates to the Certification Authority:

Use the Certification Authority snap-in to restore the CA database. To do this, follow these steps:

1-      In the Certification Authority snap-in, right-click the CA name, click All Tasks, and then click Restore CA. The Certification Authority Restore Wizard starts.

2-      Click Next

3-      Click Certificate database and certificate database log.

4-      Type the backup folder location, and then click Next.

5-      Verify the backup settings. The Issued Log and Pending Requests settings should be displayed.

6-      Click Finish, and then click Yes to restart Certificate Services when the CA database is restored.

7-      In the Certification Authority snap-in, manually add or remove certificate templates based on the templates published at the CA using the CAtemplates.txt file 

Comments
  • Thanks for blogging about the PKI, its really helpfull.

    one of the thing which i was trying to clarrify and not yet succeeded is when a CA (Issuing CA) is down how long can i survive.

    what i heard from folks around is that the certs will be working fine till the CRL expires. but will it work till the Certificate expire even if the CRL expired?

  • This depends on your distribution points, the certificate type used (if it checks for revocation) and the CRL period.

    For example, if your distribution point is LDAP, your certificates can access LDAP and already have a revocation time of 2 days left, then your issuing CA can be down for 2 days, until the client pulls downs a new CRL from the distribution point. Likewise for the HTTP distribution point.

    The most important thing to check is the Next Update in the CRL file itself because the client caches the CRL until the next update, and of course making sure the CRL file is already published at your distribution points for any client to pull it down.

    As for expired certificates, they will not be used, so it doesn’t matter if the CRL was expired or not.

  • One thing I am trying to get clarification on is what to do if the old CA is unavailable, like with a hardware failure.  How does one "decommission" the old CA from AD if you can't boot the server?

  • Does the CA have backups? If that is the case then you can restore it. You can always clean the CA records from Active Directory using support.microsoft.com/.../889250

  • I have a question regarding the script, the row:

    Echo Y| del c:\backup

    that is put after the creation of the CSP.txt and CATemplates.txt removes these two files, is there a point to create them in the first case then?

    I am probably lacking some deeper knowledge of CA:s but I would appreciate a clarification.

    Thank you !

    Best regards

    /Ordeith

  • Just to be crystal clear.  

    By saying that the Windows Server 2008 and 2008R2 system state backup does not backup the private keys of the CA, You are saying that Brian Komars Book "Windows Server 2008 PKI and Certificate Security" is wrong on this? (Page 310)

  • Yes! This was reported and verified by our engineers

  • Ordeith, the script was updated. Thanks,

    Amer

  • Per TechieGirl, we just ran into a CA crash where we found out we did not have the entire CA backed up. So we had to "decommission' the CA, but there are a number of blog entries on this (see support.microsoft.com/.../555151 for some documentation). One key point is removing the CA entries from the ActiveDirectory containers. Keep in mind that, once you do that, AD will automaticaly *remove* the CA root cert from every client computer. The net effect is that any older certificates you had (and want to replace with the new CA) are immediately invalidated. This means no more email signing, no certificate-based VPN connections (if you use that instead of PSK), etc. Be sure that's what you want and that (if possible) you've setup the new CA and updated all client computers (and email security) *before* removing the old CA entries from AD sites and services.

    If you are not using automatic CRL, you can continue using the old certficates from the crashed CA until they expire as long as the crashed CA remains in AD sites and services. Just be aware and choose wisely...

  • certcollection.org has an exact copy of this blog post here:

    certcollection.org/.../53883-disaster-recovery-procedures-for-active-directory-certificate-services-adcs

  • Hi

    Thanks for the post. Shall I use the same script to backup my Windows 2003 CA

    Regards

    LMS

  • Yes, you can use the same script to backup a 2003 CA

  • Hello,

    are there any windows updates that the private keys will store in future (2008 R) in the systemstate backup or is this block up to date what the backup of private keys belongs?

    i have done the points, i wonder that the Database Directory has 17 MB, after a new run 18 MB.

    every backup i made the database directory grow about 1 MB.

    Why?

    Regards Eax

  • This blog is up to date and applies to Windows Server 2008 and 2008 R2.

  • Can't I just archive the private key from my Enterprise CA and use System State to back up the CA going forward?  I'd imagine that, after a restore of system-state (after a failure), I'd then have to import the private key but that shouldn't be a problem, right?

    Thanks for a great guide!

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment