When designing a public key infrastructure (PKI) for your organization, you must develop an effective disaster recovery plan to ensure that, in the event of failure of the computer hosting Certificate Services, you can recover in a timely manner with little effect on your organization.
Failed services. If Certificate Services fails to start on the certification authority (CA) computer, no certificate can be issued and certificate revocation lists (CRLs) cannot be published. Your disaster plan for recovery should include performing either System State or manual CA backups and testing recovery (on a different system) on a regular basis.
Hardware failure. Disaster plan options for recovering after hardware failure include:
· Maintaining duplicate hardware (such as spare motherboards or spare computers);
· Implementing fault-tolerant RAID 1 or RAID 5 volumes to prevent CA failure due to a single disk failure.
Network infrastructure failure. Disaster recovery plans must account for network infrastructure failures. If an application implements CRL checking and network infrastructure failure prevents the application from accessing the most recent version of the CRL, an application will not validate the certificates presentedto the application. Your disaster recovery should include methods of diagnosing network infrastructure failures and developing methods of publishing CRL information that are redundant to protect against network failure.
One of the most important tasks during the design and deployment of a PKI is to ensure that your network and configuration documentation is updated continually. When you undergo disaster recovery, this documentation is the most important source of information regarding the previous Certificate Services configuration.
You should maintain the following documentation to ensure that you can apply all required configuration of Certificate Services successfully. The backup and recovery procedure for each of these items is explained later in this document.
All certificate template definitions. In the worst case, you might have to rebuild Active Directory, which requires the redefinition of all certificate templates. By documenting the individual settings for each certificate template on a tab-by-tab basis, you can easily re-create each certificate template.
All certificate templates published at the CA. You can create a custom script file that implements the certutil –SetCAtemplates +<TemplateName> to publish certificate templates and certutil –SetCAtemplates –<TemplateName> to remove certificate templates from the CA.
All permissions and user rights assignments. CA permissions define which users or groups hold the CA administrator and certificate manager Common Criteria roles, which groups or users can read the CA configuration, and which groups or users can request certificates from the CA. In addition, the Local Security Policy or domain-based Group Policy objects (GPOs) applied to the CA’s computer account defines the user rights assigned to the computer account, including the Common Criteria backup operators and auditor role holders.
All names used for the CA. Includes the CA’s logical name, the NetBIOS name of the computer hosting Certificate Services, and the domain or workgroup membership. The certificate information is based on the CA’s specific names and must be restored exactly.
All specific settings in the properties of the CA in the Certification Authority console. Be sure to identify which certificates are designated for key recovery, if implemented, as well as certificate manager restrictions.
Any post- or preinstallation script files used to configure the CA. For example, if you run a batch file consisting of certutil commands that define the CA’s registry settings, you should store a copy of the batch file for documentation and recovery purposes. Likewise, you should keep a copy of a batch file that publishes the CA’s CRL on an externally accessible Web server.
The CA data paths. When you restore the CA, the previous file locations for the CA database, CA log files, and CA configuration information must be maintained to match the restored registry values.
The CRL and Authority Information Access (AIA) publication points. Once the CA is restored, you must publish an updated CRL and, possibly, an updated CA certificate to the designated publication points. Ensure that no previous publication points are omitted.
The cryptographic service provider (CSP) used to protect the CA’s privatekey. The same CSP must be used to restore the previous key pair for theCA. The CSP might require additional software.
The key length of the CA’s certificate. If you are reinstalling the CA orrenewing the CA certificate, you should maintain the same key length as originallydeployed.
The logical disk-partitioning scheme for the CA computer. When you restore Certificate Services configuration, the disk volumes must implement the same drive letters. Disk volumes can be different sizes or implement different RAID levels, but the drive letters and locations must remain the same for theCA database, CA logs, CA configuration folder (if implemented), and operating system.
A copy of the CAPolicy.inf file deployed in the %windir% of the CA computer. The CAPolicy.inf must be in place when renewing the CA’s certificate.
There are two methods to backup and restore the Certification Authority. The methods are:
1- System State Backup
2- Certutil command line in combination of registry export
Update: It just came to my attention that System State Backup in Windows 2008 and 2008 R2 will not backup the private key of the CA. The private key will be stored in hidden folder structure "%systemdrive\ProgramData\Microsoft\Crypto\Keys" which will be linked and accessible via "%systemdrive%\users\all users\microsoft\crypto\keys". %systemdrive%\ProgramData\Microsoft\Crypto\Keys" is not included in System State backup as it's not in system writers metadata and so will be empty when doing a System State restore.
If you prefer to have System State Backup, then you should consider applying the following hotfix http://support.microsoft.com/kb/2603469 on your CAs running Windows Server 2008 or 2008 R2 to backup the Private Key.
Each method has advantages and disadvantages. The main advantage of System State backup is simplicity, where the administrator has to join an identical piece of hardware to the domain where the CA existed and restore System State Backup. The main disadvantage of System State is dependence on identical hardware
The Certutil command in combination with the registry export allows the administrator to restore the Certification Authority to any server – hardware agnostic. The main disadvantage of the Certutil command is the amount of steps required to perform the restore.
The document hereafter will only discuss the backup and restore methods using the Certutil command. In addition the document assumes Web Enrollment Pages are installed at the Certification Authority.
Any proper backup of a CA should include the Certificate Security Protocol, Templates published at the CA, Private Key, Certificate Database and logs in addition to the configuration of the CA stored in HKLM\System\CurrentControlSet\Services\Certsvc\Configuration. The script below combines all of these steps
1- Log on as user who has CA administrator rights.
2- Create a folder under %Homedrive% called Backup.
3- Create a new text document under C:\scripts
4- Paste the following text:
Echo Backup Certification Authority, Certificates, Templates and CSP
Echo Y| del C:\backup\database
Echo Y| del c:\backup
Echo Backing up the Certification Authority and Certificates
certutil -backup -p Password c:\backup
Echo Backing up the registry keys
reg export HKLM\System\CurrentControlSet\Services\CertSvc\Configuration c:\backup\regkey.reg
Certutil –getreg CA\CSP > C:\Backup\CSP.txt
Echo Documenting all certificate templates published at the CA
Certutil –catemplates > C:\Backup\CATemplates.txt
*Note* You need to enter a valid Password in the script where noted. The immediate line following the Certification Authority backup will back up the registry
5- Save the file as “BackupCertificates.cmd”
6- Schedule a task to run every day using an administrative account.
7- Schedule your regular backup software job to backup the System State and the C:\Backup folder every day or copy the folder to a safe location.
Restoring the CA will require using the backup files taken from the Certification Authority, in addition to rebuilding a new server. The steps required are:
1- Extending the life of the CRL file
2- Decommission the Old Certification Authority
3- Install Active Directory Certificate Services (ADCS) at the new server
4- Restore the Certification Authority Configuration
5- Restore the Database and Templates to the Certification Authority
This step is necessary to ensure clients’ revocation files are processed in a timely manner,
1- Log on to a any machine in your domain as an administrator
2- Restore the CA's Private key to the machine
3- Obtain a CRL or certificate issued by the CA being tested. The CRL or certificate must correspond to the CA key and certificate being tested where you are restoring multiple keys.
4- Extend the life of the CRL by running Certutil –sign <CRLFileName.crl> ++dd, and when prompted , select the CA certificate (imported in the previous procedure) as the signing certificate.
Certutil -sign Contoso-Issuing-CA.crl ++03
5- Publish the CRL file to all distribution points as follows:
a. Copy the CRL file to the http distribution points
b. Log on to any machine in the domain as an enterprise admin and run the Certutil –f –dspublish <CRLFileName.crl>
You must now clean the keys from the test system. To clean the keys from the system
1. Log on as a member of local Administrators and delete the user profile of the administrator account using Advanced Properties in My Computer.
2. Delete the administrator account.
3. Securely erase unallocated areas of the disk to permanently remove traces of the keys by running the following command.
Note: Specifying %allusersprofile% as the path ensures that the cipher.exe command operates on the drive holding the user profiles. It clears the whole drive, not just the indicated path, hence making the machine unusable.
This procedure is explained in details in a support article. Refer to http://support.microsoft.com/kb/889250 for the steps required to decommissions the old Certification Authority
The new server must have the same computer name as the old server. Furthermore, it should have the same Operating System of the failed server
1- Partition the server with the same volume names
2- Copy or restore the files from the Backup folder. You should have the database, PKCS12 (*.P12) file, the registry, CATemplates.txt, and CSP.txt to the new server.
3- In Server Manager, click Add Roles.
4- In the Select Role Services window, select Certification Authority and Certification Authority Web Enrollment – if installed previously , and then click Next
5- In the Specify CA Type dialog box, click the appropriate CA type based on the failed server CA type.
6- Click Use custom settings to generate the key pair and CA certificate, and then click Next.
7- In the Set Up Private Key windows, select Use existing private key and then select the option select a certificate and use its associated private key. Click Next
8- In the Select Existing Certificate, choose the Import option and browse to the PKCS12 file in the backup folder, type the password you used during the backup, and click OK, then click Install
9- Follow the setup screens until the CA is restored
1- Stop the Certificate Services service.
2- Locate the registry file that you restored, and then double-click it to import the registry settings. If the path that is shown in the registry export from the old CA differs from the new path, you must adjust your registry export accordingly. By default, the new path is C:\Windows in Windows Server 2003.
Use the Certification Authority snap-in to restore the CA database. To do this, follow these steps:
1- In the Certification Authority snap-in, right-click the CA name, click All Tasks, and then click Restore CA. The Certification Authority Restore Wizard starts.
2- Click Next
3- Click Certificate database and certificate database log.
4- Type the backup folder location, and then click Next.
5- Verify the backup settings. The Issued Log and Pending Requests settings should be displayed.
6- Click Finish, and then click Yes to restart Certificate Services when the CA database is restored.
7- In the Certification Authority snap-in, manually add or remove certificate templates based on the templates published at the CA using the CAtemplates.txt file
Thanks for blogging about the PKI, its really helpfull.
one of the thing which i was trying to clarrify and not yet succeeded is when a CA (Issuing CA) is down how long can i survive.
what i heard from folks around is that the certs will be working fine till the CRL expires. but will it work till the Certificate expire even if the CRL expired?
This depends on your distribution points, the certificate type used (if it checks for revocation) and the CRL period.
For example, if your distribution point is LDAP, your certificates can access LDAP and already have a revocation time of 2 days left, then your issuing CA can be down for 2 days, until the client pulls downs a new CRL from the distribution point. Likewise for the HTTP distribution point.
The most important thing to check is the Next Update in the CRL file itself because the client caches the CRL until the next update, and of course making sure the CRL file is already published at your distribution points for any client to pull it down.
As for expired certificates, they will not be used, so it doesn’t matter if the CRL was expired or not.
One thing I am trying to get clarification on is what to do if the old CA is unavailable, like with a hardware failure. How does one "decommission" the old CA from AD if you can't boot the server?
Does the CA have backups? If that is the case then you can restore it. You can always clean the CA records from Active Directory using support.microsoft.com/.../889250
I have a question regarding the script, the row:
that is put after the creation of the CSP.txt and CATemplates.txt removes these two files, is there a point to create them in the first case then?
I am probably lacking some deeper knowledge of CA:s but I would appreciate a clarification.
Thank you !
Just to be crystal clear.
By saying that the Windows Server 2008 and 2008R2 system state backup does not backup the private keys of the CA, You are saying that Brian Komars Book "Windows Server 2008 PKI and Certificate Security" is wrong on this? (Page 310)
Yes! This was reported and verified by our engineers
Ordeith, the script was updated. Thanks,
Per TechieGirl, we just ran into a CA crash where we found out we did not have the entire CA backed up. So we had to "decommission' the CA, but there are a number of blog entries on this (see support.microsoft.com/.../555151 for some documentation). One key point is removing the CA entries from the ActiveDirectory containers. Keep in mind that, once you do that, AD will automaticaly *remove* the CA root cert from every client computer. The net effect is that any older certificates you had (and want to replace with the new CA) are immediately invalidated. This means no more email signing, no certificate-based VPN connections (if you use that instead of PSK), etc. Be sure that's what you want and that (if possible) you've setup the new CA and updated all client computers (and email security) *before* removing the old CA entries from AD sites and services.
If you are not using automatic CRL, you can continue using the old certficates from the crashed CA until they expire as long as the crashed CA remains in AD sites and services. Just be aware and choose wisely...
certcollection.org has an exact copy of this blog post here:
Thanks for the post. Shall I use the same script to backup my Windows 2003 CA
Yes, you can use the same script to backup a 2003 CA
are there any windows updates that the private keys will store in future (2008 R) in the systemstate backup or is this block up to date what the backup of private keys belongs?
i have done the points, i wonder that the Database Directory has 17 MB, after a new run 18 MB.
every backup i made the database directory grow about 1 MB.
This blog is up to date and applies to Windows Server 2008 and 2008 R2.
Can't I just archive the private key from my Enterprise CA and use System State to back up the CA going forward? I'd imagine that, after a restore of system-state (after a failure), I'd then have to import the private key but that shouldn't be a problem, right?
Thanks for a great guide!