Quite a few of my customers have opted to use nCipher security modules for the storage of their RMS private keys, instead of centrally storing them in the RMS database.

The benefit of using one of these devices, is the extra security level it offers the protection of your RMS keys. The downside of doing this is that administration becomes a bit more difficult, in that you have to manually export the keys to each server that will be joining the RMS cluster, as compared to using the software based private keys, which are stored centrally in the SQL database, and automatically available for all joining clusters to discover. Still, the nCipher modules add extra layers of security, and since the point of RMS is to protect your intellectual property, it is not a bad idea to consider using this type of hardware storage to maximize the security of your RMS keys.

Sometimes when trying to get RMS to play well with the nCipher CSP, we run into problems that require some things to be done. Here are some random tips for getting RMS working with your nCipher modules.

If the provisioning process hangs during enrollment, it is quite possible that during the installation, a specific nCipher key was not properly set. Have a look at the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\nCipher\Cryptography
UseModuleKeys

If it is set to 1, set it to 0, restart the machine, and try again. If it doesn't stop the hanging, then set the key back to where it was, reboot, and move on to the next section.

If you get the following error, or if the above steps don't resolve the issue:

Microsoft.DigitalRightsManagement.Configuration.ClientEnrollException: Failed to enroll the server. If you requesting a new certificate for a root certification server, verify that you can connect to the internet and that you have set your proxy settings if they are required, and then try again. If you are requesting a new certificate for a licensing server, verify that you can connect to the root certification server, and then try again. ---> System.Security.Cryptography.CryptographicException: An internal error occurred.

   at System.Security.Cryptography.RSACryptoServiceProvider._GenerateKey(IntPtr unknown1, Int32 unknown2, Int32 unknown3)
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 dwKeySize, CspParameters parameters, Boolean useDefaultKeySize)
   at Microsoft.DigitalRightsManagement.Cryptography.RsaCapiKey..ctor(CspParameters parameters)
   at Microsoft.DigitalRightsManagement.KeyManagement.PrivateKeyImpl.GenerateKey(String strIDType, String strID, Int32 nKeySize, String strCSP)
   at Microsoft.DigitalRightsManagement.Configuration.EnrollmentBase.PopulateKeyPair()
   at Microsoft.DigitalRightsManagement.Configuration.EnterpriseEnrollment.Enroll(Boolean fEnrollOffline)
   at Microsoft.DigitalRightsManagement.Configuration.ProvisioningBase.Enroll(Boolean fEnrollOffline)
   --- End of inner exception stack trace ---
   at Microsoft.DigitalRightsManagement.Configuration.ProvisioningBase.Enroll(Boolean fEnrollOffline)
   at Microsoft.DigitalRightsManagement.Configuration.ProvisioningBase.Run()

Then try checking the following:

Make sure you have a valid security world created, and that the nFast Server Service is running. Rebooting the machine, and verifying that the nFast Server Service is starting properly, should be a good indication of whether of not your nCipher state is healthy.

Go to c:\nfast\bin at a command prompt and type the following:
keytst -lm

This should return a list of key containers.

If it doesn't, then give the RMS Service account full control of the c:\nfast\kmdata folder, and try the provisioning process again. The provisioning may still fail, but the above command should actually show a list of key containers. 

If it does show a list of key containers, then try reprovisioning, except this time, instead of choosing the nCipher CSP for your keys, choose the option 'Use existing key pair'.

I do not claim to be an expert, and I don't even have one of these to test with, so my testing usually involves having customers test different things in their environment. One of these days I'll get one and be able to sit down and play with it.

Hopefully these tips help someone out.

Thanks to Manthan and Cagatay for the information, and Glenn for being my lab rat. :)

-Jason