I thought today I would post the solution to a problem that has haunted me for the last few weeks. Consider this scenario:
A customer had ADRMS in his environment. Users on the intranet could use RMS without problem. Users on the extranet, received errors trying to obtain their RAC. A quick check of the debug output from RMS shows this:
00000289 118.10672760  [msdrm]:CHttpBase::DispatchRequest returned hr:0,ErrorCode=200 when hitting Url=https://rms.domain.com/_wmcs/certification/Certification.asmx with Post size=13956 00000290 118.13926697  [msdrm]: VerifyCertificateChain FAILED with hr = 80090006
This error basically is a CAPI (Crypto API) error that translates to NTE_BAD_SIGNATURE.
So what does this mean? This means that from the time the RMS server issued the certificate, to the time it arrived at the client, somone or some thing modified the certificate, which invalidated the signature.
Now what? I had the customer try to RMS protect something as somone not signed into the AD, but on the *inside* of the ISA firewall. We did, and everything was successfull. OK, so that means that it only happens to people on the outside of ISA, so basically something on ISA was changing 'something' in the certificate. But what? I couldn't get the certificate, because once the failure happened RMS didn't drop it into the DRM folder. Also it was over SSL so I can't see what's in the packet easily. Now we *do* know from the server side logs, what the certificate looks like that was issued.
One of the developers gave me a tool that he wrote that would bypass the sig checking and get me a copy of the cert. I used this, and what we saw (actually Vlad the dev for server) was that the RMS server was issuing certificates with the URL HTTPS://rms.domain.com:443/_wmcs listed in the certificate, however, when it arrived at the client the url had changed slightly to https://rms.domain.com/_wmcs, which is different, which means something was changing that value. When inside the ISA firewall, the certificate arrives unmodified.
So, it took a long time toget this far, and a quick call to an ISA support engineer (once we knew what was changing in the cert) revealed the problem. Apparently there is an option on ISA publishing rule that you can set, which will look at all of the packets coming through, and scan for alternate URLs that don't match the ISA publsihing rule URL. If it finds one, it will correct it. Well this is *bad* when you are talking about a signed certificate.
Simply unchecking this option of Link Translation on the publishing rule, solved the problem.
Damn you and your crazy link translations, ISA!!! LOL.