A PKCS#10 request through the enrollment server fails with an invalid signature error
2 years ago
Originally Published: 2008-09-03
Article Number
000043595
Applies To
RSA Certificate Manager (RCM)
RSA Certificate Manager 6.8
RSA Certificate Manager 6.7
Sun Solaris 9
Sun Solaris 10
Microsoft Windows 2003 SP1
nCipher Hardware Security Module (HSM)
FIPS 140-2 Level III Strict mode enabled in nCipher Security World
nCipher cryptographic provider selected as Signature Verification Cryptographic Provider in RSA Certificate Manager
A PKCS#10 request is submitted through the enrollment server, which fails with the following message:

  This certificate request has been refused because it contains an invalid signature.

The request then goes into the refused state, but can be successfully approved from the queue.
If the same request is submitted with Software Cryptographic Provider selected as the signature verification cryptographic provider, the enrollment is successful.

This affects *all* requests (not just cut&past P10) - especially FireFox seems to be affected but I wouldn't bet that Opera or others wouldn't suffer the same fate.


Captured what is sent by FireFox and did a ASN.1 decode of the public key blob that is being send:

 

    0:d=0  hl=4 l= 576 cons: SEQUENCE         

    4:d=1  hl=4 l= 296 cons: SEQUENCE         

    8:d=2  hl=4 l= 290 cons: SEQUENCE         

   12:d=3  hl=2 l=  13 cons: SEQUENCE         

   14:d=4  hl=2 l=   9 prim: OBJECT            :rsaEncryption

   25:d=4  hl=2 l=   0 prim: NULL             

   27:d=3  hl=4 l= 271 prim: BIT STRING       

  302:d=2  hl=2 l=   0 prim: IA5STRING         :

  304:d=1  hl=2 l=  13 cons: SEQUENCE         

  306:d=2  hl=2 l=   9 prim: OBJECT            :md5WithRSAEncryption

  317:d=2  hl=2 l=   0 prim: NULL             

  319:d=1  hl=4 l= 257 prim: BIT STRING

 

As long as "md5" appears and the HSM is in strict FIPS mode and it is being used as the signature validation device it'll cause a "invalid signature"

 

Not a FireFox problem, not a HSM problem, not our problem but this combination will cause issues until FF switches to SHAx for the POP.


Issue
A PKCS#10 request through the enrollment server fails with an invalid signature error
Cause
When the PKCS#10 request has a Digest Algorithm that is not one that conforms to an allowed strict FIPS 140-2 Level III mode mechanism - in this case it was RSA MD5 - an invalid mechanism type message will be sent back to RSA Certificate Manager by nCipher. This will cause the invalid signature message to appear.

FIPS-approved algorithms: The following FIPS-approved Cryptographic algorithms are used: DSA (Cert. #143); Triple-DES (Cert. #378); AES (Cert, #303); RSA (Cert. #96); SHA-1; Diffie-Helman (used for key exchange in SSH2 is allowed in FIPS Mode but not approved).

The following algorithms are not available in FIPS Mode: MD5; Twofish; Blowfish; RC4.


Resolution
Either use an allowed mechanism - nominally SHA1 - or select software cryptographic provider for such PKCS#10 requests.

Related Articles

Trending Articles

Don't see what you're looking for?