'Error: Startup of Secure Directory Server failed!' when starting RCM services after resigning a CA
2 years ago
Originally Published: 2013-05-23
Article Number
000048734
Applies To
RSA Certificate Manager 6.9
Red Hat Enterprise Linux (RHEL) 64-bit

Two phase logging enabled via RCM admin interface => System Configuration workbench => Secure Logging configuration


Issue
"Error: Startup of Secure Directory Server failed!" when starting RCM services after resigning a CA
When attempting to startup RCM services (using START script), the Secure Logging Server start up fine however the Secure Directory Server (2nd of the four services to start) fails to start with the following error:

...
SSL key passphrase for Secure Directory Server:
Error: Startup of Secure Directory Server failed!
...
Passphrase entry for CA  "Test System CA"

MD5=3ab2d276108c3c309f5f47755c7ac953

Perform one of the following at the prompt:
- enter the passphrase
- enter "PROMPT" for Prompt Every Time mode

:

Passphrase entry for CA  "Test Administrative CA"

MD5=0e5feeb460b05a3814f2151842a1de4f

Perform one of the following at the prompt:
- enter the passphrase
- enter "PROMPT" for Prompt Every Time mode

:
SSL key passphrase for Secure Directory Server:
Broken pipe
In debugging the startup problem, stepped through the "START" program and got this error eventually using "strace xudad -f ../conf/xudad.conf":

cd LogServer/bin; ./start-xslogsrv
SSL key passphrase for Secure Logging Server:
testbox:/opt/RSA_CM/LogServer/bin/ > cd ..
testbox:/opt/RSA_CM/LogServer/ > cd ..
testbox:/opt/RSA_CM/ > cd Xudad/bin; strace xudad -f ../conf/xudad.conf

lsof |grep :5150
xslogsrv  5454     caadmin    8u     IPv4      16323                TCP *:5150 (LISTEN)


If you look below its trying to communicate with 5150 (log server started before this), and gets disconnected as to what broken pipe means.  The port is up and active though throughout the whole process.

connect(14, {sa_family=AF_INET, sin_port=htons(5150), sin_addr=inet_addr("10.10.44.14")}, 16) = 0
gettid()                                = 5463
gettimeofday({1369182428, 159459}, NULL) = 0
gettimeofday({1369182428, 159501}, NULL) = 0
write(14, "\26\3\1\0]\1\0\0Y\3\1Q\234\20\334\352\213\324j\217\ty\344\226E\247\"\370\254\254*w"..., 98) = 98
read(14, "\26\3\1\0Q", 5)               = 5
read(14, "\2\0\0M\3\1Q\234\20\334D~A\241\244\232\367m/le\4\260zb&\264s\276\361p\224"..., 81) = 81
read(14, "\26\3\1\6?", 5)               = 5
read(14, "\v\0\6;\0\0068\0\2\2600\202\2\2540\202\1\224\2\21\0\265\332\345\303\374\201nO<\32\273"..., 1599) = 1599
gettimeofday({1369182428, 162231}, NULL) = 0
gettimeofday({1369182428, 162273}, NULL) = 0
...
gettimeofday({1369182428, 163443}, NULL) = 0
gettimeofday({1369182428, 163489}, NULL) = 0
gettid()                                = 5463
read(14, "\26\3\1\1\r", 5)              = 5
read(14, "\f\0\1\t\0@\332X<\26\331\205\"\211\320\344\257uoL\312\222\335K\3453\270\4\373\17\355\224"..., 269) = 269
read(14, "\26\3\1\0\17", 5)             = 5
read(14, "\r\0\0\7\4\3\4\1\2\0\0\16\0\0\0", 15) = 15
write(14, "\26\3\1\0074\v\0\0070\0\7-\0\2\2650\202\2\2610\202\1\231\2\20\vU-\336\217\222e"..., 2048) = 2048
write(14, "2J\r8>\265\361\16-\270\203\3138\2761\24\3\1\0\1\1\26\3\1\0000\205\362\261L\307*"..., 74) = -1 EPIPE (Broken pipe)
--- SIGPIPE (Broken pipe) @ 0 (0) ---
+++ killed by SIGPIPE +++
Cause
When re-signing a custom CA, inadvertently the System CA was chosen on the page and hence the System CA certificate ended-up getting re-signed with the self-signed CA.  This caused the System CA certificate to chain up to the self-signed CA.  When RCM services were restarted, Secure Directory Server could not establish connection with the Secure Logging Server to log the startup event and hence failed to startup as two phase logging requires the Secure Logging Server to be reachable to log events.  The Secure Logging Server in this scenario refused connection as it no longer trusted the root CA of Secure Directory Server's SSL certificate which now chained up to the self-signed CA rather than the System CA, whereas the Secure Logging Server by default trusts the System CA (as a root) only.
Resolution
Any of the two approaches below can be followed to recover from the situation.  Approach (A) is recommended as it does not result in any data loss.


A) Temporarily trust the new root CA (the self-signed CA that inadvertantly signed the System CA) at Secure Logging Server to allow connections from Secure Directory Server.  Once RCM services are started and admin interface is reachable, restore the System CA certifcate back to the original one.  To do so, follow the process below:

1. Stop RCM services to make sure all RCM services are stopped (as the Secure Logging Server may still be running after the last attempt to start RCM services)

2. Make a full backup of RCM (copy the entire contents of RSA_CM folder)

3. Obtain the certificate (PEM formatted) of the self-signed CA that inadvertanetly re-signed the System CA.  The CA certificate can possibly be obtained from the Secure Logging Server logs (when the CA was originally created).  (The CA certificate can also be retrieved from the RCM database dumped to an ldif file (using ldbmcat tool); contact RSA Customer Support if assistance is required with this step.)

4. Make a backup of RSA_CM/LogServer/ssl.certs/cas.cert

5. Edit RSA_CM/LogServer/ssl.certs/cas.cert, and replace the System CA certificate with the self-signed CA certificate that signed the System CA (ensure the format and header/footer remain consistent in the file)

6. Start RCM services (the services should start up without any errors)

7. Go to RCM admin interface => CA Operations workbench => View the System CA, confirm that it was re-signed by the self-signed CA (indicated by the fields 'Certificate Chain', 'Issuing Jurisdiction ID', 'Issuing Jurisdiction Name', 'Valid From' etc) => click on Replace button in section 'CA Certificate Operations' while viewing the System CA => paste in the original System CA certificate obtained from the original copy of RSA_CM/LogServer/ssl.certs/cas.cert, and click 'Replace Certificate' button.

8. Obtain the System CA Jurisdiction's ID (md5 value): view System CA on CA Operations workbench, click 'View Configuration' button under Jurisdiction Configuration section, make a note of the value against 'Jurisdiction ID' field

9. Use the tool listuclass.xuda to find the System CA object (objectclass: xuda_ca), and update DOMAINID field with the md5 value noted in step #8 above

10. Restore RSA_CM/LogServer/ssl.certs/cas.cert back to the original one

11. Restart RCM services (the services should start up with any errors)


B)  Restore RCM database to the point just before re-signing of the System CA.  To do so, follow the process below:

1. CAUTION:  Restoring from a database backup means any updates/additions to the db (such as new certificates issued, certificate status changed, etc) after the db backup was taken until the last operations, will be lost.  If the impacted RCM is a test environment, this approach can be taken to resolve the issue.  For production environments, approach (A) is recommended.

2. Stop RCM services to make sure all RCM services are stopped (as the Secure Logging Server may still be running after the last attempt to start RCM services)

3. Make a full backup of RCM (copy the entire contents of RSA_CM folder)

4. Follow steps in section 'Database Recovery Procedure', pages 333-334, in RSA Certificate Manager 6.9 Administrator's Guide

5. After RCM database has been restored, start RCM services (the services should start up with any errors)
Workaround
Re-signed a self-signed CA certificate through CA Operations workbench => viewed a CA => clicked 'Re-sign' button and followed prompts to resign the CA (in an attempt to add extensions to the certificate), then restarted RCM services to complete the process

Related Articles

Trending Articles

Don't see what you're looking for?