Replica promotion for maintenance fails with certificate not verified error in RSA Authentication Manager 8.x
3 months ago
Originally Published: 2020-08-19
Article Number
000043825
Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.x

Issue

  • Replica rsa06 promotion fails when the console certificates were replaced on the primary rsa02 with certificates from a public Certificate Authority, CA but the replica has the default RSA self-signed console certificate installed. Or if primary and replica have console replacement certs signed by different CAs, e.g. Entrust and Komodo. Note: GoDaddy.com and GoDaddy Group are different CAs.
  • The replica rsa06 needs to trust the primary rsa02's replacement console cert, but the replica promotion process looks in /opt/rsa/am/server/security/trust.jks not /opt/rsa/am/server/security/webserver-inactive.jks to confirm the trust.
  • Attempting promotion for maintenance fails pre-promotion checks as shown in the replica's Operations Console:

Task status
Pre-promotion checks

Checking that services are running on this instance….
SUCCESS. 

Checking Replication status on this instance….
SUCCESS.
ERROR: The Operations Console on the primary instance is not reachable to check replication status or reachability with other instances.
ERROR: Could not access HTTP invoker remote service at [https://RSAprimary.abccompany.com:7072/operations-console/dispatcher/HttpInvokerPlannerPromotion]; nested exception
javax.net.ssl.SSLException: Certificate not verified

SUCCESS: The software version of this instance matches the primary instance...
  • Checking that the primary instance is reachable and healthy….
  • Attempting to reach the Operations Console on the primary instance: rsa02….
  • Checking that all instances are reachable and healthy….
  • Checking continueonerror replication state on: rsa02...
  • Checking replication status of replica instances and reachability to other replica instances….
ERROR: The Operations Console on the primary instance is not reachable to check replication status or reachability with other replica instances.

Checking the replication status of all RADIUS servers….
ERROR: Could not access HTTP invoker remote service at [https://rsa02.xxxxx.zzz:7072/operations-console/dispatcher/HttpInvokerPlannedPromotion]; nested exception is javax.net.ssl.SSLException: Certificate not verified.
Cause

The original RSA Authentication Manager primary server rsa02 has a replacement console certificate, while the replica being promoted has RSA self-signed console certificate. Or if primary and replica have console replacement certs signed by different CAs, e.g. Entrust and Komodo. Note: GoDaddy.com and GoDaddy Group are different CAs. Because of this the replica does not trust the primary replacement console certificate, and vice versa.

Resolution

To resolve this issue,

  1. If you need to obtain the RootCA file from the original primary, rsa02, refer to the article on how to obtain the RSA root CA certificate from RSA Authentication Manager 8.x. In this example we will name the cert file Go_Daddy_Root_Certificate_Authority_C3_84_6B.cer
  2. If you need to obtain the RootCA file from the replica, rsa06, repeat step 1 and name the cert file e.g. Go_Daddy_Root_Certificate_Authority_G2_45_14_0B.cer
  3. Obtain the SSL Trust Store File Password on the replica, in our example rsa06. Repeat on primary, in our example rsa06
    1. First, enable Enable Secure Shell on the Appliance.
    2. Then log On to the Appliance Operating System with SSH.
    3. Go to /opt/rsa/am/utils.
    4. Run the following command:
./rsautil manage-secrets -a listall
rsautil_manage-secrets_listall
    1. Scroll down the list to find the SSL Trust Store File Password. This value is different in each deployment of RSA Authentication Manager.
  1. Make a backup of your /opt/rsa/am/server/security/trust.jks file on the replica. In SSH,
SCP/SFTP Go_Daddy_Root_Certificate_Authority_G2_45_14_0B.cer to RSA02 /tmp
SSH RSA02
cd /opt/rsa/am/server/security/
cp trust.jks trust.jks.original
/opt/rsa/am/appserver/jdk/jre/bin/keytool -importcert -alias GoDadComRoot -keystore ./trust.jks -file /tmp/Go_Daddy_Root_Certificate_Authority_G2_45_14_0B.cer

List the contents of the /opt/rsa/am/server/security/trust.jks on the replica:

../../appserver/jdk/bin/keytool -list -keystore ./trust.jks
No password needed for list
  1. Import the Primary replacement Console Root CA signing Certificate file into trust.jks Java Key Store file on the replica with keytool -importcert
    ../../appserver/jdk/bin/keytool -importcert  -keystore ./trust.jks -file //tmp/Go_Daddy_Root_Certificate_Authority_C3_84_6B.cer      or ../../appserver/jdk/bin/keytool -importcert -alias GoDadComRoot -keystore ./trust.jks -file /tmp/Go_Daddy_Root_Certificate_Authority_C3_84_6B.cer
    Enter Keystore password: s6TD7qb7M91kYWa5YoIdey8vvjPIMC  DOES NOT DISPLAY
    Trust  this certificate? [no]:  yes
    Certificate was added to keystore

If no -alias is specified on keytool command line, -list will display all aliases, but -importcert will default to alias = mycert

Verify that the primary's Root CA cert imported successfully by listing the contents of the /opt/rsa/am/server/security/trust.jks on the replica again.
    ../../appserver/jdk/bin/keytool -list -alias GoDaddy_Group_CA -keystore ./trust.jks
No password needed for list
Your keystore contains 9 entries.
Alias name: GoDaddy_Group_CA                                [default = mykey, can only be used once]
Creation date: Aug 17, 2020
Entry type: trustedCertEntry
Owner: CN=Go Daddy Secure Certificate Authority - G2, OU=.../repository/, O="GoDaddy.com, Inc."
Issuer: CN=Go Daddy Root Certificate Authority - G2, O="GoDaddy.com, Inc."
Valid from: Tue May 21 16:48:38 EDT 2019 until: Mon May 20 16:58:37 EDT 2024

         Step 6 Restart AM services -       /opt/rsa/am/server/rsaserv restart all

rsaserv_restart_all 
 Repeat process on Primary, copy replica Root CA to primary and import into trust.jks

SCP/SFTP Go_Daddy_Root_Certificate_Authority_G2_45_14_0B.cer to RSA06 /tmp
SSH RSA06
cd /opt/rsa/am/server/security/
cp trust.jks trust.jks.original
/opt/rsa/am/appserver/jdk/jre/bin/keytool -importcert -alias GoDadComRoot -keystore ./trust.jks -file /tmp/Go_Daddy_Root_Certificate_Authority_G2_45_14_0B.cer

 

Workaround
Notes
  1. RSA Support strongly recommends making backup copies of any Java Key Store, .JKS that your edit
  2. RSA Support also strongly recommends against deleting any certificate or keys with keytool, as you could make your AM server inoperable.  BE VERY CAREFUL with keytool.  Open a Support Case for Assistance.
  3. SSL Trust Store File Password is only displayed with ./rsautil manage-secrets -a listall 
         ./rsautil manage-secrets -a list com.rsa.ssl.trust.store.password does not provide anything of use, it only displays some default passwords

    Related info: found in this article on how to delete old or pending certificate signing requests for RSA Authentication Manager console or virtual host replacement certificates.