AFX Server remains in a 'Not running' State, afx status shows 'timed out waiting for AFX applications to start' and esb.AFX-INIT.log has a 'Keystore was tampered, or password was incorrect' error in RSA Identity Governance & Lifecycle
8 months ago
Originally Published: 2014-12-17
Article Number
000050075
Applies To
RSA Product Set: RSA Identity Governance &  Lifecycle
RSA Version/Condition: 6.9.1, 7.x, 8.0.0
 
Issue
The AFX Server in RSA Identity Governance & Lifecycle is in a Not running State in the user interface (AFX > Servers).
User-added image

When logged into the application server as the afx user, the afx status command shows the startup timed out and the AFX Server never fully starts. 
$ afx status
● afx_server.service - Afx Server
   Loaded: loaded (/etc/systemd/system/afx_server.service; enabled; vendor preset: disabled)
   Active: active (exited) since Sun 2020-01-05 09:24:06 EST; 1h 37min ago
  Process: 30415 ExecStop=/etc/init.d/afx_server stop (code=exited, status=0/SUCCESS)
  Process: 31129 ExecStart=/etc/init.d/afx_server start (code=exited, status=0/SUCCESS)
 Main PID: 31129 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)
   CGroup: /system.slice/afx_server.service

Jan 05 09:23:06 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:16 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:26 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:36 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:46 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:23:56 acm-711 afx_server[31129]: Waiting for AFX applications to start...
Jan 05 09:24:06 acm-711 afx_server[31129]: WARNING!! Timed out waiting for AFX applications to start. 
Please check AFX application log files for detailed status information.
Jan 05 09:24:06 acm-711 afx_server[31129]: done
Jan 05 09:24:06 acm-711 systemd[1]: Started Afx Server.

When starting AFX, the following errors are logged to the AFX log files:

$AFX_HOME/esb/logs/mule_ee.log:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '10_AFX-INIT', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: UnrecoverableKeyException: Password verification failed
$AFX_HOME/esb/logs/esb.AFX-INIT.log:
[ERROR] com.aveksa.afx.server.init.ConfigureDefaultSSLContextComponent:107 - Error configuing default SSL context
java.io.IOException: Keystore was tampered with, or password was incorrect

[ERROR] com.aveksa.afx.server.init.ServerInitializationComponent:79 - Server initialization failed! Please correct the issue and restart AFX.
java.io.IOException: Keystore was tampered with, or password was incorrect

Version 8.0.0 >>>

When starting AFX, the following errors are logged to the AFX log files:

$AFX_HOME/esb/logs/mule_ee.log:

Loading certificate for activeMQ JMX service from : /home/oracle/AFX/esb/conf/brokerCert
Error configuring default SSL context: Keystore was tampered with, or password was incorrect
          $AFX_HOME/esb/logs/esb.AFX-INIT.log: 
ERROR 2024-09-04 15:44:45,939 [[MuleRuntime].uber.08: [05-AFX-INIT].initializationFlow.BLOCKING @205d5093] [processor: initializationFlow/errorHandler/0/processors/0/route/4/processors/0; event: 14b80cc1-6af6-11ef-8130-00505601403a] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: The RSA Identity Governance and Lifecycle server was unable to authorize AFX request. This usually indicates that the AFX SSL certificate and/or ID currently configured for this installation do not match with records in the RSA Identity Governance and Lifecycle database.
ERROR 2024-09-04 15:44:46,380 [[MuleRuntime].uber.01: [05-AFX-INIT].initializationFlow.CPU_INTENSIVE @7eac4a2b] [processor: ; event: 14b80cc1-6af6-11ef-8130-00505601403a] org.mule.runtime.core.internal.exception.DefaultSystemExceptionStrategy:
********************************************************************************
Message               : Invocation of Method 'configureDefaultSSLContext(String,String)' from Class 'com.aveksa.afx.server.component.ConfigureDefaultSSLContextComponent' with arguments [java.lang.String arg0, java.lang.String arg1] resulted in an error.
Expected arguments are [java.lang.String arg0, java.lang.String arg1].
Cause: java.io.IOException - Keystore was tampered with, or password was incorrect
Element               : initializationFlow/processors/9 @ 05-AFX-INIT:afx-init.xml:109 (configureDefaultSSLContext)
Element DSL           : <java:invoke doc:name="configureDefaultSSLContext" doc:id="064fa5c0-2d93-484b-b7ed-dbc72322bfbc" class="com.aveksa.afx.server.component.ConfigureDefaultSSLContextComponent" instance="#[vars.defaultSSLContextComponent]" method="configureDefaultSSLContext(String,String)">
<java:args><![CDATA[
#[{
        arg0 : vars.trustore as String,
        arg1 : vars.trustorePassword as String
}]
]]></java:args>
</java:invoke>
Error type            : JAVA:INVOCATION
FlowStack             : at initializationFlow(initializationFlow/processors/9 @ 05-AFX-INIT:afx-init.xml:109 (configureDefaultSSLContext))

  (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************

 
Cause
This issue occurs because the truststore password that is in use for the JDK truststore (cacerts) does not match the Default Truststore Password defined for the AFX Server.
 
Resolution
The default JDK truststore (cacerts) password is changeit. If the default password has not been changed, then that is the password that must be used for the Default Truststore Password under AFX > Servers > {AFX Server name} > Edit. If the default JDK truststore password has been changed, then the modified password must be used in the Default Truststore Password definition for the AFX Server.

To resolve this issue, follow the steps below.
  1. Determine what the JDK truststore password is for cacerts. The default truststore password for cacerts is changeit.
    1. Login as the afx user.
    2. Navigate to the appropriate directory with the following command:  
cd $JAVA_HOME/jre/lib/security
  1. List the keystore contents with the following command:  
keytool -list -v -storepass changeit -keystore cacerts
If the cacerts password is changeit, then this command will list the contents of the keystore. If the cacerts password is not changeit, the following error will occur:
keytool -list -v -storepass rowan -keystore cacerts
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at sun.security.tools.keytool.Main.doCommands(Main.java:839)
        at sun.security.tools.keytool.Main.run(Main.java:368)
        at sun.security.tools.keytool.Main.main(Main.java:361)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
        ... 7 more
If the password is not changeit, someone at your site has changed the password and you will need to determine what that password is.

To confirm that you have the correct password, run the keytool -list command again with the correct password and verify that the truststore contents are listed. 
keytool -list -v -storepass {password} -keystore cacerts
  1. Once you confirm the cacerts password, modify the password in the RSA Identity Governance & Lifecycle user interface. Do this even if the password has not been modified, i.e. if the password is changeit.
    1. In the user interface, go to AFX > Servers > {AFX Server name} > Edit. 
    2. In the Default Truststore Password field, type in the cacerts password.
    3. Press OK.
  2. Restart the AFX service as the afx user.
$ afx start

 

Related Articles

Trending Articles

Don't see what you're looking for?