AFX Server stuck in 'Not running' State, with error 'timed out waiting for AFX applications to start'
a day ago
Originally Published: 2014-12-18
Article Number
000050087
Applies To
  • Product: RSA Identity Governance & Lifecycle
  • Versions: 6.9.1, 7.x, 8.0.0
  • Component: AFX Server

 

Issue

The AFX Server fails to start and remains in a Not running state in the RSA Identity Governance & Lifecycle UI under AFX > Servers. 
 

User-added image
 

Observable symptoms:

  • The AFX Server status shows Not running in the UI and does not recover after restart attempts.
  • Running afx status as the afx user on the application server shows the startup timed out:
$ afx status
● afx_server.service - Afx Server
   Loaded: loaded (/etc/systemd/system/afx_server.service; enabled; vendor preset: disabled)
   Active: active (exited) since Mon 2020-01-06 12:30:28 EST; 11s ago
  Process: 19999 ExecStop=/etc/init.d/afx_server stop (code=exited, status=0/SUCCESS)
  Process: 20643 ExecStart=/etc/init.d/afx_server start (code=exited, status=0/SUCCESS)
 Main PID: 20643 (code=exited, status=0/SUCCESS)

Jan 06 12:29:18 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:29:28 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:29:38 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:29:48 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:29:58 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:30:08 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:30:18 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:30:28 acm-711 afx_server[20643]: WARNING!! Timed out waiting for AFX applications to start. 
Please check AFX application log files for detailed status information.
Jan 06 12:30:28 acm-711 afx_server[20643]: done
Jan 06 12:30:28 acm-711 systemd[1]: Started Afx Server.

 

When strting AFX, the following key errors appear in the AFX log files: 
 
In the $AFX_HOME/esb/logs/mule_ee.log:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '10_AFX-INIT', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: CertPathBuilderException: Could not build a validated path.
...
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: 
java.security.cert.CertPathBuilderException: Could not build a validated path.
...
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: 
Could not build a validated path.
 ...  
Caused by: java.security.cert.CertPathBuilderException: Could not build a validated path.
        at com.rsa.cryptoj.o.qb.engineBuild(Unknown Source)

and

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '15_AFX-MAIN', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: IllegalArgumentException: Could not resolve placeholder 
'afx.server.activemq.password' in string value "${afx.server.activemq.password}"

In the $AFX_HOME/esb/logs/esb.AFX-INIT.log:
 
2020-01-06 12:27:24.425 [ERROR] com.aveksa.afx.server.init.SubmitInitializationRequestComponent:162 - 
Unable to establish secure (SSL) connection with RSA Identity Governance and Lifecycle server.
2020-01-06 12:27:24.425 [ERROR] com.aveksa.afx.server.init.SubmitInitializationRequestComponent:171 - 
SSL certificates for RSA Identity Governance and Lifecycle server and AFX were not issued by the same 
RSA Identity Governance and Lifecycle Certificate Authority(CA). You may encounter this problem if the 
RSA Identity Governance and Lifecycle certificate store has been changed, but either the 
RSA Identity Governance and Lifecycle server OR AFX installation hasn't been updated with the respective 
keystore containing new certificate and CA entries. Please update both the 
RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the 
RSA Identity Governance and Lifecycle application.
2020-01-06 12:27:24.426 [ERROR] com.aveksa.afx.server.init.ServerInitializationComponent:79 - 
Server initialization failed! Please correct the issue and restart AFX.

and

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path building failed: java.security.cert.CertPathBuilderException: Could not build a validated path.

In the $AFX_HOME/esb/logs/esb.AFX-MAIN.log:
java.lang.IllegalArgumentException: Could not resolve placeholder 'afx.server.activemq.password' 
in string value "${afx.server.activemq.password}"

Version 8.0.0 >>>

When starting AFX, the following errors are logged to the AFX log files
  • In the $AFX_HOME/esb/logs/esb.AFX-INIT.log:
ERROR 2024-09-04 16:28:36,140 [[MuleRuntime].uber.06: [05-AFX-INIT].heartbeatFlow.BLOCKING @e2c8c4d] [processor: ; event: 42d59090-6afc-11ef-b468-00505601403a] org.mule.runtime.core.internal.exception.DefaultSystemExceptionStrategy:
********************************************************************************
Message               : HTTP POST on resource 'https://acm-vapp.vcloud.local:8444/aveksa/afx/heartbeat' failed: Received fatal alert: certificate_unknown.
Element               : heartbeatFlow/processors/7 @ 05-AFX-INIT:afx-init.xml:234 (Request)
Element DSL           : <http:request method="POST" doc:name="Request" doc:id="45e475c3-af71-4a7d-9660-c6fd7c38703a" config-ref="HTTPS_Request_configuration" path="${afx.config.heartbeat.request.path}" outputMimeType="text/plain" responseTimeout="${afx.config.heartbeat.response.timeout}">
<http:headers><![CDATA[
#[output application/java
---
{
        "afx.server.id" : "${afx.config.server.id}",
        "afx.server.version" : "${afx.config.server.version}"
}]
]]></http:headers>
<http:response-validator>
<http:success-status-code-validator values="200"></http:success-status-code-validator>
</http:response-validator>
</http:request>
Error type            : HTTP:CONNECTIVITY
FlowStack             : at heartbeatFlow(heartbeatFlow/processors/7 @ 05-AFX-INIT:afx-init.xml:234 (Request))

  (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
  • In the $AFX_HOME/esb/logs/esb.AFX-MAIN.log:
ERROR 2024-09-04 16:28:34,329 [[MuleRuntime].uber.07: [10-AFX-MAIN].uber@org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain.initialise:648 @7c76f073] [processor: PRIMARY_REQUEST/processors/5/route/2/processors/1; event: 416f22c0-6afc-11ef-b468-00505601403a] org.mule.runtime.core.internal.exception.OnErrorPropagateHandler:
********************************************************************************
Message               : HTTP POST on resource 'https://acm-vapp.vcloud.local:8444/aveksa/afx/primary' failed: Received fatal alert: certificate_unknown.
Element               : PRIMARY_REQUEST/processors/5/route/2/processors/1 @ 10-AFX-MAIN:primary-request-components.xml:126 (ACM Https Request)
Element DSL           : <http:request method="POST" doc:name="ACM Https Request" doc:id="751537dc-f9b0-4f2f-8ba1-7d4e0a5ad9f8" config-ref="HTTPS_Request_configuration" path="${afx.config.primary.request.path}">
<http:headers><![CDATA[
#[output application/java
---
{
        "afx.server.id" : "${afx.config.server.id}",
        "afx.server.version" : "${afx.config.server.version}"
}]
]]></http:headers>
</http:request>
Error type            : HTTP:CONNECTIVITY
FlowStack             : at PRIMARY_REQUEST(PRIMARY_REQUEST/processors/5/route/2/processors/1 @ 10-AFX-MAIN:primary-request-components.xml:126 (ACM Https Request))

  (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************




 
Cause

The AFX Server fails to start because the Default Truststore Password configured in the RSA Identity Governance & Lifecycle UI does not match the actual password of the JDK truststore (cacerts). 

When AFX initializes, it attempts to access the JDK truststore using the password stored in its configuration. If the two passwords do not match, the SSL context cannot be established and the server fails to start.

This commonly happens when:

  • The JDK is updated or reinstalled, resetting the cacerts password.
  • Someone manually changed the cacerts password without updating the AFX Server configuration in the UI.
  • The AFX Server was configured with an incorrect password during initial setup.


Please see RSA Knowledge Base Article 000038314 -- How to update the root (server) and client certificates in RSA Identity Governance & Lifecycle for possible root causes for this error.

Resolution
 
To resolve this error:
  • Generate a new root (server) certificate for each AFX server and remote agent.
  • Generate a new client certificate for each AFX server and remote agent.
  • Redeploy all certificates.
  • Restart the RSA Identity Governance & Lifecycle application, the AFX application, and the remote agents.