AFX Server remains in a 'Not running' State, afx status shows 'timed out waiting for AFX applications to start' and mule_ee.log has a 'Could not build a validated path' error in RSA Identity Governance & Lifecycle
8 months ago
Originally Published: 2014-12-18
Article Number
000050087
Applies To
RSA Product Set: RSA Identity Governance & Lifecycle 
RSA Version/Condition: 6.9.1, 7.x, 8.0.0

 
Issue
The AFX Server in RSA Identity Governance & Lifecycle is in a Not running State in the user interface (AFX > Servers).
 
User-added image
 
When logged into the application server as the afx user, the afx status command shows the startup timed out and the AFX Server never fully starts.
 
$ afx status
● afx_server.service - Afx Server
   Loaded: loaded (/etc/systemd/system/afx_server.service; enabled; vendor preset: disabled)
   Active: active (exited) since Mon 2020-01-06 12:30:28 EST; 11s ago
  Process: 19999 ExecStop=/etc/init.d/afx_server stop (code=exited, status=0/SUCCESS)
  Process: 20643 ExecStart=/etc/init.d/afx_server start (code=exited, status=0/SUCCESS)
 Main PID: 20643 (code=exited, status=0/SUCCESS)

Jan 06 12:29:18 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:29:28 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:29:38 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:29:48 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:29:58 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:30:08 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:30:18 acm-711 afx_server[20643]: Waiting for AFX applications to start...
Jan 06 12:30:28 acm-711 afx_server[20643]: WARNING!! Timed out waiting for AFX applications to start. 
Please check AFX application log files for detailed status information.
Jan 06 12:30:28 acm-711 afx_server[20643]: done
Jan 06 12:30:28 acm-711 systemd[1]: Started Afx Server.
 
When starting AFX, the following errors are logged to the AFX log files:
 
In the $AFX_HOME/esb/logs/mule_ee.log:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '10_AFX-INIT', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: CertPathBuilderException: Could not build a validated path.
...
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: 
java.security.cert.CertPathBuilderException: Could not build a validated path.
...
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: 
Could not build a validated path.
 ...  
Caused by: java.security.cert.CertPathBuilderException: Could not build a validated path.
        at com.rsa.cryptoj.o.qb.engineBuild(Unknown Source)

and

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Failed to deploy artifact '15_AFX-MAIN', see below       +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
org.mule.module.launcher.DeploymentInitException: IllegalArgumentException: Could not resolve placeholder 
'afx.server.activemq.password' in string value "${afx.server.activemq.password}"

In the $AFX_HOME/esb/logs/esb.AFX-INIT.log:
 
2020-01-06 12:27:24.425 [ERROR] com.aveksa.afx.server.init.SubmitInitializationRequestComponent:162 - 
Unable to establish secure (SSL) connection with RSA Identity Governance and Lifecycle server.
2020-01-06 12:27:24.425 [ERROR] com.aveksa.afx.server.init.SubmitInitializationRequestComponent:171 - 
SSL certificates for RSA Identity Governance and Lifecycle server and AFX were not issued by the same 
RSA Identity Governance and Lifecycle Certificate Authority(CA). You may encounter this problem if the 
RSA Identity Governance and Lifecycle certificate store has been changed, but either the 
RSA Identity Governance and Lifecycle server OR AFX installation hasn't been updated with the respective 
keystore containing new certificate and CA entries. Please update both the 
RSA Identity Governance and Lifecycle server and AFX installations with latest respective keystore available for download in the 
RSA Identity Governance and Lifecycle application.
2020-01-06 12:27:24.426 [ERROR] com.aveksa.afx.server.init.ServerInitializationComponent:79 - 
Server initialization failed! Please correct the issue and restart AFX.

and

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path building failed: java.security.cert.CertPathBuilderException: Could not build a validated path.

In the $AFX_HOME/esb/logs/esb.AFX-MAIN.log:
java.lang.IllegalArgumentException: Could not resolve placeholder 'afx.server.activemq.password' 
in string value "${afx.server.activemq.password}"

Version 8.0.0 >>>

When starting AFX, the following errors are logged to the AFX log files
  • In the $AFX_HOME/esb/logs/esb.AFX-INIT.log:
ERROR 2024-09-04 16:28:36,140 [[MuleRuntime].uber.06: [05-AFX-INIT].heartbeatFlow.BLOCKING @e2c8c4d] [processor: ; event: 42d59090-6afc-11ef-b468-00505601403a] org.mule.runtime.core.internal.exception.DefaultSystemExceptionStrategy:
********************************************************************************
Message               : HTTP POST on resource 'https://acm-vapp.vcloud.local:8444/aveksa/afx/heartbeat' failed: Received fatal alert: certificate_unknown.
Element               : heartbeatFlow/processors/7 @ 05-AFX-INIT:afx-init.xml:234 (Request)
Element DSL           : <http:request method="POST" doc:name="Request" doc:id="45e475c3-af71-4a7d-9660-c6fd7c38703a" config-ref="HTTPS_Request_configuration" path="${afx.config.heartbeat.request.path}" outputMimeType="text/plain" responseTimeout="${afx.config.heartbeat.response.timeout}">
<http:headers><![CDATA[
#[output application/java
---
{
        "afx.server.id" : "${afx.config.server.id}",
        "afx.server.version" : "${afx.config.server.version}"
}]
]]></http:headers>
<http:response-validator>
<http:success-status-code-validator values="200"></http:success-status-code-validator>
</http:response-validator>
</http:request>
Error type            : HTTP:CONNECTIVITY
FlowStack             : at heartbeatFlow(heartbeatFlow/processors/7 @ 05-AFX-INIT:afx-init.xml:234 (Request))

  (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
  • In the $AFX_HOME/esb/logs/esb.AFX-MAIN.log:
ERROR 2024-09-04 16:28:34,329 [[MuleRuntime].uber.07: [10-AFX-MAIN].uber@org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain.initialise:648 @7c76f073] [processor: PRIMARY_REQUEST/processors/5/route/2/processors/1; event: 416f22c0-6afc-11ef-b468-00505601403a] org.mule.runtime.core.internal.exception.OnErrorPropagateHandler:
********************************************************************************
Message               : HTTP POST on resource 'https://acm-vapp.vcloud.local:8444/aveksa/afx/primary' failed: Received fatal alert: certificate_unknown.
Element               : PRIMARY_REQUEST/processors/5/route/2/processors/1 @ 10-AFX-MAIN:primary-request-components.xml:126 (ACM Https Request)
Element DSL           : <http:request method="POST" doc:name="ACM Https Request" doc:id="751537dc-f9b0-4f2f-8ba1-7d4e0a5ad9f8" config-ref="HTTPS_Request_configuration" path="${afx.config.primary.request.path}">
<http:headers><![CDATA[
#[output application/java
---
{
        "afx.server.id" : "${afx.config.server.id}",
        "afx.server.version" : "${afx.config.server.version}"
}]
]]></http:headers>
</http:request>
Error type            : HTTP:CONNECTIVITY
FlowStack             : at PRIMARY_REQUEST(PRIMARY_REQUEST/processors/5/route/2/processors/1 @ 10-AFX-MAIN:primary-request-components.xml:126 (ACM Https Request))

  (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************




 
Cause
server.keystore (ACM) and the client.keystore (AFX) can get out of sync with each other which can cause AFX to fail.

Please see RSA Knowledge Base Article 000038314 -- How to update the root (server) and client certificates in RSA Identity Governance & Lifecycle for possible root causes for this error.
Resolution
The process to resolve this error is to generate a new root (server) certificate and a new client certificate for each AFX server and remote agent, redeploy all certificates, and restart the RSA Identity Governance & Lifecycle  application, AFX application, and remote agents. 

This process is described in detail in RSA Knowledge Base Article 000038314 -- How to update the root (server) and client certificates in RSA Identity Governance & Lifecycle.