AFX Requests fails to process as the Usage Store Manager is full in RSA Governance & Lifecycle
2 years ago
Article Number
000067980
Applies To
RSA Product Set: RSA Governance & Lifecycle
RSA Version/Condition:  7.2.x, 7.5.x
Issue
The AFX requests fail as the ActiveMQ kahaDB Storage is full:

The following error is being logged in /home/{afxuser}/AFX/esb/logs/esb.AFX-MAIN.log:
 
2022-10-11 11:09:14.294 [ERROR] org.mule.exception.DefaultMessagingExceptionStrategy:337 -
********************************************************************************
Message               : Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=jms://AFX.INBOUND?connector=jmsConnector, connector=ActiveMQJmsConnector
{  
	name=jmsConnector
	lifecycle=start  
	this=34281e5b  
	numberOfConcurrentTransactedReceivers=1  
	createMultipleTransactedReceivers=true  
	connected=true  
	supportedProtocols=[jms]  
	serviceOverrides=<none>
}
,  name='endpoint.jms.AFX.INBOUND', mep=ONE_WAY, properties={connector=jmsConnector}, transactionConfig=Transaction{factory=null, action=INDIFFERENT, timeout=0}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=10000, endpointEncoding=UTF-8, disableTransportTransformer=false}. Message payload is of type: ActiveMQTextMessageCode                  : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. Usage Manager Store is Full, 100% of 1073741824. Stopping producer (ID:clrv0000234346.ic.ing.net-45827-1664872697929-0:1:729507:1) to prevent flooding queue://AFX.INBOUND. See http://activemq.apache.org/producer-flow-control.html for more info(JMS Code: null) (javax.jms.ResourceAllocationException)  org.apache.activemq.broker.region.BaseDestination:581 (http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/ResourceAllocationException.html)
2. Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=jms://AFX.INBOUND?connector=jmsConnector, connector=ActiveMQJmsConnector
{
  name=jmsConnector
  lifecycle=start
  this=34281e5b
  numberOfConcurrentTransactedReceivers=1
  createMultipleTransactedReceivers=true
  connected=true  supportedProtocols=[jms]
  serviceOverrides=<none>
},
  name='endpoint.jms.AFX.INBOUND', mep=ONE_WAY, properties={connector=jmsConnector}, transactionConfig=Transaction{factory=null, action=INDIFFERENT, timeout=0}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=10000, endpointEncoding=UTF-8, disableTransportTransformer=false}. Message payload is of type: ActiveMQTextMessage (org.mule.api.transport.DispatchException)  org.mule.transport.AbstractMessageDispatcher:117 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/transport/DispatchException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
javax.jms.ResourceAllocationException: Usage Manager Store is Full, 100% of 1073741824. Stopping producer (ID:clrv0000234346.ic.ing.net-45827-1664872697929-0:1:729507:1) to prevent flooding queue://AFX.INBOUND. See http://activemq.apache.org/producer-flow-control.html for more info
    at org.apache.activemq.broker.region.BaseDestination.waitForSpace(BaseDestination.java:581)
    at org.apache.activemq.broker.region.Queue.checkUsage(Queue.java:735)
    at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:665)
    + 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************

 
Cause
This issue may occur if there are many AFX requests in non-terminal state.
Workaround
  1. Log in as the AFX user
  2. Stop AFX
  3. Create a new KahaDB following this article: How to create a new ActiveMQ KahaDB for use with AFX in RSA Identity Governance & Lifecycle
  4. Edit /home/{afxuser}/AFX/activemq/conf/log4j.properties to add the rollover settings:
log4j.appender.kahadb=org.apache.log4j.RollingFileAppender
log4j.appender.kahadb.file=${activemq.base}/data/kahadb.log
log4j.appender.kahadb.maxFileSize=100MB
log4j.appender.kahadb.maxBackupIndex=5
log4j.appender.kahadb.append=true
log4j.appender.kahadb.layout=org.apache.log4j.PatternLayout
log4j.appender.kahadb.layout.ConversionPattern=%d [%-}}\{{15}}{{.15t] %5p %30.30c{1} - %m%n
log4j.logger.org.apache.activemq.store.kahadb.MessageDatabase=TRACE, kahadb
  1. Start AFX