RADIUS Servers status become "Cannot Determine Status" both Primary and Replica when rollbacking from AM8.4 P14 to P13.
Originally Published: 2021-02-05
Article Number
Applies To
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.4..0.13
Platform: null
Platform (Other): null
O/S Version: null
Product Name: null
Product Description: null
Issue
RADIUS Servers status become "Cannot Determine Status" both Primary and Replica when rollbacking from AM8.4 P14 to P13.
In this state, when clicking ”Initiate Replication, below error is occurred.
- In Primary
There was a problem processing your request. An unknown system error occurred.
imsConsoleTrace.log shows below error.
@@@2020-10-20 15:21:37,210, [[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'], (EJBRemoteTargetBase.java:187), trace.com.rsa.command.EJBRemoteTargetBase, ERROR, naka-am840-01.cs-rsa.local,,,,Exception during command execution.@@@2020-10-20 15:21:37,210, [[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'], (EJBRemoteTargetBase.java:187), trace.com.rsa.command.EJBRemoteTargetBase, ERROR, naka-am840-01.cs-rsa.local,,,,Exception during command execution.javax.ejb.EJBException: Unpredictable runtime error is caught. See the nested exception. at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:105) at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:101) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52) at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:94) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy262.executeCommand(Unknown Source) at com.rsa.command.CommandServerEjb30_vraifm_CommandServerEjb30Impl.__WL_invoke(Unknown Source) at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invokeInternal(SessionRemoteMethodInvoker.java:54) at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:21) at com.rsa.command.CommandServerEjb30_vraifm_CommandServerEjb30Impl.executeCommand(Unknown Source) at com.rsa.command.CommandServerEjb30_vraifm_CommandServerEjb30Impl_WLSkel.invoke(Unknown Source) at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:645) at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:246) at weblogic.rmi.internal.BasicServerRef$2.run(BasicServerRef.java:534) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:370) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163) at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:531) at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:138) at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352) at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337) at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57) at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41) at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420) at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)Caused by: java.lang.AbstractMethodError: javax.net.ssl.X509ExtendedTrustManager.checkServerTrusted([Ljava/security/cert/X509Certificate;Ljava/lang/String;Ljava/net/Socket;)V at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:630) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:465) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:361) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:376) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:451) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:428) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:184) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:154) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1198) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1107) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:400) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:727) at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:74) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1012) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:827) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1525) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1975) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:993) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324) at com.rsa.authmgr.internal.radius.sbr.xui.impl.XUIAccessImpl.read(XUIAccessImpl.java:344) at com.rsa.authmgr.internal.admin.radius.impl.RadiusServerAdministrationImpl.lookupSBRRadiusServer(RadiusServerAdministrationImpl.java:570) at com.rsa.authmgr.internal.admin.radius.impl.RadiusServerAdministrationImpl.lookupSBRRadiusServer(RadiusServerAdministrationImpl.java:558) at com.rsa.authmgr.internal.admin.radius.impl.RadiusServerAdministrationImpl.lookupSBRRadiusServer(RadiusServerAdministrationImpl.java:1854) at com.rsa.authmgr.internal.admin.radius.impl.RadiusServerAdministrationImpl.ensureRadiusReplicationIsEnabled(RadiusServerAdministrationImpl.java:1807) at com.rsa.authmgr.internal.admin.radius.impl.RadiusReplicationAdministrationImpl.ensureRadiusReplicationIsEnabled(RadiusReplicationAdministrationImpl.java:354) at com.rsa.authmgr.internal.admin.radius.impl.RadiusReplicationAdministrationImpl.publishReplication(RadiusReplicationAdministrationImpl.java:327) at com.rsa.authmgr.internal.admin.radius.impl.RadiusReplicationAdministrationImpl.publishReplication(RadiusReplicationAdministrationImpl.java:301) at com.rsa.authmgr.admin.radius.PublishReplicationCommand$Executive.execute(PublishReplicationCommand.java:161) at com.rsa.authmgr.admin.radius.PublishReplicationCommand.performExecute(PublishReplicationCommand.java:69) at com.rsa.command.LocalTarget.executeCommand(LocalTarget.java:119) at com.rsa.ims.command.LocalTransactionalCommandTarget.access$0(LocalTransactionalCommandTarget.java:1) at com.rsa.ims.command.LocalTransactionalCommandTarget$2.doInTransaction(LocalTransactionalCommandTarget.java:268) at com.rsa.ims.command.LocalTransactionalCommandTarget$2.doInTransaction(LocalTransactionalCommandTarget.java:1) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:131) at com.rsa.ims.command.LocalTransactionalCommandTarget.executeCommand(LocalTransactionalCommandTarget.java:260) at com.rsa.command.CommandServerEngine$CommandExecutor.run(CommandServerEngine.java:933) at com.rsa.command.CommandServerEngine$CommandExecutor.run(CommandServerEngine.java:1) at com.rsa.ims.security.spi.SimpleSecurityContextImpl.doAs(SimpleSecurityContextImpl.java:113) at com.rsa.security.SecurityContext.doAs(SecurityContext.java:439) at com.rsa.command.CommandServerEngine.executeCommand(CommandServerEngine.java:445) at com.rsa.command.CommandServerEngine.executeCommand(CommandServerEngine.java:373) at com.rsa.command.CommandServerBean.executeCommand(CommandServerBean.java:89) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101) ... 34 more
- Replica
There was a problem processing your request. You are currently accessing a read-only replica instance. Please try your request again on a primary instance.
imsConsoleTrace.log shows below error.
@@@2020-10-20 18:09:21,609, [[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'], (EJBRemoteTargetBase.java:187), trace.com.rsa.command.EJBRemoteTargetBase, ERROR, naka-am840-02.cs-rsa.local,,,,Exception during command execution.
com.rsa.command.CommandRejectedException: Command is rejected, try execute this command at primary instance
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:309)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:555)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:313)
at com.rsa.command.CommandServerEjb30_vraifm_CommandServerEjb30Impl_12213_WLStub.executeCommand(Unknown Source)
at sun.reflect.GeneratedMethodAccessor445.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:87)
at com.sun.proxy.$Proxy309.executeCommand(Unknown Source)
at com.rsa.command.EJBRemoteTargetBase$CommandExecutor.run(EJBRemoteTargetBase.java:261)
at com.rsa.command.EJBRemoteTargetBase$CommandExecutor.run(EJBRemoteTargetBase.java:1)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:370)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
at weblogic.security.Security.runAs(Security.java:62)
at com.rsa.command.WebLogicSecurityContextWrapper.runAs(WebLogicSecurityContextWrapper.java:51)
at com.rsa.command.EJBRemoteTargetBase.executeCommand(EJBRemoteTargetBase.java:168)
at com.rsa.command.TargetableCommand.execute(TargetableCommand.java:295)
at com.rsa.command.TargetableCommand.execute(TargetableCommand.java:263)
at com.rsa.ims.management.console.util.CommandUtil.executeCommand(CommandUtil.java:85)
at com.rsa.authmgr.web.console.action.handler.RadiusServerHandler.notifyAll(RadiusServerHandler.java:170)
at com.rsa.authmgr.web.console.action.RadiusServerAction.nvReplicate(RadiusServerAction.java:152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:269)
at com.rsa.ui.common.struts.action.RSABaseDispatchAction.execute(RSABaseDispatchAction.java:180)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
at com.rsa.ims.web.console.common.ConsoleRequestProcessor.processActionPerform(ConsoleRequestProcessor.java:66)
at com.rsa.ui.common.util.RSAWebRequestProcessor.process(RSAWebRequestProcessor.java:221)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1926)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:464)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:100)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.rsa.ui.common.filter.GZIPFilter.doFilterInternal(GZIPFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.rsa.ui.common.filter.UrlValidationFilter.doFilter(UrlValidationFilter.java:133)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.rsa.ui.common.security.csrf.CSRFFilter.doFilterInternal(CSRFFilter.java:196)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.rsa.ims.management.console.security.filter.RSAConsoleSignOnFilter$1.run(RSAConsoleSignOnFilter.java:142)
at com.rsa.ims.security.spi.SimpleSecurityContextImpl.doAs(SimpleSecurityContextImpl.java:113)
at com.rsa.security.SecurityContext.doAs(SecurityContext.java:439)
at com.rsa.ims.management.console.security.filter.RSAConsoleSignOnFilter.doFilter(RSAConsoleSignOnFilter.java:139)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.rsa.ims.sso.filter.SSOFilter.doFilter(SSOFilter.java:710)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.rsa.ui.common.filter.I18NFilter.doFilter(I18NFilter.java:96)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at com.rsa.ims.sso.filter.HeaderValidationFilter.doFilter(HeaderValidationFilter.java:174)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3706)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3672)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:328)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2443)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2291)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2269)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1705)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1665)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Cause
1.AM8.4 P13
JDK 1.8.0_251、JSAFE crypt-j 6.2.4
2.AM8.4 P14
JDK 1.8.0_261(Version Up)、JSAFE crypt-j 6.2.5(Version Up)
３．AM8.4 P13にロールバック
JDK 1.8.0_261(NOT rollback)、JSAFE crypt-j 6.2.4(rollback)
The JSAFE 6.2.4 does not have compatibility with the JDK 1.8.0_261.
Workaround
1.After rollback from AMP13. Confirm the JDK version
rsaadmin@am840-01:~> /opt/rsa/am/appserver/jdk/jre/bin/java -version java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b25) Java HotSpot(TM) 64-Bit Server VM (build 25.261-b25, mixed mode)
2. Check the backup file in /opt/rsa/am/updates/..(Both Primary and Replica)
rsaadmin@am840-01:~> ls -ltr /opt/rsa/am/updates/*jdk*.tar.gz
-r-------- 1 rsaadmin rsaadmin 120368578 Sep 13 2019 /opt/rsa/am/updates/backup_jdk_190913123102.tar.gz
-r-------- 1 rsaadmin rsaadmin 123631179 Dec 4 2019 /opt/rsa/am/updates/backup_jdk_191204104428.tar.gz
-r-------- 1 rsaadmin rsaadmin 122712814 Sep 25 10:04 /opt/rsa/am/updates/backup_jdk_200925100341.tar.gz
-r-------- 1 rsaadmin rsaadmin 124074167 Oct 19 16:20 /opt/rsa/am/updates/backup_jdk_201019162022.tar.gz
→最後の、backup_jdk_201019162022.tar.gzを復元する。
3. Rename current JDK and restore the JDK backup.(Both Primary and Replica)
rsaadmin@am840-01:/opt/rsa/am/appserver> mv jdk jdk-p14 rsaadmin@naka-am840-01:/opt/rsa/am/appserver> tar -xzf /opt/rsa/am/updates/backup_jdk_201019162022.tar.gz
Then, you can confirm the JDK version is the same of before update AMP13..(Both Primary and Replica)
rsaadmin@am840-01:/opt/rsa/am/appserver> ./jdk/jre/bin/java -version java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08) Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)
5. Restart AM service
rsaadmin@am840-01:/opt/rsa/am/appserver> /opt/rsa/am/server/rsaserv restart all
6. Confirm the RADIUS server is synchronized both Primary and Replica in Security Console.
