​​​​​​​Error 413--Request Entity Too Large, now system cannot be restarted when updating RSA Authentication Manager 8.3.0.5.0 to 8.4 via web browser
2 months ago
Originally Published: 2018-12-13
Article Number
000040296
Applies To
RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.3.0.5.0 
Issue
  • With a server running RSA Authentication Manager 8.3.0.5.0, when uploading the 8.4 update to the Operations Console via any browser, it processes for a few minutes then fails with the following error:
Error 413--Request Entity Too Large
  • After the error, the user is prompted for the RSA admin password.
  • This leaves the system in a state where a reboot or restart of services fails.


 
Cause
The /opt/rsa/am/server/pending/config.xml is a stale file that should not ordinarily exist on RSA servers.  The root cause as to why the file is there is as-of-yet unknown.  This file does no harm to anything that we know of outside of the scope of this particular knowledge article.

The /opt/rsa/am/server/config/config.xml is modified the instant the upload starts and before the rsaadmin password is requested.  The config.xml modification is done in order to accommodate a file size of over 2GB for the upload.  When it fails, the config.xml is not reverted back to the original version; rather, it is left behind with incorrect pointers to the 8.3.0.4.0 files, which do not exist on an 8.3.0.5.0 system.  Because of this discrepancy, if the system is ever restarted after this occurs, services will not start.
 
Resolution

I.  To get system operational again after failure

This modification has been approved by CE as a valid workaround to get the server running again.

  1. Open an SSH session to the server.
  2. Manually edit the /opt/rsa/am/server/config/config.xml.
  3. Change any references in the file for 0.4.0 to 0.5.0.  See the Workaround section below for an easy way to edit the config.xml using the sed command.

Do not change items which may show 8.3.0.0.0, these you leave alone.

  1. Restart services:
/opt/rsa/am/server/rsaserv restart all
 


II.  To get Authentication Manager 8.4 to update from 8.3.0.5.0 successfully where it failed earlier on browser upload


1.  Update via hotfix

This older solution uses the hotfix.  It still works, but a better method exists below with option 2 below.

Note that a customer must first open a case and request the hotfix files for AM-32518 from RSA Support.
  1. Obtain the hotfix zip file from AM-32518 or attached to this article.
  2. Follow the instructions in the readme, as they may be updated. but the instructions are also written below.
  3. There will be one jar file named ims-container-weblogic-8.3.0.5.0.jar, which replaces four copies of the jar file in four directories.
    1. Copy the hotfix jar file to /tmp.
    2. Create four .bak files:
mv /opt/rsa/am/server/servers/AdminServer/tmp/_WL_user/console-shared-library/8hkrcb/WEB-INF/lib/ims-container-weblogic-8.3.0.5.0.jar /opt/rsa/am/server/servers/AdminServer/tmp/_WL_user/console-shared-library/8hkrcb/WEB-INF/lib/ims-container-weblogic-8.3.0.5.0.jar-bak
 
mv /opt/rsa/am/server/servers/biztier/tmp/_WL_user/am-app/mxboc6/APP-INF/lib/ims-container-weblogic-8.3.0.5.0.jar /opt/rsa/am/server/servers/biztier/tmp/_WL_user/am-app/mxboc6/APP-INF/lib/ims-container-weblogic-8.3.0.5.0.jar-bak
 
mv /opt/rsa/am/server/servers/radiusoc/tmp/_WL_user/am-radius-app/cbsd0y/APP-INF/lib/ims-container-weblogic-8.3.0.5.0.jar /opt/rsa/am/server/servers/radiusoc/tmp/_WL_user/am-radius-app/cbsd0y/APP-INF/lib/ims-container-weblogic-8.3.0.5.0.jar-bak
 
mv /opt/rsa/am/server/servers/console/tmp/_WL_user/console-shared-library/t5l98w/WEB-INF/lib/ims-container-weblogic-8.3.0.5.0.jar /opt/rsa/am/server/servers/console/tmp/_WL_user/console-shared-library/t5l98w/WEB-INF/lib/ims-container-weblogic-8.3.0.5.0.jar-bak
  1. Put in the new jar file in the following four locations:
cp /tmp/ims-container-weblogic-8.3.0.5.0.jar /opt/rsa/am/server/servers/AdminServer/tmp/_WL_user/console-shared-library/8hkrcb/WEB-INF/lib/

cp /tmp/ims-container-weblogic-8.3.0.5.0.jar /opt/rsa/am/server/servers/biztier/tmp/_WL_user/am-app/mxboc6/APP-INF/lib/

cp /tmp/ims-container-weblogic-8.3.0.5.0.jar /opt/rsa/am/server/servers/radiusoc/tmp/_WL_user/am-radius-app/cbsd0y/APP-INF/lib/

cp /tmp/ims-container-weblogic-8.3.0.5.0.jar /opt/rsa/am/server/servers/console/tmp/_WL_user/console-shared-library/t5l98w/WEB-INF/lib/
  1. Restart the Authentication Manager services:
/opt/rsa/am/server/rsaserv restart all
  1. If this is a primary, confirm that replication is normal.
  2. Now upload 8.4 update via browser and it will succeed
 

2.  New Better solution from Engineering released 4 January 2019, no hotfix needed

  1. Navigate to the /opt/rsa/am/server/pending directory.
  2. Delete any config.xml files in he directory.  There should be one, which is the root cause of this problem.
  3. Make sure all of the Authentication Manager services are stopped:
/opt/rsa/am/server/rsaserv stop all
  1. Edit the /opt/rsa/am/server/config/config.xml to correct the version numbers as shown in the Workaround section below.
  2. Restart the Authentication Manager services:
/opt/rsa/am/server/rsaserv restart all
  1. If this is a primary, confirm that replication is normal.
  2. Now upload 8.4 update via browser and it will succeed
Workaround

Workaround to get the system functional after failure

Revert to a prior snapshot, or try the steps below:

This modification has been approved by CE as a valid workaround to get Authentication Manager running again.
  1. Manually edit /opt/rsa/am/server/config/config.xml and change any references of 0.4.0 to 0.5.0.
I have only seen 8.3.0.4.0 on an 8.3.0.5.0 system, but do not know if it might affect other versions. So, just edit config.xml and find any references 8.3.0.x.0 which do not match what the current system is, to the correct version for the system as it was before you tried to patch.  in my case it was 8.3.0.4.0 incorrectly listed multiple times in the config.xml on an 8.3.0.5.0 system.
 
These would be the lines needing to be changed from 8.3.0.4.0 to 8.3.0.5.0:

config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/annex/8.3.0.4.0/annex-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/operations-console/8.3.0.4.0/operations-console-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/am-app/8.3.0.4.0/am-app-8.3.0.4.0.ear</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/am-radius-app/8.3.0.4.0/am-radius-app-8.3.0.4.0.ear</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/ims-authn-idp/8.3.0.4.0/ims-authn-idp-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/console-ims/8.3.0.4.0/console-ims-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/ctkip-ws/8.3.0.4.0/ctkip-ws-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/ucm-rba-war/8.3.0.4.0/ucm-rba-war-8.3.0.4.0.war</source-path>
config.xml:   <source-path>/opt/rsa/am/components/compile/com.rsa.am/ucm-rba-scm/8.3.0.4.0/ucm-rba-scm-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/console-selfservice/8.3.0.4.0/console-selfservice-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/console-selfservice-infocenter/8.3.0.4.0/console-selfservice-infocenter-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/console-infocenter/8.3.0.4.0/console-infocenter-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.am/console-shared-library/8.3.0.4.0/console-shared-library-8.3.0.4.0.war</source-path>
config.xml:    <source-path>/opt/rsa/am/components/compile/com.rsa.gtk/guitoolkit-shared-library/8.3.0.4.0/guitoolkit-shared-library-8.3.0.4.0.war</source-path>

 

Using stream editor (sed) to update the config.xml instead of line-by-line editing

  1. Make a backup copy of config.xml

cp /opt/rsa/am/server/config/config.xml-bak

  1. Navigate to /opt/rsa/am/server/config.
  2. Run this sed command to replace all 8.3.0.4 to 8.3.0.5

sed -i -e 's/8\.3\.0\.4/8\.3\.0\.5/g' config.xml

  1. Double check the config.xml for the changes:

fgrep 8.3.0. config.xml

The output should show all lines 8.3.0.5 and some 8.3.0.0.0

  1. Restart services:
/opt/rsa/am/server/rsaserv restart all
 

Workaround to upload 8.4 update via browser successfully on 8.3.0.5.0 system with earlier failure
 

  • The old workaround is to attempt applying the the hotfix as documented in section 1 of the Resolution section above before trying the upgrade to 8.4 again.
  • New Review the Engineering-approved workaround released on  4 January 2019, as documented in section 2 of the Resolution section above before trying the upgrade to 8.4 again.
 

Workaround to bypass this issue altogether

Use NFS or a Windows share and 8.4 will update successfully.