Authentication Manager Web Tier Offline After Hard Shutdown with "System Fingerprint Encrypted Key Is Missing"
5 days ago
Originally Published: 2016-01-21
Article Number
000049834
Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.x
Platform: Red Hat Linux

Issue
After a hard shutdown of the RSA Authentication Manager Web Tier server, the Web Tier services fail to start on reboot and the Web Tier appears offline.
 
Observable symptoms:
  • Web Tier services do not start after the server is powered back on
  • Web Tier status shows as Offline in the Authentication Manager Operations Console
  • The following errors are found in the Web Tier server logs on restart
Where to look: 
  • /opt/RSASecurity/RSAAuthenticationManagerWebtier/server/logs/AdminServer.log
  • /opt/RSASecurity/RSAAuthenticationManagerWebtier/server/logs/AdminServerWrapper.log
Key errors:
 
System fingerprint encrypted key is missing Failed to reload password database
Cause




A hard shutdown corrupts the /opt./rsa/am/utils/etc/systemfields.properties file if the Web Tier's bootstrapper service is writing to it at the moment power is lost.


 


The bootstrapper service writes to /opt./rsa/am/utils/etc/systemfields.properties every two minutes. If the Web Tier server is shut down abruptly; for example, due to a power loss or an unplugged server and the file is mid-write at that moment, it becomes corrupted. When the server reboots, the Web Tier services attempt to read the file, fail, and do not start.


 


This commonly occurs in environments where the Web Tier server does not go through a graceful shutdown sequence (e.g., during a power outage or forced restart).


 


NOTE: The full error stack traces from /opt/rsa/am/server/log/AdminServer.log and and  /opt/rsa/am/server/log/AdminServerWrapper.log are provided below for reference. The two key error lines that confirm this cause are:


 




System fingerprint encrypted key is missing
Failed to reload password database






Resolution
Solution 1: Uninstall and Reinstall the Web Tier


Uninstall the Web Tier and reinstall it. This is the simpler option if you do not need to preserve the existing Web Tier configuration.


 


Note: If you prefer to restore the Web Tier without reinstalling, use Solution 2 below.




Solution 2: Restore the systemfields.properties File


  1. Connect to the primary Authentication Manager server via SSH as rsaadmin. Enter the operating system password when prompted.
  2. Navigate to the /opt./rsa/am/utils/etc directory:




cd /opt/rsa/am/utils/etc/




  1. Confirm that the /opt./rsa/am/utils/etc/systemfields.properties file is present:






ls -alh systemfields.properties


The expected output is:




-rw------- 1 rsaadmin rsaadmin 11K Jan 15 16:22 systemfields.properties




  1. Copy the file to tmp:






cp systemfields.properties /tmp/






  1. Use WinSCP or another SCP client to connect to the primary Authentication Manager server as the rsaadmin user.
  2. Download systemfields.properties from /tmp to your local PC, then disconnect.
  3.   Connect via SSH to the Web Tier server (Linux or Windows).
  4. Use sudo to elevate to the root user (Linux) or administrator (Windows):






sudo su -






  1. Enter the operating system password when prompted.
  2. Upload the systemfields.properties file from your local PC to /tmp on the Web Tier server using WinSCP or another SCP client. 
  3. Navigate to the Web Tier's /opt/RSASecurity/RSAAuthenticaitonManagerWebtier/utils/etc directory:






cd /opt/RSASecurity/RSAAuthenticationManagerWebtier/utils/etc




NOTE: If your Web Tier was installed to a non-default location, adjust this path accordingly.


 




  1. Rename the corrupted file as a backup:






mv systemfields.properties systemfields.properties.orig




CAUTION: Do not delete the original file. Renaming it preserves it in case RSA Support needs it for further diagnosis.


 




  1. Copy the file retrieved from the primary Authentication Manager server into the Web Tier directory:


cp /tmp/systemfields.properties ./


 




  1. Check permissions on both files to determine the correct owner and group:


ls -alh systemfields.properties*


Expected output (owner/group names will vary by environment):




-rw-------. 1 root    root    11K Jan 21 08:42 systemfields.properties
-rw-------. 1 webtier webtier   0 Jan  4 05:36 systemfields.properties.orig




NOTE: The owner and group of the .orig file reflect the user defined during Web Tier installation (e.g., webtier). Use the values from your .orig file, not the example values shown here.


 




  1. Set the correct permissions on the new file:






chmod 600 systemfields.properties


 




  1. Set the correct owner and group to match the original file (replace webtier:webtier with the owner/group from your environment):






chown webtier:webtier systemfields.properties


 




  1. Switch to the Web Tier install user (replace webtier with the user selected during your Web Tier installation):






sudo su - webtier


 




  1. Navigate to the Web Tier's /opt/RSASecurity/RSAAuthenticaitonManagerWebtier/utils directory and run the manage-secrets utility to identify the available versions:






cd /opt/RSASecurity/RSAAuthenticationManagerWebtier/utils/
./rsautil manage-sec


Expected output:




Invalid argument. Multiple CLU's starting with 'manage-sec' found:
    manage-secrets-8.1.1.10.0
    manage-secrets-8.1.1.2.0
    manage-secrets-8.1.1.9.0




NOTE This error is expected. It lists the available versions of the utility. Check the Web Tier patch level in the Operations Console and use the matching version in the next step.


 




  1. Run the manage-secrets command that matches your Web Tier patch level with the -a recover flag (example shown for Authentication Manager 8.7 SP1 patch 1): 






./rsautil manage-secrets-8.7.1.1.0 -a recover


Enter the Operations Console administrator credentials from the primary Authentication Manager server when prompted:




Please enter OC Administrator username: <OC admin username>
Please enter OC Administrator password: <OC admin password>


Expected output:




Machine fingerprint restored successfully.


 




  1. Start the Web Tier services:


cd /opt/RSASecurity/RSAAuthenticationManagerWebtier/server/
./rsaserv start


 




  1. To verify, log in to the Authentication Manager Operations Console and navigate to Deployment > Web Tier Deployments > Manage Existing. Within approximately 5 minutes, confirm the Web Tier status returns to Online. If the Web Tier status shows Online, the issue has been resolved.






















































Notes
Related Articles
Trending Articles
Don't see what you're looking for?