RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition:  8.8 

Applying am-update-8.8.0.0.0 fails with "Failed to apply update AM 8.8" and "Failed to prepare update". Prerequisites: Couldn't upgrade because of failed validation of the MFA API credential ERROR: exec returned: 3. prerequisites_ims_config_mfa_api_keys.sql.log ERROR: auth_manager.rest_service.old_access_key is not found.

Found in the update-8.8.0.0.0-build1445095-xxxxxxxxxxxxxx log:

26990  2025-04-24 20:07:24,267 INFO: Executing prerequisites_ims_config_mfa_api_keys.sql.  Log output will be found here: /opt/rsa/am/install_logs/dbscripts/prerequisites_ims_config_mfa_api_keys.sql-20250424200724.log
26991  2025-04-24 20:07:24,268 INFO: Executing /opt/rsa/am/pgsql/bin/psql
27435  2025-04-24 20:07:24,712 INFO: === Prerequisites: Couldn't upgrade because of failed validation of the MFA API credential. For more details, refer to the log output.===
27436  2025-04-24 20:07:24,713 ERROR: : exec returned: 3
Exception in thread "main" : exec returned: 3
    at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:646)
    at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
    at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)
    at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
    at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
    at Utils.exec(Utils.groovy:325)
    at Utils$exec$8.call(Unknown Source)
    at DBRefresh.runPsqlScript(DBRefresh.groovy:314)
    at UpdateRollback.preUpdate_InvokeGroovy_Run_prerequisites_ims_config_mfa_api_keys_sql(UpdateRollback.groovy:228)
    at UpdateRollback$_preUpdate_closure1.doCall(UpdateRollback.groovy:23)
    at UpdateRollback$_preUpdate_closure1.doCall(UpdateRollback.groovy)
    at Utils.withArtifacts(Utils.groovy:48)
    at Utils$withArtifacts$5.call(Unknown Source)
    at UpdateRollback.preUpdate(UpdateRollback.groovy:12)
    at com.rsa.plugins.install.GroovyInstallEngine.invokeScript(GroovyInstallEngine.groovy:68)
    at com.rsa.plugins.install.GroovyInstallEngine$_runTask_closure2.doCall(GroovyInstallEngine.groovy:57)
    at com.rsa.plugins.install.GroovyInstallEngine.runTask(GroovyInstallEngine.groovy:56)
    at com.rsa.plugins.install.GroovyInstallEngine$_runTasks_closure3.doCall(GroovyInstallEngine.groovy:106)
    at com.rsa.plugins.install.GroovyInstallEngine.runTasks(GroovyInstallEngine.groovy:105)
    at com.rsa.plugins.install.GroovyInstallEngine$runTasks.call(Unknown Source)
    at com.rsa.plugins.install.CommandLineInstallEngine.main(CommandLineInstallEngine.groovy:40)
Configuration step UpdateRollback:preUpdate [FAILED]
[ERROR] Error: Failed to invoke update engine: Failed to prepare the update.
java.lang.Exception: Failed to prepare the update.

Found in prerequisites_ims_config_mfa_api_keys.sql-xxxxxxxxxxxxxx under /opt/rsa/am/install_logs/dbscripts/ 

CREATE FUNCTION
psql:prerequisites_ims_config_mfa_api_keys.sql:123: INFO:  instance_type is PRIMARY
psql:prerequisites_ims_config_mfa_api_keys.sql:123: INFO:  Number of replica servers in deployment: 1
psql:prerequisites_ims_config_mfa_api_keys.sql:123: INFO:  The type of this AM server is: PRIMARY
psql:prerequisites_ims_config_mfa_api_keys.sql:123: INFO:  Performing ims-config pre-requisites for MFA API credentials.
psql:prerequisites_ims_config_mfa_api_keys.sql:123: INFO:  auth_manager.rest_service.old_access_key is not found
psql:prerequisites_ims_config_mfa_api_keys.sql:123: ERROR:  auth_manager.rest_service.old_access_key is not found
CONTEXT:  PL/pgSQL function inline_code_block line 65 at RAISE

The Authentication Manager 8.8 upgrade conducts a pre-check and validation of the MFA API credentials as a necessary step before implementing the system update. This does not indicate a failure of the upgrade; instead, it signifies that the pre-checks were unsuccessful, preventing the upgrade from advancing.

The MFA API credential validation check fails when the access old ID and access old Key are set to null.

rsaadmin@am8.8:~> /opt/rsa/am/utils/rsautil manage-rest-access-credential -a list -u <admin> -p <password>
access ID: ggjo7h717yquk5m5fgnk10xf7s7ki8b6947ph6f9a8pab7s38wyx57u9820s47q0
access Key: 6u6izx437d19p8w4ot6901jn15eyq3x6f8l6o3fi77157k4u77oxv0jdwoje62ox
access old ID: null
access old Key: null
access retain days[default is 60]: 60
Successfully listed the RSA SecurID REST Access credential.

TO resolve the issue,

1. First perform a backup of the server from the Operations Console. If using a virtual platform, also create a snapshot of the primary instance.
2. Ensure SSH access is enabled on the primary.
3. Log on to the primary Authentication Manager server with SSH.
4. Navigate to /opt/rsa/am/utils/rsautil.
5. Run the command ./rsautil manage-rest-access-credential -a generate at the prompt.
6. Enter your super admin user name and press Enter.
7. Enter the password for the super admin user and press Enter.

rsaadmin@am8.8:~> cd /opt/rsa/am/utils/
  rsaadmin@am8.8:~> ./rsautil manage-rest-access-credential -a generate
  admin user name: <enter super admin user name>
  admin password: <enter super admin user password>
  Successfully generated the RSA SecurID REST Access credential.

access ID: 52c4jpd1sb56kv81s1c07a0j5j4s20x080fm9n08hs2hq301em9wst380857etcc
access Key: s71nh8l8d033iuxz4c4mux2g52vg5219ax3438eea0871kl20m4x130p35kr89mj

7. Wait a few minutes, and run /opt/rsa/am/utils/rsautil manage-rest-access-credential -a list -u <super admin user name> -p <super admin password> to ensure there are no null values in the API credentials.

   rsaadmin@am8.8:~> /opt/rsa/am/utils/rsautil manage-rest-access-credential -a list -u <admin> -p <password>
   
   access ID: 52c4jpd1sb56kv81s1c07a0j5j4s20x080fm9n08hs2hq301em9wst380857etcc
   access Key: s71nh8l8d033iuxz4c4mux2g52vg5219ax3438eea0871kl20m4x130p35kr89mj
   access old ID: ggjo7h717yquk5m5fgnk10xf7s7ki8b6947ph6f9a8pab7s38wyx57u9820s47q0
   access old Key: 6u6izx437d19p8w4ot6901jn15eyq3x6f8l6o3fi77157k4u77oxv0jdwoje62ox
   access retain days[default is 60]: 60
   Successfully listed the RSA SecurID REST Access credential.
   rsaadmin@am8.8:~>

8. To apply the changes to the replica instances, on each replica instance,
   1. Log on to the Security Console.
   2. Go to Setup > System Settings.
   3. Under  Authentication Settings, click RSA SecurID Authentication API then click Apply Settings.
9. Retry the upgrade to Authentication Manager 8.8 upgrade, and the server will now upgrade successfully.

Both the current and old MFA API keys will continue to be valid following the upgrade to Authentication Manager 8.8. 

• Per the Release Notes for Authentication Manager 8.8, the upgrade includes a mandatory pre-check to validate MFA API credentials. If this validation fails, the upgrade is blocked. This does not indicate an upgrade failure, but rather that a required pre-check was unsuccessful. The validation fails if the access old ID and access old key values are set to null. Note that  the default for the access retain days is 60. After 60 days they expire.
• Some customers report that the solution above does not work in their environments when the absolute path is used. The option to first navigate to /opt/rsa/am/utils and then run the command as ./rsautil manage-rest-access-credential -a generate -u <admin> -p <password> works for those customers.
• Both the  current and the old MFA API keys will continue to be valid following the upgrade to Authentication Manager 8.8 until it is deprecated or deleted by the Administrator.
• The CLU for managing the REST access credential manage-rest-access-credential is no longer available following the upgrade to Authentication Manager 8.8 because 8.8 accommodates the ability to have multiple Access IDs and Access Keys.
• Prior to Authentication Manager 8.8, Authentication Manager allowed for the extension of access retain days; however, in Authentication Manager 8.8, the introduction of multiple REST API keys eliminates the necessity for retain days. See RSA Announces the Release of RSA Authentication Manager 8.8 then scroll to the section on Multiple MFA REST API Key Support where it states that "[s]upport for multiple MFA REST API keys allows for secure, flexible communication between Authentication Manager and its agents—ideal for high-security environments or multi-agent deployments.