Unable to attach Authenticaiton Manager 8.1 SP1 replica

[#1]

Article Number

000029488

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.1.0 SP1
Platform:  VMware ESXi 5
 

Issue

Attach replica process fails at "Starting services". 
[Begin Attach] choose replica_package.zip (used new ocadmin and password)
Processing request. Please wait. . . start of task 11:30:33am CST 
Transferring data from primary instance Completed 
Configuring replica instance Completed 
Starting services in progress 7mins...(X) Failed 
Sending.... 
Failed to attach replica instance. 
You must redeploy this replica instance by performing the following steps: 
Image description
Install log FATAL: Did not receive the first change from the primary within the expected time period
select latest_sweep_id from rsa_rep_util.AM_P2R_ReplicaStatus where action_cd = 'propagated'; 
select latest_sweep_id from rsa_rep_util.AM_P2R_PrimaryStatus where action_cd = 'written'; 
but 0 rows for each 
select * from rsa_rep_util.AM_PRIMARY_CAPTURE; 0 rows
No Sweeps files in the /opt/rsa/am/replication directory, just latest_chgsets

Cause

Installing P5 or P6 on AM 8.1 before upgrade to AM 8.1 SP1 leaves behind a file, a 32-bit /lib/libreadline.so.5.2 as well as the symbolic link /lib/libreadline.so.5, which prevents the postgres utility {{psql}} (and others) from starting cleanly.
Symptom is 
ERROR: ld.so: object '/lib/libreadline.so.5' from LD_PRELOAD cannot be preloaded: ignored.}}
in the /opt/rsa/am/replication/addreplicaxxxxxxxxxxx<date>.log 

Resolution

Use work-around until fix announced in probably AM 8.1 SP1 P1 (scheduled) or later
Image description

Workaround

Short version:
Delete  /lib/libreadline.so.5, reboot the primary and try clean replica attachment with new replica deployment

Engineering Note: If any one is attempting to do this manually

• Only delete the 32-bit libreadline.so.5 from the /lib directory
• DO NOT delete the 64-bit libreadline.so.* files from under /lib64 (deleting the wrong files may cause the AM system to fail to start and the system console to be inaccessible).

Longer answer:
The fix is a simple cleanup script which removes the 32-bit /lib/libreadline.so.5.2 as well as the symbolic link /lib/libreadline.so.5
The change is introduced near the beginning of the patch manifest additions for bug fixes. When the patch P1 is installed the log will show:
Cleanup /lib/libreadline installation
Removing: /lib/libreadline.so.5
Removing: /lib/libreadline.so.5.2
or, if the files are already removed:
Cleanup /lib/libreadline installation
No need to clean: /lib/libreadline.so.5
No need to clean: /lib/libreadline.so.5.2
________________________________________
The fix for this is not backed-out with un-install or rollback.
The fix can be applied multiple times - has no effect if the files are already removed.

Notes

Cannot get replica to attach. AM81 primary is at SP1. VMware ESX host version 5. OVA installs correctly; primary and replica resolve forward and reverse DNS.

If you attempt to run ./psql -h localhost -p 7050 -d db -U rsa_dba and see the the symptom
ERROR: ld.so: object '/lib/libreadline.so.5' from LD_PRELOAD cannot be preloaded: ignored
You have this problem, even if you get to the PGSQL db=# prompt