How to Decomission a Host from the Puppet Trust Model
3 years ago
Originally Published: 2017-06-14
Article Number
000065064
Applies To
RSA Product Set: Security Analytics
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 10.4.x, 10.5.x, 10.6.x
Platform: CentOS
O/S Version: EL6
Component Name: Puppet
Issue
How to manually remove host if the "Remove and Repurpose Host" feature in Administration \ Appliances (SA 10.4.X) or Administration \ Hosts (SA 10.5.X and later) is not an option.
 
Resolution
On the Host to be Removed (if it is still available)
1. Either SSH or use a Console (such as iDRAC console) to connect to node you are removing from the trust model

2. Obtain the puppet node_id
On that Node, obtain the node_id 
/etc/puppet/scripts/node_id.py
Example Output: 
# /etc/puppet/scripts/node_id.py
b93bdb62-b0e4-4141-a03b-dac803da73a7
Other ways of obtaining puppet node_id 
echo $(</var/lib/puppet/node_id)
facter -p node_id
3. Stop services 
service collectd stop
service mcollective stop
puppet resource service puppet ensure=stopped
service rabbitmq-server stop
4. Clean-up puppet 
rm -rf /var/lib/puppet/ssl
rm -f /var/lib/puppet/state/agent_catalog_run.lock
rm -f /var/lib/puppet/client_data/catalog/$(/etc/puppet/scripts/node_id.py).json
rm -f /var/lib/puppet/yaml/facts/$(/etc/puppet/scripts/node_id.py).yaml
rm -f /var/lib/puppet/yaml/node/$(/etc/puppet/scripts/node_id.py).yaml
rm -rf /var/lib/puppet/reports/$(/etc/puppet/scripts/node_id.py)

5. Remove trustpeers 
find /etc/netwitness/ng -name 'trustpeers' -type d -exec rm -rf {} \; 2>/dev/null

On the NetWitness Head Unit Server/SA Server which hosts the puppetmaster:
6. The following 3 commands are optional however may improve the likelihood of success 
service puppetmaster restart
service mcollective restart
service rabbitmq-server restart

7. Run delNode.py
Syntax: 
/etc/puppet/scripts/delNode.py <node_id>
Example: 
/etc/puppet/scripts/delNode.py b93bdb62-b0e4-4141-a03b-dac803da73a7


8. The previous step runs puppet cert clean <node_id>
Check that there are no current requests for this node_id:

ls /var/lib/puppet/ssl/ca/requests -l
ls /var/lib/puppet/ssl/ca/signed -l


9. Remove from puppetmaster inventory.txt

mv /var/lib/puppet/ssl/ca/inventory.txt /var/lib/puppet/ssl/ca/inventory.old
Syntax: 
grep -v <node_id> /var/lib/puppet/ssl/ca/inventory.old > /var/lib/puppet/ssl/ca/inventory.txt
Example: 
grep -v b93bdb62-b0e4-4141-a03b-dac803da73a7 /var/lib/puppet/ssl/ca/inventory.old > /var/lib/puppet/ssl/ca/inventory.txt
 

10. Remove references from mongoDB
Syntax:

node_target=<node_id>
applianceId=$(echo 'db.appliance_info.find( { "identity" : "'$node_target'" },{"_id":1} )' | mongo asg | grep -o "ObjectId(.*" | cut -d\" -f2)
echo 'db.appliance_info.remove( { "identity" : "'$node_target'" } )' | mongo asg
echo 'db.service_info.remove( { "applianceId" : "'$applianceId'" } )' | mongo asg
echo 'db.appliance_update.remove( { "identity" : "'$node_target'" } )' | mongo puppet
echo 'db.entitlement.remove( { "identity" : "'$node_target'" } )' | mongo sa
echo 'db.service_info.remove( { "hostId" : "'$node_target'" } )' | mongo les
Example: 
node_target='b93bdb62-b0e4-4141-a03b-dac803da73a7'
applianceId=$(echo 'db.appliance_info.find( { "identity" : "'$node_target'" },{"_id":1} )' | mongo asg | grep -o "ObjectId(.*" | cut -d\" -f2)
echo 'db.appliance_info.remove( { "identity" : "'$node_target'" } )' | mongo asg
echo 'db.service_info.remove( { "applianceId" : "'$applianceId'" } )' | mongo asg
echo 'db.appliance_update.remove( { "identity" : "'$node_target'" } )' | mongo puppet
echo 'db.entitlement.remove( { "identity" : "'$node_target'" } )' | mongo sa
echo 'db.service_info.remove( { "hostId" : "'$node_target'" } )' | mongo les
 

On the Host to be Removed
11. Stop services in upstart

declare -a services_to_stop=( $(find /etc/init \( -name "nw*" -o -name "rsa*" \) -type f -exec basename {} \; | sed 's/.conf$//g' | grep -v -E "(bad|disable|old)" | sort -r) )
for target in "${services_to_stop[@]}"; do $(stop "$target") 2>/dev/null; done

12. Restart/Reboot OS 
shutdown -r now
Don't see what you're looking for?