User Search using API, searching beyond the 20000 results limit
2 years ago
Originally Published: 2015-03-16
Article Number
000067501
Applies To
RSA Product Set: Silver Tail
RSA Product/Service Type: Forensics - Silver Tail
RSA Version/Condition: 4.6
Platform: Other
Platform (Other): Red Hat Enterprise Linux Server release 6.5 (Santiago)
O/S Version: Other
Product Name: null
Product Description: null
Issue
WTD UI has limitation of maximum 20,000 records and the only option to change the number returned is to check the box to limit to 100 results.

Using direct API calls it's possible to search and obtain results without limitations and with added flexibility
Resolution
First authenticate and store the cookie: 
# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/login?user=admin&passwd=silvertail&app=ui";

Next Create the search ( this will return a jobid):
# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/search?find_parts=yes&find=user,page,User%5c%20IP%5c%20Changed,ip.country,ip,host&indexes=page='/accounttransfer.php'+and+host='www.prymida.com'&annotate=yes&hour=2015.3.16.1&len=672&limit=5000000&queryid=admin18446744072688845000&searchname=test&version=2&async=true"

You can view the job status in the UI, or with API, in the API you will need the jobid as return from previous command:
# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/search?cmd=status&search-job-id=20339ae09bca2374a292bdf7a3bb6925&noIdleReset=true" 

Once the job is complete, obtain the results and store in "output.txt" with:
# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/search?search-job-id=20339ae09bca2374a292bdf7a3bb6925&cmd=page&firstrecord=0&lastrecord=199999" -o output.txt

As you will notice you can choose which records you want, eg the above retrieves the first 2 million records.

Delete the finished job via the UI or using the API with:
# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/search?cmd=cancel&search-job-id=20339ae09bca2374a292bdf7a3bb6925&noIdleReset=true"

Anything available in the search UI can be included and as shown in this example I set the result limit to 5,000,000 records 

Note that the limitations here are that such large searches will have a large impact of memory and CPU while the run, You may want to run such jobs when there is low activity and just after the hourly batch.
Don't see what you're looking for?