How to analyze the data log files in RSA Web Threat Detection

2 years ago

Originally Published: 2016-11-21

Article Number

000060656

Applies To

RSA Product Set: Web Threat Detection
RSA Product/Service Type: Mitigator
RSA Version/Condition: 5.x, 6.x

Issue

Customers might state that they have missing data, comparing two systems in same environments where they see data differences, missing clickstreams or simply want to view raw data in their shards.

Tasks

These need to be investigated in the actual log files located in the /var/opt/silvertail/data directory.

The directories are arranged by the following:

year
month
day
hour

Resolution

Do a webex session with the customer and ask them to go to the data in question to obtain the shard files.

Determine the shard that needs to be extracted. This will depend on the issue, however, in general the shards are arranged by the number of bits specified. The shards are distributed by IP's across the shards. When you list the directory, the name of the shard will contain a bit value.
The customer or CS Engineer will use a built-in utility to decrypt the targeted data files called logcat. This utility is not officially supported but can be used.
1. The command below will decrypt and move to a temp folder for collection.
```
logcat -f conf FILE
```
  The location of conf file is required and is usually in two places var/opt/silvertail/etc/logcat.conf or universal_conf.txt. For example:
```
/var/opt/silvertail/bin/logcat –f /var/opt/silvertail/etc/logcat.conf <logcat file>.crypt > <temp folder>
```
2. Ask the customer to send these via SFTP so FRI CS Engineer can analyze on their lab system.
  Note: Make your own .conf file that includes this information from Universal_conf of the system that will use logcat.
```
------------- contents of file -----------------
<build>
    <tls>
        <x509 cert="/var/opt/silvertail/certs/silvertail.crt"
               key="/var/opt/silvertail/certs/silvertail.key" />
    </tls>
    <logger priority="INFO" />
</build>
```
Further analysis on the shard files using getshard.
```
getshard –i <ip> -b 8
```
The 8 is seen in the naming of the shards is the number of bits in size. For example:
```
getshard -b 8 -i 192.168.1.2
```
Explore shards with imported utility loggrep. The loggrep utility can be obtained by FRI CS Engineers and can be used to look through the contents of a single shard. For example:
```
# var/opt/silvertail/bin/loggrep –I <ip address> -m POST
```

Example
Cat the shard and run loggrep to find a particular IP address and REQUEST in the contents, do a count and print the unique entries. (This will show if there are too many counts for one page, or too many unique pages because page normalization has not been performed.)

cat <shardname.txt> | loggrep –I <ip address> | awk ‘/REQUEST/ {print $3} | awk –F’&’ ‘{print $2}’ | sort | uniq –c |sort –n

There are many ways to approach the analysis of shard files but this is an introductory to using these tools.

Notes

Decrypted shards will contain raw data and may contain sensitive information. These files should be handled in a secure fashion.

Use of getshard

[root@wtd etc]# /var/opt/silvertail/bin/getshard
Silver Tail shard generator version 5.1.1.5
getshard -i <ip> -b <bits> [-H] -s <string> [-v]
    -i <ip>      IP address to show the shard (required)
    -b <bits>    Number of shard bits. Default 0.
    -H           Display the hash value of ip (optional)
    -s <string>  Show the hash of string.
    -v           Display the version

Use of logcat

logcat -f conf FILE...

Use of loggrep

[root@gcVarzArchive verizon]# cat prodshard23Test2.txt |loggrep
Usage: /varz/bin/loggrep [options] ...

Options :
        -i <IP>
        -a <User-agent>
        -u <User>
        -m <Method>
        -p <Page>
        -s <Status>   (HTTP Response Code)
        -h <Host>     (HTTP Host Header)
        -z <Other>    (Any other header - see below)
        -v            Invert matching

All fields are Regex. IP Address is anchored match, all others are partial.

The "Other" format is TTTT&ffff&vvvv where TTTT is the log entry type, ffff is the field name, and vvvv is the value. eg, to find the "test" cookie being set to "true" use:

-z SETCOOKIE&test&^true$

Examples:

Find all POST requests from IP 192.0.2.55
```
/varz/bin/loggrep -i 192.0.2.55 -m POST
```
Find all requests from Firefox with an argument of "search" with any value
```
/varz/bin/loggrep -a firefox -z 'ARGS&search&.'
```

Don't see what you're looking for?

Potential Impact from Salesforce Device Activation Change

Related Articles

Trending Articles