RSA Web Threat Detection: How to troubleshoot SSL Cache and 'Resume Cache Misses' in Varz
2 years ago
Originally Published: 2018-01-31
Article Number
000063388
Applies To
RSA Product Set: Web Threat Detection
RSA Product/Service Type: Mitigator
RSA Version/Condition: 6.x
 
Issue
We have seen that the sslcache on the Silvertap servers appears to rebuild very slowly whenever there is an extended outage in the dataflow.  For instance, when we have an AWS deployment and spin up new web servers, sometimes the iptables rules used to forward packets to the taps fail to update. Once we find and correct the problem, it takes many hours, or even days, for the cache to rebuild.  The symptoms we see are a high number of "resume cache misses" in varz, as well as dropped bits. This all recovers on its own, so I assume that is time spent rebuilding the sslcache. Is there some misconfigured setting which makes it grow slowly? We have had this outage many times, and it is urgent we find a way to fix the problem more quickly.
Tasks
We need to know how the organization's webserver SSL Cache is configure so must ask the following questions of the web server administrators -- 
  • Are the organization's webservers configured for session reuse?
  • What is the webserver cache size set to in Mb?
  • How many SSL sessions does the Customer get in 4 hours?
  • What is the size of the cache in their Filesystem?  ie. /var/opt/silvertail/data/sslcache
Resolution
Below are settings in WTD that pertain to the SSL Cache

<silvertap
        vmLimitMb="6000"
        rssLimitMb="3000"
        pcapBufMb="1024"
        httpServerPort="&lt;service.serviceClass.httpServerPort&gt;"
        sslSessionIdTimeoutSeconds="14400"
        sslSessionCacheSize="200000"
        sslSessionPersistPeriodSecs="30"
        sslSessionPersistPath="/var/opt/silvertail/data/sslcache"
        acceptXffFromPublicProxy="true"
        clientIpHeaderName="true-client-ip"


Background

If we restart Silvertap,(or it goes offline somehow) how does that affect this SSL Cache?
 
What we see in the Varz log is not specifically about the SSLcache, we only see resume cache misses errors.  An understanding of the differences and relationship between the two caches involved is needed here:
-- Webserver SSL Cache
-- SilverTap SSL Cache

The SSL information is exchanged between the browser and the Webapp and a session key is created.  That session key is stored in the SSL Cache on the Webserver to reuse later (if this is configured*).  The purpose of this is so that a key exchange is not required again in the same session as long as the cache does not get full.  This is also subject to a timeout.

 (*We  would need to know if the organization's webserver is configured for session reuse)

The Silvertap is passive, it is just listening and stores a copy of the webserver's private key (configured with Certificate Manager) in its own (SilverTap) SSL Cache. 

What happens if the SilverTap cache has a smaller cache than the webserver?  
The session information would be discarded faster.  Some returning users would have to reauthenticate more frequently than they would have if the SilverTap cache were larger than the Webserver SSL Cache. 

What happens if the SilverTap service is taken down purposely or when it crashes?
When the service is stopped or unavailable the session cache still holds data.
That is, the tap but in persistent memory which is a separate configuration, and smaller than the normal cache memory.

When the tap comes back online, it retrieves the session information from the persistent memory while it discards any that are over the timeout limits and  while any new SSL keys that were exchanged while it was offline are not in the SSL Cache(local memory) or the Persistent memory

So a config of sslSessionPersistPeriodSecs="30" means  a copy from the SSL Cache Memory is put in the persistent memory every 30 seconds

So as long as the tap is able to hold a list bigger than the web server and it never stops listening to the exchanges of new SSL session keys, it will never have a resume error.
Don't see what you're looking for?