01388463
2 years ago
Originally Published: 2019-06-04
Article Number
000043949
Applies To
RSA Product Set: Access Manager
RSA Product/Service Type: Access Manager
RSA Version/Condition: 6.2
Issue
Access Manager connectivity fails when establishing an SSL/TLS connection between client and server.  Event messages usually contain the words "handshake failure".  For example, a WebAgent client may log an event message containing the following:
 
description='Fatal Alert received: Handshake Failure.',details='javax.net.ssl.SSLException: Fatal Alert received: Handshake Failure.'

That event is reporting that connection establishment was aborted by the server during the SSL/TLS handshake protocol, and the server sent the reason code "handshake_failure(40)" to the client.
 
This problem will occur when:

  • Some or all connections are configured to use anonymous SSL/TLS (anon), and
  • The Access Manager server and/or client machine has been upgraded to Java 8 update 201 or later
The issue may occur on connections between one or more of the following:
  • Access Manager Servers and Access Manager Agents
  • Access Manager Servers and Web Applications
  • The Administration Console or Administrative API clients, and the Entitlements Server
  • Access Manager Runtime and Administrative Web Services and your application server (Apache Tomcat or WebLogic)
  • Web browsers and the Administration Console
Cause
This KB article focuses specifically on handshake failure caused by the use of anonymous SSL/TLS with Java 8 update 201 and later.   Anonymous SSL/TLS is disabled by default in Java 8 update 201 and later, but was enabled by default in earlier Java versions.  Oracle Java 8 Update 201 (8u201) includes the following change, according to https://java.com/en/download/faq/release_changes.xml: 
 
* Change: TLS anon and NULL Cipher Suites are Disabled 
The TLS anon (anonymous) and NULL cipher suites have been added to the jdk.tls.disabledAlgorithms security property and are now disabled by default. 
Resolution
There are three options available to fix this issue.  These are (listed in order from most to least secure):
  1. Reconfigure affected Access Manager servers and clients to use authenticated SSL/TLS (auth).
  2. Modify Java configuration to allow anonymous SSL/TLS (anon).
  3. Reconfigure affected Access Manager servers and clients to use unencrypted communications (clear).
An explanation of the security implications of each of these options is in RFC 5246, section F.1.1.  Authentication and Key Exchange (or equivalent section in the RFCs of other SSL/TLS versions).
RSA Recommendation: For stronger security, use auth.

Options 1 and 3:  Setting Access Manager Clients and Servers to use either authenticated SSL/TLS (auth) or unencrypted communications (clear)

Reference:  RSA Access Manager Server 6.2.4 Security Configuration Guide (or equivalent Security Configuration Guide for other RSA Access Manager versions).

Access Manager Server Configuration

The following configuration files determine whether connections to and from and between Access Manager servers use auth, anon or clear.   To resolve the handshake failure issue with Java 8 update 201 or later, set the parameters listed below to either auth or clear, according to the instructions in the comments in the files (abbreviated comments copied below).  The choice of auth or clear will depend on security policy in your organization, assessment of risk and any operational or business considerations associated with using PKI digital certificates for auth connections.  If auth is chosen, then additional parameters will need to be configured in the files below and associated clients, with PKI digital certificate and private key details.
After making any configuration file changes, the RSA Access Manager servers must be restarted for the change to take effect.

admingui.cfg

# Specifies the SSL mode to use when connecting to the Entitlements Server.
#
# Dependencies:
#   This setting must match the cleartrust.eserver.api_port.use_ssl setting 
#   in the Entitlements Server configuration file.
#
cleartrust.admingui.admin_api.use_ssl=clear

eserver.conf

# Specifies the level of security used for network connections between the
# Entitlements Server and other components.
#
# Dependencies:
#   This value must match the cleartrust.net.ssl.use parameters in the
#   aserver.conf, dispatcher.conf, and admingui.cfg files, and the
#   cleartrust.agent.ssl.use parameter in the webagent.conf file.
#
# Note:
#   All of the other cleartrust.net.ssl parameters are ignored
#   if this parameter is set to clear.
#
cleartrust.net.ssl.use=clear

# Specifies if communications between the Administrative Console (or
# Administrative API clients) and the Entitlements Server should take place
# over anonymous SSL (anon) or clear text (clear). This allows you to
# disable SSL for the Administrative API port on the Entitlements Server
# when the rest of the system is using SSL. This setting applies to both
# C and Java Administrative API clients. The cleartrust.net.ssl.use parameter
# controls the SSL settings between the Entitlements Server and the other
# RSA Access Manager Servers.
#
cleartrust.eserver.api_port.use_ssl=clear

aserver.conf

# Specifies the level of security used for network connections between the
# Authorization Server and other components.
#
# Dependencies:
#   This value must match the cleartrust.net.ssl.use parameters in the
#   eserver.conf and dispatcher.conf files, and the cleartrust.agent.ssl.use
#   parameter in the webagent.conf file. If you are using version 4.7.x Agents
#   for web servers and you set cleartrust.net.ssl.use to true, you must also
#   set cleartrust.dispatcher.list_port.backward_compatible to true in your
#   Dispatcher configuration file.
#
# Note:
#   All of the other cleartrust.net.ssl parameters are ignored
#   if this parameter is set to clear.
#
cleartrust.net.ssl.use=clear

# Minimum network connection security level allowing connected clients to
# request certain operations of this Authorization Server, including SSO token 
# creation and user property retrieval. Normally this will be set to the same 
# value as cleartrust.net.ssl.use, however in environments where these 
# operations are not required, disabling them can improve security.
#
# Note: this parameter needs to be set to a value equal to or lower than that
# of the cleartrust.net.ssl.use parameter in order for clients
# (including RSA Access Manager Agents) to create tokens.
#
# Dependencies:
#   The value of this parameter must be the same for all RSA Access Manager 
#   Authorization Servers deployed in your environment.
#
cleartrust.runtime_api.security=clear

dispatcher.conf

# Specifies the level of security used for network connections between the
# Dispatcher and other components.
#
# Dependencies:
#   This value must match the cleartrust.net.ssl.use parameters in the
#   aserver.conf and eserver.conf files, and the cleartrust.agent.ssl.use
#   parameter in the webagent.conf file.
#
# Note:
#   All of the other cleartrust.net.ssl parameters are ignored
#   if this parameter is set to clear.
#
cleartrust.net.ssl.use=clear

keyserver.conf

# Specifies the level of security used for network connections between the
# Key Server and other components.
#
# Dependencies:
#   This value must match the cleartrust.net.ssl.use parameter in all
#   of the keyserver.conf files in your installation, but it does not
#   have to match the cleartrust.net.ssl.use parameter in the
#   configuration files of other RSA Access Manager Servers, such as
#   Authorization Servers or Entitlements Servers.
#
# Note:
#   All of the other cleartrust.net.ssl parameters are ignored
#   if this parameter is set to clear.
#
cleartrust.net.ssl.use=clear

lserver.conf

# Specifies the level of security used for network connections between the
# Log Server and other components.
#
# Dependencies:
#   This value must match the cleartrust.net.ssl.use parameters in the
#   eserver.conf, aserver.conf and dispatcher.conf files.
#
# Note:
#   All of the other cleartrust.net.ssl parameters are ignored
#   if this parameter is set to clear.
#
cleartrust.net.ssl.use=clear

iserver.conf

# Specifies the level of security used for network connections between the
# Instrumentation Server and other components.
#
# Dependencies:
#   This value must match that set for cleartrust.net.ssl.use in the
#   other Server configuration files.
#
# Note:
#   All of the other cleartrust.net.ssl parameters are ignored
#   if this parameter is set to clear.
#
cleartrust.net.ssl.use=clear

Web Agent, Application Server and other components

For instructions to enable either SSL/TLS auth or clear:
  • Web Agents:  refer to the RSA Access Manager Agent 5.0 Installation and Configuration Guide
  • Application servers:  r​​​​efer to vendor documentation
  • Web Services clients and Runtime API and Administrative API clients: refer to the Developer's Guide within the sdk\<package name>\Docs\dev_guide folder within your RSA Access Manager downloaded software package.

Option 2: Modify Java configuration to allow anonymous SSL/TLS (anon)

You can alternatively modify the Java configuration to re-enable anon cipher suites in Java 8 update 201 and later.  To do so, follow these steps:
  1. Optionally, take a backup of the java.security file.
  2. Open the java.security file in a text editor.
  3. Delete anon from the list of algorithms in the jdk.tls.disabledAlgorithms security property.
  4. Save your file changes.
  5. Stop and restart all RSA Access Manager software running on the server.
The java.security file is normally located in <jre_home>\lib\security in Java 8.
This change may need to be re-applied after installing any subsequent Java updates.
Notes
The reason code "handshake_failure(40)" can also have other causes, and it may occur with any version of Java. The TLS v1.2 standard,  RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2, states that handshake_failure is used in the following circumstances:
  • If the server is unable to negotiate a supported set of parameters (i.e., there is no overlap between the client and server parameters, such as cipher suites), it MUST abort the handshake with either a "handshake_failure" or "insufficient_security" fatal alert.
  • If the client does not send any certificates, the server MAY at its discretion either continue the handshake without client authentication, or respond with a fatal handshake_failure alert.
  • It is a fatal handshake_failure alert for an anonymous server to request client authentication.