WARNING: Version 5.5 of the Elastic Stack has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
X-Pack security Troubleshooting
editX-Pack security Troubleshooting
editsettings
edit- Some settings are not returned via the nodes settings API
-
This is intentional. Some of the settings are considered to be highly sensitive: all
ssl
settings, ldapbind_dn
,bind_password
). For this reason, we filter these settings and do not expose them via the nodes info API rest endpoint. You can also define additional sensitive settings that should be hidden using thexpack.security.hide_settings
setting. For example, this snippet hides theurl
settings of theldap1
realm and all settings of thead1
realm.xpack.security.hide_settings: xpack.security.authc.realms.ldap1.url, xpack.security.authc.realms.ad1.*
users
edit- I configured the appropriate roles and the users, but I still get an authorization exception
-
Verify that the role names associated with the users match the roles defined in the
roles.yml
file. You can use theusers
tool to list all the users. Any unknown roles are marked with*
. - ERROR: extra arguments […] were provided
-
This error occurs when the
users
tool is parsing the input and finds unexepected arguments. This can happen when there are special characters used in some of the arguments. For example, on Windows systems the,
character is considered a parameter separator; in other words-r role1,role2
is translated to-r role1 role2
and theusers
tool only recognizesrole1
as an expected parameter. The solution here is to quote the parameter:-r "role1,role2"
.
Active Directory
edit- Certain users are being frequently locked out of Active Directory
-
Check your realm configuration; realms are checked serially, one after another. If your Active Directory realm is being checked before other realms and there are usernames that appear in both Active Directory and another realm, a valid login for one realm may be causing failed login attempts in another realm.
For example, if
UserA
exists in both Active Directory and a file realm, and the Active Directory realm is checked first and file is checked second, an attempt to authenticate asUserA
in the file realm would first attempt to authenticate against Active Directory and fail, before successfully authenticating against thefile
realm. Because authentication is verified on each request, the Active Directory realm would be checked - and fail - on each request forUserA
in thefile
realm. In this case, while the authentication request completed successfully, the account on Active Directory would have received several failed login attempts, and that account may become temporarily locked out. Plan the order of your realms accordingly.Also note that it is not typically necessary to define multiple Active Directory realms to handle domain controller failures. When using Microsoft DNS, the DNS entry for the domain should always point to an available domain controller.
LDAP
edit- I can authenticate to LDAP, but I still get an authorization exception
-
A number of configuration options can cause this error.
group identification
Groups are located by either an LDAP search or by the "memberOf" attribute on the user. Also, If subtree search is turned off, it will search only one level deep. See the LDAP Settings for all the options. There are many options here and sticking to the defaults will not work for all scenarios.
group to role mapping
Either the
role_mapping.yml
file or the location for this file could be misconfigured. See Security Files for more.role definition
The role definition may be missing or invalid.
To help track down these possibilities, add the following lines to the end of the
log4j2.properties
configuration file in theCONFIG_DIR
:logger.authc.name = org.elasticsearch.xpack.security.authc logger.authc.level = DEBUG
A successful authentication should produce debug statements that list groups and role mappings.
Encryption & Certificates
edit-
curl
on the Mac returns a certificate verification error even when the--cacert
option is used -
Apple’s integration of
curl
with their keychain technology disables the--cacert
option. See http://curl.haxx.se/mail/archive-2013-10/0036.html for more information.You can use another tool, such as
wget
, to test certificates. Alternately, you can add the certificate for the signing certificate authority MacOS system keychain, using a procedure similar to the one detailed at the Apple knowledge base. Be sure to add the signing CA’s certificate and not the server’s certificate.
SSLHandshakeException causing connections to fail
editA SSLHandshakeException
will cause a connection to a node to fail and indicates that there is a configuration issue. Some of the
common exceptions are shown below with tips on how to resolve these issues.
-
java.security.cert.CertificateException: No name matching node01.example.com found
-
Indicates that a client connection was made to
node01.example.com
but the certificate returned did not contain the namenode01.example.com
. In most cases, the issue can be resolved by ensuring the name is specified during certificate creation. Another scenario is when the environment does not wish to use DNS names in certificates at all. In this scenario, all settings inelasticsearch.yml
should only use IP addresses including thenetwork.publish_host
setting. -
java.security.cert.CertificateException: No subject alternative names present
-
Indicates that a client connection was made to an IP address but the returned certificate did not contain any
SubjectAlternativeName
entries. IP addresses are only used for hostname verification if they are specified as aSubjectAlternativeName
during certificate creation. If the intent was to use IP addresses for hostname verification, then the certificate will need to be regenerated with the appropriate IP address. -
javax.net.ssl.SSLHandshakeException: null cert chain
andjavax.net.ssl.SSLException: Received fatal alert: bad_certificate
-
The
SSLHandshakeException
above indicates that a self-signed certificate was returned by the client that is not trusted as it cannot be found in thetruststore
orkeystore
. TheSSLException
above is seen on the client side of the connection. -
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
andjavax.net.ssl.SSLException: Received fatal alert: certificate_unknown
-
The
SunCertPathBuilderException
above indicates that a certificate was returned during the handshake that is not trusted. This message is seen on the client side of the connection. TheSSLException
above is seen on the server side of the connection. The CA certificate that signed the returned certificate was not found in thekeystore
ortruststore
and needs to be added to trust this certificate.
Other SSL/TLS related exceptions
editThe are other exceptions related to SSL that may be seen in the logs. Below you will find some common exceptions and their meaning.
- WARN: received plaintext http traffic on a https channel, closing connection
-
Indicates that there was an incoming plaintext http request. This typically occurs when an external applications attempts to make an unencrypted call to the REST interface. Please ensure that all applications are using
https
when calling the REST interface with SSL enabled. -
org.elasticsearch.common.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
-
Indicates that there was incoming plaintext traffic on an SSL connection. This typically occurs when a node is not configured to use encrypted communication and tries to connect to nodes that are using encrypted communication. Please verify that all nodes are using the same setting for
xpack.security.transport.ssl.enabled
. -
java.io.StreamCorruptedException: invalid internal transport message format, got
-
Indicates an issue with data received on the transport interface in an unknown format. This can happen when a node with encrypted communication enabled connects to a node that has encrypted communication disabled. Please verify that all nodes are using the same setting for
xpack.security.transport.ssl.enabled
. -
java.lang.IllegalArgumentException: empty text
-
The exception is typically seen when a
https
request is made to a node that is not usinghttps
. Ifhttps
is desired, please ensure the following setting is inelasticsearch.yml
:xpack.security.http.ssl.enabled: true
- ERROR: unsupported ciphers […] were requested but cannot be used in this JVM
-
This error occurs when a SSL/TLS cipher suite is specified that cannot supported by the JVM that Elasticsearch is running in. Security will try to use the specified cipher suites that are supported by this JVM. This error can occur when using the Security defaults as some distributions of OpenJDK do not enable the PKCS11 provider by default. In this case, we recommend consulting your JVM documentation for details on how to enable the PKCS11 provider.
Another common source of this error is requesting cipher suites that use encrypting with a key length greater than 128 bits when running on an Oracle JDK. In this case, you will need to install the JCE Unlimited Strength Jurisdiction Policy Files.
Internal Server Error in Kibana
editIf the Security plugin is enabled in Elasticsearch but disabled in Kibana, you must
still set elasticsearch.username
and elasticsearch.password
in kibana.yml
.
Otherwise, Kibana cannot connect to Elasticsearch. In 5.1.1, this results in an
UnhandledPromiseRejectionWarning
and Kibana displays an Internal Server Error.