Security

elasticsearch-hadoop can work in secure environments and has support for authentication and authorization. However it is important to understand that elasticsearch-hadoop per-se is a connector, that is, it bridges two different systems. So when talking about security, it is important to understand to what system it applies: the connector can run within a secure Hadoop environment talking to a vanilla/non-secured Elasticsearch cluster. Or vice-versa, it can run within a non-secured Spark environment while talking securely to a Elasticsearch cluster. Of course, the opposite can happen as well; the connector running within a secure Hadoop environment and communicating with a secured Elasticsearch cluster or the most common use-case, running from an open Spark environment to a default, non-secured Elasticsearch install. This enumeration of setups is actually on purpose, to illustrate that based on what piece of the environment is secured, its respective connector configuration needs to be adjusted.

As the connector runs as a library within Hadoop or Spark, for the most part it does not require any special configuration as it will inherit and run using the enclosing job/task credentials. In other words, as long as your Hadoop/Spark job is properly configured to run against the secure environment, elasticsearch-hadoop as library simply runs within that secured context using the already configured credentials. Settings this up is beyond the purpose of this documentation however it typically boils down to setting up the proper credentials on the configuration object used by the job/task.

Elasticsearch itself can be secured which impacts clients (like elasticsearch-hadoop )on two fronts: transport layer which is now encrypted and access layer which requires authentication. Note that typically it is recommended to enable both options (secure transport and secure access).

In case of an encrypted transport, the SSL/TLS support needs to be enabled in elasticsearch-hadoop in order for the connector to properly communicate with Elasticsearch. This is done by setting es.net.ssl property to true and, depending on your SSL configuration (whether the certificates are signed by a CA or not, whether they are global at JVM level or just local to one application), might require setting up the keystore and/or truststore, that is where the credentials are stored (keystore - which typically stores private keys and certificates) and how to verify them (truststore - which typically stores certificates from third party also known as CA - certificate authorities). Typically (and again, do note that your environment might differ significantly), if the SSL setup for elasticsearch-hadoop is not already done at the JVM level, one needs to setup the keystore if the elasticsearch-hadoop security requires client authentication (PKI - Public Key Infrastructure), and setup truststore if SSL is enabled.

The authentication support in elasticsearch-hadoop is of two types: