Kerberos

Elasticsearch for Apache Hadoop only needs a few settings to configure Kerberos authentication. It is best to set these properties in your core-site.xml configuration so that they can be obtained across your entire Hadoop deployment, just like you would for turning on security options for services in Hadoop.

<configuration>
    ...
    <property>
        <name>es.security.authentication</name>
        <value>kerberos</value>
    </property>
    <property>
        <name>es.net.spnego.auth.elasticsearch.principal</name>
        <value>HTTP/_HOST@REALM.NAME.HERE</value>
    </property>
    ...
</configuration>

Create a role for your end users that will be querying Hive. In this example, we will make a simple role for accessing indices that match hive-index-*. All our Hive users will end up using this role to read, write, and update indices in Elasticsearch.

PUT _xpack/security/role/hive_user_role 
{
  "run_as": [],
  "cluster": ["monitor", "manage_token"], 
  "indices": [
      {
        "names": [ "hive-index-*" ],  
        "privileges": [ "read", "write", "manage" ]
      }
  ]
}

Now that the user role is created, we must map the Kerberos user principals to the role. Elasticsearch does not know the complete list of principals that are managed by Kerberos. As such, each principal that wishes to connect to Elasticsearch must be mapped to a list of roles that they will be granted after authentication.

POST /_xpack/security/role_mapping/hive_user_1_mapping
{
  "roles": [ "hive_user_role" ], 
  "enabled": true,
  "rules": {
    "field" : { "username" : "hive.user.1@REALM" } 
  }
}

Mirroring the user to the native realm will allow Elasticsearch to accept authentication requests from the original principal as well as accept requests from Hive which is impersonating the user. You can create a user in the native realm like so:

PUT /_xpack/security/user/hive.user.1 
{
  "enabled" : true,
  "password" : "swordfish", 
  "roles" : [ "hive_user_role" ], 
  "metadata" : {
    "principal" : "hive.user.1@REALM" 
  }
}

Once you have configured Elasticsearch with a role mapping for your Kerberos principals and native users for impersonation, you must create a role that Hive will use to impersonate those users.

PUT _xpack/security/role/hive_proxier
{
  "run_as": ["hive.user.1"] 
}

Now that there are users to impersonate, and a role that can impersonate them, make sure to map the Hive principal to the proxier role, as well as any of the roles that the users it is impersonating would have. This allows the Hive principal to create and read indices, documents, or do anything else its impersonated users might be able to do. While Hive is impersonating the user, it must have these roles or else it will not be able to fully impersonate that user.

POST /_xpack/security/role_mapping/hive_hiveserver2_mapping
{
  "roles": [
    "hive_user_role", 
    "hive_proxier" 
  ],
  "enabled": true,
  "rules": {
    "field" : { "username" : "hive/hiveserver2.address@REALM" } 
  }
}

All implementations of the Spark credential providers use settings from only a few places:

Settings that are configured from the user code are not used because the provider must run once for all jobs that are submitted for a particular Spark application. User code is not guaranteed to be run before the provider is loaded. To make things more complicated, a credential provider is only given the local Hadoop configuration to determine if they should load delegation tokens.

These limitations mean that the settings to configure elasticsearch-hadoop for Kerberos authentication need to be in specific places:

First, es.security.authentication MUST be set in the local Hadoop configuration files as kerberos. If it is not set in the Hadoop configurations, then the credential provider will assume that simple authentication is to be used, and will not obtain delegation tokens.

Secondly, all general connection settings for elasticsearch-hadoop (like es.nodes, es.ssl.enabled, etc…​) must be specified either in the local Hadoop configuration files, in the local Spark configuration file, or from the command line. If these settings are not available here, then the credential provider will not be able to contact Elasticsearch in order to obtain the delegation tokens that it requires.

$> bin/spark-submit \
    --class org.myproject.MyClass \
    --master yarn \
    --deploy-mode cluster \
    --jars path/to/elasticsearch-hadoop.jar \
    --conf 'spark.es.nodes=es-node-1,es-node-2,es-node-3' 
    --conf 'spark.es.ssl.enabled=true'
    --conf 'spark.es.net.spnego.auth.elasticsearch.principal=HTTP/_HOST@REALM' 
    path/to/jar.jar

Since streaming jobs are expected to run continuously without stopping, you should configure Spark so that the credential provider can obtain new tokens before the original tokens expire.

Configuring Spark to obtain new tokens is different from configuring YARN to renew and cancel tokens. YARN can only renew existing tokens up to their maximum lifetime. Tokens from Elasticsearch are not renewable. Instead, they have a simple lifetime of 7 days. After those 7 days elapse, the tokens are expired. In order for an ongoing streaming job to continue running without interruption, completely new tokens must be obtained and sent to worker tasks. Spark has facilities for automatically obtaining and distributing completely new tokens once the original token lifetime has ended.

When submitting a Spark application on YARN, users can provide a principal and keytab file to the spark-submit command. Spark will log in with these credentials instead of depending on the local Kerberos TGT Cache for the current user. In the event that any delegation tokens are close to expiring, the loaded credential providers are given the chance to obtain new tokens using the given principal and keytab before the current tokens fully expire. Any new tokens are automatically distributed by Spark to the containers on the YARN cluster.

$> bin/spark-submit \
    --class org.myproject.MyClass \
    --master yarn \ 
    --deploy-mode cluster \ 
    --jars path/to/elasticsearch-hadoop.jar \
    --principal client@REALM 
    --keytab path/to/keytab.kt \ 
    path/to/jar.jar

When elasticsearch-hadoop is on the classpath, EsServiceCredentialProvider is ALWAYS loaded by Spark. If Kerberos authentication is enabled for elasticsearch-hadoop in the local Hadoop configuration, then the provider will attempt to load delegation tokens for Elasticsearch regardless of if they are needed for that particular job.

It is advised that you do not add elasticsearch-hadoop libraries to jobs that are not configured to connect to or interact with Elasticsearch. This is the easiest way to avoid the confusion of unrelated jobs failing to launch because they cannot connect to Elasticsearch.

If you find yourself in a place where you cannot easily remove elasticsearch-hadoop from the classpath of jobs that do not need to interact with Elasticsearch, then you can explicitly disable the credential provider by setting a property at launch time. The property to set is dependent on your version of Spark:

In order for the AutoElasticsearch plugin to obtain credentials, Kerberos authentication must be enabled for elasticsearch-hadoop in its settings. You must specify the es.security.authentication setting in either the storm.yaml file or on the topology configuration.

The AutoElasticsearch plugin provides two settings for denoting the principal and keytab to be used when executing:

Nimbus must be configured to use AutoElasticsearch as a credential plugin from the storm.yaml configuration file. It is safe to specify AutoElasticsearch in these settings even if your topology does not interact with Elasticsearch. The plugin will perform no operations unless AutoElasticsearch is explicitly enabled on the topology.

nimbus.autocredential.plugins.classes: ["org.elasticsearch.storm.security.AutoElasticsearch"] 
nimbus.credential.renewers.classes: ["org.elasticsearch.storm.security.AutoElasticsearch"] 
nimbus.credential.renewers.freq.secs: 30 

In order for the plugin to be loaded, elasticsearch-hadoop must be present on the Nimbus classpath. You can add it to the classpath by using an environment variable on Nimbus.

export STORM_EXT_CLASSPATH=/path/to/elasticsearch-hadoop.jar

Once Nimbus is configured, you must add AutoElasticsearch to your topology configuration in order for delegation tokens to be obtained and updated. If you do not specify it in the topology configuration, then Storm will not attempt to obtain Elasticsearch delegation tokens when the topology is submitted.

Config conf = new Config();
List plugins = new ArrayList();
plugins.add(AutoElasticsearch.class.getName());
conf.put(Config.TOPOLOGY_AUTO_CREDENTIALS, plugins); 
...
conf.put(ConfigurationOptions.ES_SECURITY_AUTHENTICATION, "kerberos"); 
conf.put(ConfigurationOptions.ES_NET_SPNEGO_AUTH_ELASTICSEARCH_PRINCIPAL, "HTTP/elasticsearch.node.address@REALM");
...