Getting Started with Shield Document Level Security in Elasticsearch

Elastic Shield is capable of filtering documents using query criteria. In this blog post I'll demonstrate how to use this feature by using a simple, two document data set in Elasticsearch where documents will be filtered for the user according to the value of a field and displayed in a simple Kibana visualization.  Users and roles will be created with the Users and Roles API.

The Scenario

There are three users; a sales manager and two account representatives. Documents are indexed using a code for one of two regions, EAST and WEST.  Each account representative needs read access to documents in their region and the manager will need access to both regions.  All users will be using the same Kibana visualizations to view the counts of their documents. In Kibana, users will be allowed to create/modify visualizations and dashboards to view their documents.

The following is the setup used to demonstrate DLS:

• Elasticsearch 2.3.X+ 
• Shield 2.3.X+
• Kibana 4.5.X+ 
• Sense 2.X+ recommended

*commands can also be issued using cURL.
*For this example; Shield is required on Elasticsearch but not on Kibana.

For the the initial steps, you will need to log in using an account with sufficient privileges to create  indices, roles and users.  The following REST API examples can be used directly in Kibana through the Sense plugin.

Elasticsearch and Shield

STEP 1: Configure Authentication Realm

In this example, I'll be using the Native realm to create users so the native realm needs to be configured in Elasticsearch for Shield.

elasticsearch.yml Shield configuration example:

#-----SHIELD CONFIG-------
shield:
  authc:
    realms:
      native:
        type: native
        order: 0

STEP 2: Index Sample Data

First, we will index two simple documents into one index.

This creates a document for the EAST region:

PUT myindex/mytype/1
{
  "name" : "ABC Company",
  "region" : "EAST"
}

This creates a document for the WEST region:

PUT myindex/mytype/2
{
  "name" : "DEF Company",
  "region" : "WEST"
}

You can verify the index data using:

GET myindex/mytype/_search

STEP 3: Create Roles

Next, create the roles that will be used to allow users of each region to access their documents. We are specifying two indices: .kibana and myindex.  The .kibana index is set up with the "all" privilege so that our users can log in, change settings and create visualizations. The myindex index is set up with the "read" privilege and we use a query to specify the documents that the users belonging to the role will be able to see.

*Note 1: The quotes inside the query have been escaped with the backslash character "\".  This is required since we want the query itself to be indexed as a string and not as an object with a "match" property.

*Note 2: These are the minimum options for the goals of this exercise, you can find more advanced options here:

https://www.elastic.co/guide/en/shield/current/defining-roles.html

This creates a role that will have access to only the EAST region:

POST /_shield/role/myroleEAST
{
  "indices": [ 
    {
      "names": [ ".kibana*"],
      "privileges": ["all"]
    },
    {
      "names": [ "myindex*" ], 
      "privileges": [ "read", "view_index_metadata" ],
      "query": "{\"match\": {\"region\": \"EAST\"}}"
    }
  ]
}

This creates a role that will have access to only the WEST region:

POST /_shield/role/myroleWEST
{
  "indices": [ 
    {
      "names": [ ".kibana*"],
      "privileges": ["all"]
    },
    {
      "names": [ "myindex*" ], 
      "privileges": ["read", "view_index_metadata" ],
      "query": "{\"match\": {\"region\": \"WEST\"}}"
    }
  ]
}

You can verify the roles by using:

GET /_shield/role

STEP 4: Create Users

Now we are ready to create the three users. One user will be assigned the role that allows access to documents marked with the EAST region, another user to the WEST region and the last account will include both roles.

*Note 1: A user assigned several roles will get the cumulative total of privileges. The permissions will get effectively OR'd together and will result in "most privilege" access.

*Note 2: These are the minimum fields required to create the users, for more advanced options:

https://www.elastic.co/guide/en/shield/current/shield-rest.html#shield-users-rest

This will create the user myuserEAST and assign it the myroleEAST role:

POST /_shield/user/myuserEAST
{
  "password" : "mypassword", 
  "roles" : [ "myroleEAST" ]
}

This will create the user myuserWEST and assign it the myroleWEST role:

POST /_shield/user/myuserWEST
{
  "password" : "mypassword", 
  "roles" : [ "myroleWEST" ]
}

This will create the user myuserManager and assign it both roles:

POST /_shield/user/myuserManager
{
  "password" : "mypassword", 
  "roles" : [ "myroleEAST","myroleWEST" ]
}

You can verify the users by:

GET /_shield/user

Kibana

In this section, we will configure Kibana to read our new index, myindex, then create a visualization. You will need to log into Kibana with an admin role user to configure the initial dashboard.

STEP 1: Set Index Pattern in Kibana

1. Select the Settings Tab
2. Uncheck the "Index contains time-based events" box.
3. In the "index name or pattern" text box, enter "myindex*"
4. Click on "Create"

STEP 2: Create Visualization in Kibana

1. Select Visualize tab
2. Select the Vertical bar chart
3. In the Step 2, click "From a new search"
4. In the Select an Index Pattern drop-down box, select "myindex*"



5. Click on X-Axis in buckets section under the "select bucket type"
6. In Aggregation, Select "Terms"
7. In Field, Select "region"



8. Click on "Apply Changes" button (Green button with white arrow)
9. Click "Save Visualization" button



10. Type name for the visualization, "DLS Visualization Demo" and click "save"

STEP 3: Test Users

The tests below demonstrate how the query clause defined in the role restricts the data for the users.  If there are several visualizations in a dashboard on the same data, the window into the data will also be applied.

1. Log into Kibana with the myuserEAST user
2. Select the Visualizations tab
3. Click on DLS Visualization Demo in the saved visualizations.

You should see the visualization with only the EAST bar:

Repeat the test with the myuserWEST user and you should see only the WEST bar.

Repeat the test with the myuserManager and you should see both EAST and WEST bars.

Summary

This article provided a base starting point to demonstrate how Shield can be configured with Elasticsearch to restrict Kibana visualization data.

Although we used very basic and minimal configurations, the procedures can be customized in several ways to achieve security goals in your architecture and allow for more complex configurations.

For example:

• You can use additional authentication realms such as Active Directory, LDAP, etc. in your Shield configuration.
• You can configure a role to multiple indices.
• Field Level Security can also be used to restrict data in the documents.
• Roles can be defined using the roles.yml file although from Shield 2.3.0, it is recommended to use the Roles API to define them.

*Note that if the roles are defined in the roles.yml and in the Roles API, the file definitions will take precedence.

• A wide array of privileges can be defined for the Cluster and Index levels to meet your particular needs.
• Dashboards will also inherit the filtered data set coming from the visualizations.

For more information:

Shield installation and configuration

https://www.elastic.co/guide/en/shield/current/installing-shield.html

Sense installation and configuration

https://www.elastic.co/guide/en/sense/current/installing.html

Native user authentication

https://www.elastic.co/guide/en/shield/current/native-realm.html

Configuring role based access control

https://www.elastic.co/guide/en/shield/current/configuring-rbac.html

Setting up field and document level security

https://www.elastic.co/guide/en/shield/current/setting-up-field-and-document-level-security.html