Leverage DLS from connectors in App Search

This guide explains how to use App Search tools to search documents ingested by Elastic connectors which have document level security (DLS) enabled. We’ll be using Elasticsearch-index engines in App Search for our example.

We will go through the following steps:

You’ll need to have an Enterprise Search deployment and to run a self-managed connectors service instance to ingest data with access control. Refer to Connector clients for details on how to set up a connector client and run the connectors service.

In this example, we’ll configure and sync data from SharePoint Online, using the SharePoint Online connector.

The SharePoint Online connector will create two separate Elasticsearch indices:

We want to use the data ingested by the SharePoint Online connector in an App Search engine. To do this, we need to create an Elasticsearch index-based engine.

Follow these steps:

Signed search keys in Elastic App Search give you more control over a user’s search experience. They enable you to restrict the data users can see and search over.

App Search has the concept of search keys and private keys:

App Search also has the concept of signed search keys, which can only be used to search. A signed search key is a JSON Web Token. It is signed with an API key, ideally a read-only private key, using the HMAC with SHA-256 (HS256) algorithm.

A signed API key can contain filters to restrict what a user can search over. We will use documents from the access control index created by the SharePoint Online connector to generate signed API keys.

The access control index will contain documents similar to this example:

GET search-acl-filter-sharepoint/_doc/john@example.co
{
  "_index": "search-acl-filter-sharepoint",
  "_id": "john@example.co",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "identity": {
      "email": "john@example.co",
      "access_control": [
        "john@example.co",
        "Engineering Members"
      ]
    },
    "query": {
      "template": {
        "params": {
          "access_control": [
            "john@example.co",
            "Engineering Members"
            ]
        },
        "source": """
        {
          "bool": {
            "filter": {
              "bool": {
                "should": [
                  {
                    "bool": {
                      "must_not": {
                        "exists": {
                          "field": "_allow_access_control"
                        }
                      }
                    }
                  },
                  {
                    "terms": {
                      "_allow_access_control.enum": ["john@example.co", "Engineering Members"]
                    }
                  }
                ]
              }
            }
          }
        }
        """
      }
    }
  }
}

This document contains the Elasticsearch query that describes which documents the user john@example.com has access to. The access control information is stored in the access_control field. In this case the user has access only to documents that contain "john@example.co" or "Engineering Members" in the _allow_access_control field.

A signed API key has to be signed with a search key. The key used for signing needs to have read access to the App Search engine. Note: This key should be kept secret.

To build a signed key, you need the name of the key and its value. Find these details in Kibana, by going to Search > App Search > Credentials.

Starting from the access control document, we can build an equivalent App Search signed key where the payload looks like:

{
  "filters": {
    "_allow_access_control": {{access_control}}
  },
  "api_key_name": {{name-of-private-key}}
}

In our case, the signed payload looks like this:

{
  "filters": {
    "_allow_permissions": [
      "john@example.co",
      "Engineering Members"
    ]
  },
  "api_key_name": "search-key"
}

The payload will be signed with the value of the public search key. There are various tools and libraries to create signed JWT.

Here is an example using Ruby:

require 'jwt'

key_name = 'search-key'
permissions = [
  "john@example.co",
  "Engineering Members"
]

payload = {
  'filters' => {
    '_allow_permissions' => permissions
  },
  'api_key_name' => key_name
}

key_value = 'search-y4bfy8cue3354u894s4vsnnm'
algorithm = 'HS256'

puts JWT.encode(payload, key_value, algorithm)

Once created the signed key can be used in the authorization headers of search requests.

For example:

curl -X GET 'https://my-deployment.ent.us-west2.gcp.elastic-cloud.com/api/as/v1/engines/sharepoint/search' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <API-KEY>' \
-d '{
  "query": "guidelines"
}'

Now it’s time to test that the signed key works as expected.

Ask the user to issue a search query with the signed API key. Validate that the documents returned are limited to what was specified in the filters of the API key. The results should match the permissions listed in the _allow_access_control field of the documents.

We recommend relying on the connector access control sync to automate and keep documents in sync with changes to the original content source’s user permissions.

In this workflow you will need to handle the generation of the signed API key in the backend of your application, in response to browser sign ins.

Once the key is generated, the backend will also need to return that signed key to the client (browser) to be used in subsequent search requests to your Elastic search engine.

In order to invalidate the signed API keys, you need to invalidate the API key that was used to sign it.

Additionally, if the user’s permission changes, you’ll need to recreate the signed search key.