Control access to APM data

edit

Starting in version 8.2.0, the APM UI is Kibana space aware. This allows you to separate your data—​and access to that data—​by team, use case, service environment, or any other filter that you choose.

To take advantage of this feature, your APM data needs to be written to different data steams. One way to accomplish this is with different namespaces. For example, you can send production data to an APM integration with a namespace of production, while sending staging data to a different APM integration with a namespace of staging.

Multiple APM integration instances is not required though. The simplest way to take advantage of this feature is by creating filtered aliases. See the guide below for more information.

Guide: Separate staging and production data

edit

This guide will explain how to separate your staging and production data. This can be helpful to either remove noise when troubleshooting a production issue, or to create more granular access control for certain data.

This guide assumes that you:

  • Are sending both staging and production APM data to an Elasticsearch cluster.
  • Have configured the environment variable in your APM agent configurations. This variable sets the service.environment field in APM documents. You should have documents where service.environment: production and service.environment: staging. If this field is empty, see service environment filter to learn how to set this value.
Step 1: Create filtered aliases
edit

The APM UI uses index patterns to query your APM data. An index pattern can match data streams, indices, and/or aliases. The default values are:

Index setting Default index pattern

Error

logs-apm*

Span/Transaction

traces-apm*

Metrics

metrics-apm*

The default index settings also query the apm-* data view. This data view matches APM data shipped in earlier versions of APM (prior to v8.0).

Instead of querying the default APM data views, we can create filtered aliases for the APM UI to query. A filtered alias is a secondary name for a group of data streams that has a user-defined filter to limit the documents that the alias can access.

To separate staging and production APM data, we’d need to create six filtered aliases—​three aliases for each service environment:

Index setting production env staging env

Error

production-logs-apm

staging-logs-apm

Span/Transaction

production-traces-apm

staging-traces-apm

Metrics

production-metrics-apm

staging-metrics-apm

The production-<event>-apm aliases will contain a filter that only provides access to documents where the service.environment is production. Similarly, the staging-<event>-apm aliases will contain a filter that only provides access to documents where the service.environment is staging.

To create these six filtered aliases, use the Elasticsearch Aliases API. In Kibana, open Dev Tools and run the following POST requests.

traces-apm* production alias example
POST /_aliases?pretty
{
  "actions": [
    {
      "add": {
        "index": "traces-apm*", 
        "alias": "production-traces-apm", 
        "filter": {
          "term": {
            "service.environment": {
              "value": "production" 
            }
          }
        }
      }
    }
  ]
}

This example matches the APM traces data stream

The alias must not match the default APM index (traces-apm*,apm-*)

Only match documents where service.environment: production

logs-apm* production alias example
POST /_aliases?pretty
{
  "actions": [
    {
      "add": {
        "index": "logs-apm*", 
        "alias": "production-logs-apm", 
        "filter": {
          "term": {
            "service.environment": {
              "value": "production" 
            }
          }
        }
      }
    }
  ]
}

This example matches the APM logs data stream

The alias must not match the default APM index (logs-apm*,apm-*)

Only match documents where service.environment: production

metrics-apm* production alias example
POST /_aliases?pretty
{
  "actions": [
    {
      "add": {
        "index": "metrics-apm*", 
        "alias": "production-metrics-apm", 
        "filter": {
          "term": {
            "service.environment": {
              "value": "production" 
            }
          }
        }
      }
    }
  ]
}

This example matches the APM metrics data stream

The alias must not match the default APM index (metrics-apm*,apm-*)

Only match documents where service.environment: production

traces-apm* staging alias example
POST /_aliases?pretty
{
  "actions": [
    {
      "add": {
        "index": "traces-apm*", 
        "alias": "staging-traces-apm", 
        "filter": {
          "term": {
            "service.environment": {
              "value": "staging" 
            }
          }
        }
      }
    }
  ]
}

This example matches the APM traces data stream

The alias must not match the default APM index (traces-apm*,apm-*)

Only match documents where service.environment: staging

logs-apm* staging alias example
POST /_aliases?pretty
{
  "actions": [
    {
      "add": {
        "index": "logs-apm*", 
        "alias": "staging-logs-apm", 
        "filter": {
          "term": {
            "service.environment": {
              "value": "staging" 
            }
          }
        }
      }
    }
  ]
}

This example matches the APM logs data stream

The alias must not match the default APM index (logs-apm*,apm-*)

Only match documents where service.environment: staging

metrics-apm* staging alias example
POST /_aliases?pretty
{
  "actions": [
    {
      "add": {
        "index": "metrics-apm*", 
        "alias": "staging-metrics-apm", 
        "filter": {
          "term": {
            "service.environment": {
              "value": "staging" 
            }
          }
        }
      }
    }
  ]
}

This example matches the APM metrics data stream

The alias must not match the default APM index (metrics-apm*,apm-*)

Only match documents where service.environment: staging

Step 2: Create Kibana spaces
edit

Next, you’ll need to create a Kibana space for each service environment. To create these spaces, navigate to Stack Management > Spaces > Create a space. For this guide, we’ve created two Kibana spaces, one named production and one named staging.

See Kibana spaces for more information on creating a space.

Step 3: Update APM index settings in each space
edit

Now we can change the default data views that the APM UI queries in each space.

Open the APM UI and navigate to Settings > Indices. Use the table below to update your settings for each space. The values in each column match the names of the filtered aliases we created in step one.

Index setting production space staging space

Error indices

production-logs-apm

staging-logs-apm

Span indices

production-traces-apm

staging-traces-apm

Transaction indices

production-traces-apm

staging-traces-apm

Metrics indices

production-metrics-apm

staging-metrics-apm

Step 4: Create Kibana access roles
edit

In Kibana, navigate to Stack Management > Roles and click Create role.

You’ll need to create two roles: one for staging users (we’ll call this role staging_apm_viewer) and one for production users (we’ll call this role production_apm_viewer).

Using the table below, assign each role the following privileges:

Privileges production_apm_viewer staging_apm_viewer

Index privileges

index: production-*-apm, privilege: read

index: staging-*-apm, privilege: read

Kibana privileges

space: production, feature privileges: APM and User Experience: read

space: staging, feature privileges: APM and User Experience: read

APM role config example

Alternatively, you can use the Elasticsearch Create or update roles API:

Create a production_apm_viewer role

This request creates a production_apm_viewer role:

POST /_security/role/production_apm_viewer
{
  "cluster": [ ],
  "indices": [
    {
      "names": ["production-*-apm"], 
      "privileges": ["read"]
    }
  ],
  "applications": [
    {
      "application" : "kibana-.kibana",
      "privileges" : [
        "feature_apm.read" 
      ],
      "resources" : [
        "space:production" 
      ]
    }
  ]
}

This data view matches all of the production aliases created in step one.

Assigns read privileges for the APM and User Experience apps.

Provides access to the space named production.

Create a staging_apm_viewer role

This request creates a staging_apm_viewer role:

POST /_security/role/staging_apm_viewer
{
  "cluster": [ ],
  "indices": [
    {
      "names": ["staging-*-apm"], 
      "privileges": ["read"]
    }
  ],
  "applications": [
    {
      "application" : "kibana-.kibana",
      "privileges" : [
        "feature_apm.read" 
      ],
      "resources" : [
        "space:staging" 
      ]
    }
  ]
}

This data view matches all of the staging aliases created in step one.

Assigns read privileges for the APM and User Experience apps.

Provides access to the space named staging.

Step 5: Assign users to roles
edit

The last thing to do is assign users to the newly created roles above. Users will only have access to the data within the spaces that they are granted.

For information on how to create users and assign them roles with the Kibana UI, see Securing access to Kibana.

Alternatively, you can use the Elasticsearch Create or update users API.

This example creates a new user and assigns them the production_apm_viewer role created in the previous step. This user will only have access to the production space and data with a service.environment of production. Remember to change the password, full_name, and email fields.

POST /_security/user/production-apm-user
{
  "password" : "l0ng-r4nd0m-p@ssw0rd",
  "roles" : [ "production_apm_viewer" ], 
  "full_name" : "Jane Production Smith",
  "email" : "janesmith@example.com"
}

Assigns the previously created production_apm_viewer role.

This example creates a new user and assigns them the staging_apm_viewer role created in the previous step. This user will only have access to the staging space and data with a service.environment of staging. Remember to change the password, full_name, and email fields.

POST /_security/user/staging-apm-user
{
  "password" : "l0ng-r4nd0m-p@ssw0rd",
  "roles" : [ "staging_apm_viewer" ], 
  "full_name" : "John Staging Doe",
  "email" : "johndoe@example.com"
}

Assigns the previously created staging_apm_viewer role.

Step 6: Marvel
edit

That’s it! Head back to the APM UI and marvel at your space-specific data.