Detection Alerts Migration API

After upgrading Kibana, the latest Elastic Security features will be available for any newly generated detection alerts. However, in order to enable new features for existing detection alerts, migration may be necessary. See Upgrade Elastic Security for instructions specific to your upgrade.

Migrating detection alerts is performed at the index level and requires the following steps:

You can use the Migration Status API to determine which indices contain detection alerts of a particular age, along with migration info for each of those indices.

GET <kibana host>:<port>/api/detection_engine/signals/migration_status?from=now-30d

{
  "indices": [
    {
      "index": ".siem-signals-default-000002",
      "version": 15,
      "signal_versions": [
        {
          "version": 15,
          "count": 100
        },
        {
          "version": 16,
          "count": 87
        }
      ],
      "migrations": [
        {
          "id": "924f7c50-505f-11eb-ae0a-3fa2e626a51d",
          "status": "pending",
          "version": 16,
          "updated": "2021-01-06T20:41:37.173Z"
        }
      ],
      "is_outdated": true
    },
    {
      "index": ".siem-signals-default-000003",
      "version": 16,
      "signal_versions": [
        {
          "version": 16,
          "count": 54
        }
      ],
      "migrations": [],
      "is_outdated": false
    }
  ]
}

The above response shows two indices: .siem-signals-default-000002 is outdated, with multiple versions of detection alerts and a pending migration, and .siem-signals-default-000003, which is up to date as the current write index.

With the above info, we can compile a list of indices that we wish to migrate.

Find outdated detection alerts with threat intelligence data

After upgrading to Elastic Stack version 7.15.x from release versions 7.12.0 through 7.14.2, you need to migrate detection alerts enriched with threat intelligence data to ensure threat intelligence properly displays in Elastic Security. Run this query to find outdated detection alerts with threat intelligence data:

GET .siem-signals-{KIBANA SPACE ID}/_search
{
  "query": {
    "nested": {
      "path": "threat.indicator",
      "query": {
        "exists": {
          "field": "threat.indicator.matched.*"
        }
      }
    }
  }
}

Migrations are initiated per index. While the process is neither destructive nor interferes with existing data, it may be resource-intensive. As such, it is recommended that you plan your migrations accordingly.

POST <kibana host>:<port>/api/detection_engine/signals/migration

{
  "indices": [
    {
      "index": ".siem-signals-default-000001",
      "migration_id": "923f7c50-505f-11eb-ae0a-3fa2e626a51d",
      "migration_index": ".siem-signals-default-000001-r000016"
    }
  ]
}

The response will include, for each index specified, an ID and destination index for the migration, and an error if unsuccessful.

The finalization endpoint replaces the original index’s alias with the successfully migrated index’s alias. The endpoint is idempotent; therefore, it can safely be used to poll a given migration and, upon completion, finalize it.

POST <kibana host>:<port>/api/detection_engine/signals/finalize_migration

{
  "migrations": [
    {
      "id": "924f7c50-505f-11eb-ae0a-3fa2e626a51d",
      "completed": true,
      "destinationIndex": ".siem-signals-default-000002-r000016",
      "status": "success",
      "sourceIndex": ".siem-signals-default-000002",
      "version": 16,
      "updated": "2021-01-06T22:05:56.859Z"
    }
  ]
}

Finalized migrations will show a response of completed: true, with a corresponding status. If the migration is still running when you attempt to finalize it, its response will show as completed: false.

Migrations favor data integrity over shard size. Consequently, unused or orphaned indices are artifacts of the migration process. A successful migration will result in both the old and new indices being present. As such, the old, orphaned index can (and likely should) be deleted.

While you can delete these indices manually, the endpoint accomplishes this task by applying a deletion policy to the relevant index, causing it to be deleted after 30 days. It also deletes other artifacts specific to the migration implementation.

DELETE <kibana host>:<port>/api/detection_engine/signals/migration

 {
  "migrations": [
    {
      "id": "924f7c50-505f-11eb-ae0a-3fa2e626a51d",
      "destinationIndex": ".siem-signals-default-000002-r000016",
      "status": "success",
      "sourceIndex": ".siem-signals-default-000002",
      "version": 16,
      "updated": "2021-01-06T22:05:56.859Z"
    }
  ]
}

The response will include all migrations that were successfully deleted.