Upgrade migrations

Every time Kibana is upgraded it checks to see if all saved objects, such as dashboards, visualizations, and index patterns, are compatible with the new version. If any saved objects need to be updated, then the automatic saved object migration process is kicked off.

Saved objects are stored in the index specified by the kibana.index configuration setting. By default this is .kibana_N where N is a number that increments every time Kibana runs an upgrade migration on that index. The index alias .kibana points to the most up-to-date index.

While Kibana is starting up and before serving any HTTP traffic, it checks to see if any internal mapping changes or data transformations for existing saved objects are required.

When changes are necessary, a new migration is started. To ensure that only one Kibana instance performs the migration, each instance will attempt to obtain a migration lock by creating a new .kibana_N+1 index. The instance that succeeds in creating the index will then read batches of documents from the existing index, migrate them, and write them to the new index. Once the objects are migrated, the lock is released by pointing the .kibana index alias to the newly upgraded .kibana_N+1 index.

Instances that failed to acquire a lock will log Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. The instance will then wait until .kibana points to an upgraded index before starting up and serving HTTP traffic.

This section highlights common causes of Kibana upgrade failures and how to prevent them.

We highly recommend testing your Kibana upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...} or Unable to migrate the corrupt Saved Object document .... Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.

Matching index templates which specify settings.refresh_interval or mappings are known to interfere with Kibana upgrades.

Prevention: narrow down the index patterns of any user-defined index templates to ensure that these won’t apply to new .kibana* indices.

Note: Kibana < 6.5 creates it’s own index template called kibana_index_template:.kibana and index pattern .kibana. This index template will not interfere and does not need to be changed or removed.

Problems with your Elasticsearch cluster can prevent Kibana upgrades from succeeding. Ensure that your cluster has:

Kibana does not support rolling upgrades. Stop all Kibana instances before starting a newer version to prevent upgrade failures and data loss.

If Kibana terminates unexpectedly while migrating a saved object index, manual intervention is required before Kibana will attempt to perform the migration again. Follow the advice in (preventing migration failures)[preventing-migration-failures] before retrying a migration upgrade.

As mentioned above, Kibana will create a migration lock for each index that requires a migration by creating a new .kibana_N+1 index. For example: if the .kibana alias is pointing to .kibana_5 then the first Kibana that succeeds in creating .kibana_6 will obtain the lock to start migrations.

However, if the instance that obtained the lock fails to migrate the index, all other Kibana instances will be blocked from performing this migration. This includes the instance that originally obtained the lock, it will be blocked from retrying the migration even when restarted.

If you’ve followed the advice in (preventing migration failures)[preventing-migration-failures] and (resolving migration failures)[resolve-migrations-failures] and Kibana is still not able to upgrade successfully, you might choose to rollback Kibana until you’re able to identify the root cause.

In order to rollback after a failed upgrade migration, the saved object index might also have to be rolled back to be compatible with the previous {kibana} version.

After migrations have completed, there will be multiple Kibana indices in Elasticsearch: (.kibana_1, .kibana_2, etc). Kibana only uses the index that the .kibana alias points to. The other Kibana indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling Kibana back to a previous version.