IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Standard upgrade Configure monitoring in Kibana »
Elastic Docs ›Kibana Guide [7.10] ›Set up ›Upgrade Kibana

Upgrade migrations

edit

Upgrade migrations

edit

Every time Kibana is upgraded it checks to see if all saved objects, such as dashboards, visualizations, and index patterns, are compatible with the new version. If any saved objects need to be updated, then the automatic saved object migration process is kicked off.

6.7 includes an Upgrade Assistant to help you prepare for your upgrade to 7.0. To access the assistant, go to Management > 7.0 Upgrade Assistant.

The following instructions assumes Kibana is using the default index names. If the kibana.index or xpack.tasks.index configuration settings were changed these instructions will have to be adapted accordingly.

Background

edit

Saved objects are stored in two indices:

  • .kibana_N, or if set, the kibana.index configuration setting
  • .kibana_task_manager_N, or if set, the xpack.tasks.index configuration setting

For each of these indices, N is a number that increments every time Kibana runs an upgrade migration on that index. The index aliases .kibana and .kibana_task_manager point to the most up-to-date index.

While Kibana is starting up and before serving any HTTP traffic, it checks to see if any internal mapping changes or data transformations for existing saved objects are required.

When changes are necessary, a new migration is started. To ensure that only one Kibana instance performs the migration, each instance will attempt to obtain a migration lock by creating a new .kibana_N+1 index. The instance that succeeds in creating the index will then read batches of documents from the existing index, migrate them, and write them to the new index. Once the objects are migrated, the lock is released by pointing the .kibana index alias the new upgraded .kibana_N+1 index.

Instances that failed to acquire a lock will log Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. The instance will then wait until .kibana points to an upgraded index before starting up and serving HTTP traffic.

Prior to 6.5.0, saved objects were stored directly in an index named .kibana. After upgrading to version 6.5+, Kibana will migrate this index into .kibana_N and set .kibana up as an index alias.
Prior to 7.4.0, task manager tasks were stored directly in an index name .kibana_task_manager. After upgrading to version 7.4+, Kibana will migrate this index into .kibana_task_manager_N and set .kibana_task_manager up as an index alias.

Preventing migration failures

edit

This section highlights common causes of Kibana upgrade failures and how to prevent them.

Corrupt saved objects
edit

We highly recommend testing your Kibana upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...} or Unable to migrate the corrupt Saved Object document .... Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.

User defined index templates that causes new .kibana* indices to have incompatible settings or mappings
edit

Matching index templates which specify settings.refresh_interval or mappings are known to interfere with Kibana upgrades.

Prevention: narrow down the index patterns of any user-defined index templates to ensure that these won’t apply to new .kibana* indices.

Note: Kibana < 6.5 creates it’s own index template called kibana_index_template:.kibana and index pattern .kibana. This index template will not interfere and does not need to be changed or removed.

An unhealthy Elasticsearch cluster
edit

Problems with your Elasticsearch cluster can prevent Kibana upgrades from succeeding. Ensure that your cluster has:

  • enough free disk space, at least twice the amount of storage taken up by the .kibana and .kibana_task_manager indices
  • sufficient heap size
  • a "green" cluster status
Running different versions of Kibana connected to the same Elasticsearch index
edit

Kibana does not support rolling upgrades. Stop all Kibana instances before starting a newer version to prevent upgrade failures and data loss.

Incompatible xpack.tasks.index configuration setting
edit

For Kibana < 7.5.1, if the task manager index is set to .tasks with the configuration setting xpack.tasks.index: ".tasks", upgrade migrations will fail. Kibana 7.5.1 and later prevents this by refusing to start with an incompatible configuration setting.

Resolving migration failures

edit

If Kibana terminates unexpectedly while migrating a saved object index, manual intervention is required before Kibana will attempt to perform the migration again. Follow the advice in (preventing migration failures)[preventing-migration-failures] before retrying a migration upgrade.

As mentioned above, Kibana will create a migration lock for each index that requires a migration by creating a new .kibana_N+1 index. For example: if the .kibana_task_manager alias is pointing to .kibana_task_manager_5 then the first Kibana that succeeds in creating .kibana_task_manager_6 will obtain the lock to start migrations.

However, if the instance that obtained the lock fails to migrate the index, all other Kibana instances will be blocked from performing this migration. This includes the instance that originally obtained the lock, it will be blocked from retrying the migration even when restarted.

Retry a migration by restoring a backup snapshot:
edit
  1. Before proceeding ensure that you have a recent and successful backup snapshot of all .kibana* indices.
  2. Shutdown all Kibana instances to be 100% sure that there are no instances currently performing a migration.
  3. Delete all saved object indices with DELETE /.kibana*
  4. Restore the `.kibana* indices and their aliases from the backup snapshot. See Elasticsearch snapshots
  5. Start up all Kibana instances to retry the upgrade migration.
(Not recommended) Retry a migration without a backup snapshot:
edit
  1. Shutdown all Kibana instances to be 100% sure that there are no instances currently performing a migration.
  2. Identify any migration locks by comparing the output of GET /_cat/aliases and GET /_cat/indices. If e.g. .kibana is pointing to .kibana_4 and there is a .kibana_5 index, the .kibana_5 index will act like a migration lock blocking further attempts. Be sure to check both the .kibana and .kibana_task_manager aliases and their indices.
  3. Remove any migration locks e.g. DELETE /.kibana_5.
  4. Start up all Kibana instances.

Rolling back to a previous version of Kibana

edit

If you’ve followed the advice in (preventing migration failures)[preventing-migration-failures] and (resolving migration failures)[resolve-migrations-failures] and Kibana is still not able to upgrade successfully, you might choose to rollback Kibana until you’re able to identify the root cause.

Before rolling back Kibana, ensure that the version you wish to rollback to is compatible with your Elasticsearch cluster. If the version you’re rolling back to is not compatible, you will have to also rollback Elasticsearch.
Any changes made after an upgrade will be lost when rolling back to a previous version.

In order to rollback after a failed upgrade migration, the saved object indices might also have to be rolled back to be compatible with the previous {kibana} version.

Rollback by restoring a backup snapshot:
edit
  1. Before proceeding ensure that you have a recent and successful backup snapshot of all .kibana* indices.
  2. Shutdown all Kibana instances to be 100% sure that there are no instances currently performing a migration.
  3. Delete all saved object indices with DELETE /.kibana*
  4. Restore the `.kibana* indices and their aliases from the backup snapshot. See Elasticsearch snapshots
  5. Start up all Kibana instances on the older version you wish to rollback to.
(Not recommended) Rollback without a backup snapshot:
edit

Kibana does not run a migration for every saved object index on every upgrade. A Kibana version upgrade can cause no migrations, migrate only the .kibana or the .kibana_task_manager index or both. Carefully read the logs to ensure that you’re only deleting indices created by a later version of Kibana to avoid data loss.

  1. Shutdown all Kibana instances to be 100% sure that there are no Kibana instances currently performing a migration.
  2. Create a backup snapshot of the .kibana* indices.
  3. Use the logs from the upgraded instances to identify which indices Kibana attempted to upgrade. The server logs will contain an entry like [savedobjects-service] Creating index .kibana_4. and/or [savedobjects-service] Creating index .kibana_task_manager_2. If no indices were created after upgrading Kibana then no further action is required to perform a rollback, skip ahead to step (5). If you’re running multiple Kibana instances, be sure to inspect all instances' logs.
  4. Delete each of the indices identified in step (2). e.g. DELETE /.kibana_task_manager_2
  5. Inspect the output of GET /_cat/aliases. If either the .kibana and/or .kibana_task_manager alias is missing, these will have to be created manually. Find the latest index from the output of GET /_cat/indices and create the missing alias to point to the latest index. E.g. if the .kibana alias was missing and the latest index is .kibana_3 create a new alias with POST /.kibana_3/_aliases/.kibana.
  6. Start up Kibana on the older version you wish to rollback to.

Handling old .kibana_N indices

edit

After migrations have completed, there will be multiple Kibana indices in Elasticsearch: (.kibana_1, .kibana_2, etc). Kibana only uses the index that the .kibana alias points to. The other Kibana indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling Kibana back to a previous version.

« Standard upgrade Configure monitoring in Kibana »