« YAML tips and gotchas Release notes »
Elastic Docs ›Beats Platform Reference [8.16]

Upgrade

edit

Upgrade

edit

This section gives general recommendations for upgrading Beats shippers:

If you’re upgrading other products in the stack, also read the Elastic Stack Installation and Upgrade Guide.

Upgrade between minor versions

edit

As a general rule, you can upgrade between minor versions (for example, 8.x to 8.y, where x < y) by simply installing the new release and restarting the Beat process. Beats typically maintain backwards compatibility for configuration settings and exported fields. Please review the release notes for potential exceptions.

Upgrading between non-consecutive major versions (e.g. 6.x to 8.x) is not supported.

Upgrade from 7.x to 8.x

edit

Before upgrading your Beats, review the breaking changes and the Release notes.

If you’re upgrading other products in the stack, also read the Elastic Stack Installation and Upgrade Guide.

We recommend that you fully upgrade Elasticsearch and Kibana to version 8.0 before upgrading Beats. The Beats version must be lower than or equal to the Elasticsearch version. Beats cannot send data to older versions of Elasticsearch.

If you use the Uptime app in Kibana, make sure you add heartbeat-8* and synthetics-* to Uptime indices on the Settings page. The first index is used by newer versions of a Beat, while the latter is used by Fleet.

If you’re on Beats 7.0 through 7.16, upgrade the Elastic Stack and Beats to version 7.17 before proceeding with the 8.0 upgrade.

Upgrading between non-consecutive major versions (e.g. 6.x to 8.x) is not supported.

Please read through all upgrade steps before proceeding. These steps are required before running the software for the first time.

Upgrade to Beats 7.17 before upgrading to 8.0

edit

The upgrade procedure assumes that you have Beats 7.17 installed. If you’re on a previous 7.x version of Beats, upgrade to version 7.17 first. If you’re using other products in the Elastic Stack, upgrade Beats as part of the Elastic Stack upgrade process.

After upgrading to 7.17, go to Index Management in Kibana and verify that the 7.17 index template has been loaded into Elasticsearch.

Screen capture showing that metricbeat-1.17.0 index template is loaded

If the 7.17 index template is not loaded, load it now.

If you created custom dashboards prior to version 7.17, you must upgrade them to 7.17 before proceeding. Otherwise, the dashboards will stop working because Kibana no longer provides the API used for dashboards in 7.x.

Upgrade Beats binaries to 8.0

edit

Before upgrading:

  1. Stop the existing Beats process by using the appropriate command for your system.

  2. Back up the data and config directories by copying them to another location.

    The location of these directories depends on the installation method. To see the current paths, start the Beat from a terminal, and the data and config paths are printed at startup.

To upgrade using a Debian or RPM package:

  • Use rpm or dpkg to install the new package. All files are installed in the appropriate location for the operating system and Beats config files are not overwritten.

To upgrade using a zip or compressed tarball:

  1. Extract the zip or tarball to a new directory. This is critical if you are not using external config and data directories.

  2. Set the following options in the Beats configuration file:

    • Set path.config to point to your external config directory. If you are not using an external config directory, copy your old configuration over to the new installation.
    • Set path.data to point to your external data directory. If you are not using an external data directory, copy your old data directory over to the new installation.
    • Set path.logs to point to the location where you want to store your logs. If you do not specify this setting, logs are stored in the directory you extracted the archive to.

Complete the upgrade tasks described in the following sections before restarting the Beats process.

Migrate configuration files

edit

Beats 8.0 comes with several backwards incompatible configuration changes. Before upgrading, review the 8.0 document. Also review the full list of breaking changes in the Beats version 8.0.0.

Where possible, we kept the old configuration options working, but deprecated them. However, deprecation was not always possible, so if you use any of the settings described under breaking changes, make sure you understand the alternatives that we provide.

Load the 8.0 Elasticsearch index templates

edit

Starting in version 8.0, the default Elasticsearch index templates configure data streams instead of traditional Elasticsearch indices. Data streams are optimized for storing append-only time series data. They are well-suited for logs, events, metrics, and other continuously generated data. However, unlike traditional Elasticsearch indices, data streams support create operations only; they do not support update and delete operations.

To use data streams, load the default index templates before ingesting any data into Elasticsearch.

beatname setup --index-management

If you’re not collecting time series data, you can continue to use Beats to send data to aliases and indices. To do this, create a custom index template and load it manually. To learn more about creating index templates, refer to Index templates.

Load 8.0 dashboards

edit

We recommend that you load the 8.0 Kibana dashboards after upgrading Kibana and Beats. That way, you can take advantage of improvements added in 8.0. To load the dashboards, run:

beatname setup --dashboards

Migrate custom dashboards and visualizations

edit

All Elastic Beats send events with ECS-compliant field names. If you have any custom Kibana dashboards or visualizations that use old fields, adjust them now to use ECS field names.

To learn more about ECS, refer to the ECS overview.

If you enabled the compatibility layer in 7.x (that is, if you set migration.6_to_7.enabled: true), make sure your custom dashboards no longer rely on the old aliases created by that setting. The old aliases are no longer supported. They may continue to work, but will be removed without notice in a future release.

Start your upgraded Beats

edit

After you’ve completed the migration, start the upgraded Beat. Use the command that works with your system.

Check the console and logs for errors.

In Kibana, go to Discover and verify that events are streaming into Elasticsearch.

Troubleshoot Beats upgrade issues

edit

This section describes common problems you might encounter when upgrading to Beats 8.x.

You can avoid some of these problems by reading Upgrade from 7.x to 8.x before upgrading Beats.

Beats is unable to send update or deletion requests to a data stream

edit

Data streams are designed for use cases where existing data is rarely, if ever, updated. You cannot send update or deletion requests for existing documents directly to a data stream.

If needed, you can update or delete documents by submitting requests directly to the document’s backing index. To learn how, refer to the docs about using a data stream.

Timestamp field is missing

edit

Beats requires a timestamp field to send data to data streams. If the timestamp field added by Beats is inadvertently removed by a processor, Beats will be unable to index the event. To fix the problem, modify your processor configuration to avoid removing the timestamp field.

Missing fields or too many fields in the index

edit

You may have run the Beat before loading the required index template. To clean up and start again:

  1. Delete the index that was created when you ran the Beat. For example:

    DELETE metricbeat-8.16.0-2019.04.02*

    Be careful when using wildcards to delete indices. Make sure the pattern matches only the indices you want to delete. The example shown here deletes all data indexed into the metricbeat-8.16.0 indices on 2019.04.02.

  2. Delete the index template that was loaded earlier. For example:

    DELETE /_index_template/metricbeat-8.16.0
  3. Load the correct index template. See Load the 8.0 Elasticsearch index templates.
  4. Restart Beats.
« YAML tips and gotchas Release notes »