Upgrade
editUpgrade
editThis section gives general recommendations for upgrading Beats shippers:
If you’re upgrading other products in the stack, also read the Elastic Stack Installation and Upgrade Guide.
Upgrade between minor versions
editAs a general rule, you can upgrade between minor versions (for example, 8.x to 8.y, where x < y) by simply installing the new release and restarting the Beat process. Beats typically maintain backwards compatibility for configuration settings and exported fields. Please review the release notes for potential exceptions.
Upgrading between non-consecutive major versions (e.g. 6.x to 8.x) is not supported.
Upgrade from 7.x to 8.x
editBefore upgrading your Beats, review the breaking changes and the Release notes.
If you’re upgrading other products in the stack, also read the Elastic Stack Installation and Upgrade Guide.
We recommend that you fully upgrade Elasticsearch and Kibana to version 8.0 before upgrading Beats. The Beats version must be lower than or equal to the Elasticsearch version. Beats cannot send data to older versions of Elasticsearch.
If you use the Uptime app in Kibana, make sure you add heartbeat-8*
and synthetics-*
to Uptime indices
on the Settings page. The first index is used by newer versions of a Beat, while the latter is used by Fleet.
If you’re on Beats 7.0 through 7.16, upgrade the Elastic Stack and Beats to version 7.17 before proceeding with the 8.0 upgrade.
Upgrading between non-consecutive major versions (e.g. 6.x to 8.x) is not supported.
Please read through all upgrade steps before proceeding. These steps are required before running the software for the first time.
Upgrade to Beats 7.17 before upgrading to 8.0
editThe upgrade procedure assumes that you have Beats 7.17 installed. If you’re on a previous 7.x version of Beats, upgrade to version 7.17 first. If you’re using other products in the Elastic Stack, upgrade Beats as part of the Elastic Stack upgrade process.
After upgrading to 7.17, go to Index Management in Kibana and verify that the 7.17 index template has been loaded into Elasticsearch.
If the 7.17 index template is not loaded, load it now.
If you created custom dashboards prior to version 7.17, you must upgrade them to 7.17 before proceeding. Otherwise, the dashboards will stop working because Kibana no longer provides the API used for dashboards in 7.x.
Upgrade Beats binaries to 8.0
editBefore upgrading:
- Stop the existing Beats process by using the appropriate command for your system.
-
Back up the
data
andconfig
directories by copying them to another location.The location of these directories depends on the installation method. To see the current paths, start the Beat from a terminal, and the
data
andconfig
paths are printed at startup.
To upgrade using a Debian or RPM package:
-
Use
rpm
ordpkg
to install the new package. All files are installed in the appropriate location for the operating system and Beats config files are not overwritten.
To upgrade using a zip or compressed tarball:
-
Extract the zip or tarball to a new directory. This is critical if you
are not using external
config
anddata
directories. -
Set the following options in the Beats configuration file:
-
Set
path.config
to point to your externalconfig
directory. If you are not using an externalconfig
directory, copy your old configuration over to the new installation. -
Set
path.data
to point to your external data directory. If you are not using an externaldata
directory, copy your old data directory over to the new installation. -
Set
path.logs
to point to the location where you want to store your logs. If you do not specify this setting, logs are stored in the directory you extracted the archive to.
-
Set
Complete the upgrade tasks described in the following sections before restarting the Beats process.
Migrate configuration files
editBeats 8.0 comes with several backwards incompatible configuration changes. Before upgrading, review the 8.0 document. Also review the full list of breaking changes in the Beats version 8.0.0.
Where possible, we kept the old configuration options working, but deprecated them. However, deprecation was not always possible, so if you use any of the settings described under breaking changes, make sure you understand the alternatives that we provide.
Load the 8.0 Elasticsearch index templates
editStarting in version 8.0, the default Elasticsearch index templates configure data streams instead of traditional Elasticsearch indices. Data streams are optimized for storing append-only time series data. They are well-suited for logs, events, metrics, and other continuously generated data. However, unlike traditional Elasticsearch indices, data streams support create operations only; they do not support update and delete operations.
To use data streams, load the default index templates before ingesting any data into Elasticsearch.
beatname setup --index-management
beatname setup --index-management
./beatname setup --index-management
./beatname setup --index-management
docker run --net="host" docker.elastic.co/beats/beatname:8.13.4 setup --index-management
Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select Run As Administrator).
From the PowerShell prompt, change to the directory where you installed a Beat, and run:
PS > .\beatname.exe setup --index-management
If you’re not collecting time series data, you can continue to use Beats to send data to aliases and indices. To do this, create a custom index template and load it manually. To learn more about creating index templates, refer to Index templates.
Load 8.0 dashboards
editWe recommend that you load the 8.0 Kibana dashboards after upgrading Kibana and Beats. That way, you can take advantage of improvements added in 8.0. To load the dashboards, run:
beatname setup --dashboards
beatname setup --dashboards
./beatname setup --dashboards
./beatname setup --dashboards
docker run --net="host" docker.elastic.co/beats/beatname:8.13.4 setup --dashboards
Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select Run As Administrator).
From the PowerShell prompt, change to the directory where you installed a Beat, and run:
PS > .\beatname.exe setup --dashboards
Migrate custom dashboards and visualizations
editAll Elastic Beats send events with ECS-compliant field names. If you have any custom Kibana dashboards or visualizations that use old fields, adjust them now to use ECS field names.
To learn more about ECS, refer to the ECS overview.
If you enabled the compatibility layer in 7.x (that is, if you set
migration.6_to_7.enabled: true
), make sure your custom dashboards no longer
rely on the old aliases created by that setting. The old aliases are no longer
supported. They may continue to work, but will be removed without notice in
a future release.
Start your upgraded Beats
editAfter you’ve completed the migration, start the upgraded Beat. Use the command that works with your system.
Check the console and logs for errors.
In Kibana, go to Discover and verify that events are streaming into Elasticsearch.
Troubleshoot Beats upgrade issues
editThis section describes common problems you might encounter when upgrading to Beats 8.x.
You can avoid some of these problems by reading Upgrade from 7.x to 8.x before upgrading Beats.
Beats is unable to send update or deletion requests to a data stream
editData streams are designed for use cases where existing data is rarely, if ever, updated. You cannot send update or deletion requests for existing documents directly to a data stream.
If needed, you can update or delete documents by submitting requests directly to the document’s backing index. To learn how, refer to the docs about using a data stream.
Timestamp field is missing
editBeats requires a timestamp field to send data to data streams. If the timestamp field added by Beats is inadvertently removed by a processor, Beats will be unable to index the event. To fix the problem, modify your processor configuration to avoid removing the timestamp field.
Missing fields or too many fields in the index
editYou may have run the Beat before loading the required index template. To clean up and start again:
-
Delete the index that was created when you ran the Beat. For example:
DELETE metricbeat-8.13.4-2019.04.02*
Be careful when using wildcards to delete indices. Make sure the pattern matches only the indices you want to delete. The example shown here deletes all data indexed into the metricbeat-8.13.4 indices on 2019.04.02.
-
Delete the index template that was loaded earlier. For example:
DELETE /_index_template/metricbeat-8.13.4
- Load the correct index template. See Load the 8.0 Elasticsearch index templates.
- Restart Beats.