Troubleshooting Beats upgrade issues
editTroubleshooting Beats upgrade issues
editThis section describes common problems you might encounter when upgrading to Beats 7.x.
You can avoid some of these problems by reading Upgrade from 6.x to 7.x before upgrading Beats.
Missing fields or too many fields in the index
editYou may have run the Beat before loading the required index template. To clean up and start again:
-
Delete the index that was created when you ran the Beat. For example:
DELETE metricbeat-7.2.1-2019.04.02*
Be careful when using wildcards to delete indices. Make sure the pattern matches only the indices you want to delete. The example shown here deletes all data indexed into the metricbeat-7.2.1 indices on 2019.04.02.
- If you want the index to work with 6.x dashboards, turn on the compatibility layer. See Enable the compatibility layer for Elastic Common Schema (ECS) fields.
-
Delete the index template that was loaded earlier. For example:
DELETE /_template/metricbeat-7.2.1
Because the index template was loaded without the compatibility layer enabled, the required aliases were not created.
- Load the correct index template. See Upgrade the Elasticsearch index template.
- Restart Beats.
User is not authorized
editBecause index lifecycle management is on by default in 7.0, you might encounter new errors related to user authorization when you run version 7.0 against an Elasticsearch cluster that supports index lifecycle management.
[cluster:monitor/main] is unauthorized for user
editProblem: The Beats user is unable to send monitoring information.
Solution: Grant the monitor
cluster privilege.
[cluster:admin/ilm/put] is unauthorized for user
editProblem: The Beats user is not authorized to load ILM policies.
Solution: Grant the manage_ilm
cluster privilege.
[indices:admin/template/put] is unauthorized for user
editProblem: Automatic index template loading is required when ILM is enabled, but the user is not authorized to manage index templates.
Solution: Grant the manage_index_templates
cluster privilege.
[indices:admin/aliases] is unauthorized for user
editProblem: The Beats user is unable to set up aliases needed by the compatibility layer.
Solution: Grant the manage
privilege on the Beats indices.
[indices:data/write/bulk] is unauthorized for user
editProblem: The Beats user is unable to write events to Elasticsearch.
Solution: Grant the index
privilege on the Beats indices.
6.x dashboards not showing data from 7.0 shippers
editYou might have run the Beat without turning on the compatibility layer. See Enable the compatibility layer for Elastic Common Schema (ECS) fields then clean up your environment as described under Missing fields or too many fields in the index.
Data parsed by Logstash not appearing in 7.0 dashboards
editYou might be writing to an index that doesn’t match the index pattern used by Beats. See How to use versioned index templates when the output is not Elasticsearch.
"shard failed" error when viewing Beats dashboards in Kibana
editAfter upgrading to Elasticsearch 7.0, any indices created by Beats 6.6 or older will not
work in Kibana dashboards until the index.query.default_field
setting is added
to each index. Indices created in Beats 6.7 or later are unaffected.
Background: Starting in Elasticsearch 7.0, some query types, such as Simple Query
String, have a limit to the number of fields they will query against. Because
Beats indices often contain more fields than this cap, the
index.query.default_field
index setting is used to inform Elasticsearch which fields to
use by default when no field is specified for a query.
To add the setting to the index, you can use the 7.0 Upgrade Assistant, or add the setting manually.
To add the setting manually, first identify the list of fields that you want to
set as the default search fields, then specify the list of fields in the
default_field
setting.
For example, here’s a snippet that shows how to add default search fields to a Metricbeat 6.6 index. This example is truncated. Full examples for Metricbeat and Filebeat are available in this file.
PUT /metricbeat-6.6.2-2019.04.09/_settings { "index": { "query": { "default_field": [ "aerospike.namespace.name", "aerospike.namespace.node.host", "aerospike.namespace.node.name", "apache.status.hostname", "beat.hostname", "beat.name", "beat.timezone", "beat.version", ... ] } } }