- Filebeat Reference: other versions:
- Overview
- Get started
- Set up and run
- Upgrade
- How Filebeat works
- Configure
- Inputs
- General settings
- Project paths
- Config file loading
- Output
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_json_fields
- decompress_gzip_field
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- registered_domain
- rename
- script
- timestamp
- translate_sid
- truncate_fields
- Autodiscover
- Internal queue
- Load balancing
- Logging
- HTTP endpoint
- Regular expression support
- filebeat.reference.yml
- How to guides
- Beats central management
- Modules
- Modules overview
- ActiveMQ module
- Apache module
- Auditd module
- AWS module
- Azure module
- CEF module
- Cisco module
- CoreDNS module
- Elasticsearch module
- Envoyproxy Module
- Google Cloud module
- haproxy module
- IBM MQ module
- Icinga module
- IIS module
- Iptables module
- Kafka module
- Kibana module
- Logstash module
- MISP module
- MongoDB module
- MSSQL module
- MySQL module
- nats module
- NetFlow module
- Nginx module
- Office 365 module
- Okta module
- Osquery module
- Palo Alto Networks module
- PostgreSQL module
- RabbitMQ module
- Redis module
- Santa module
- Suricata module
- System module
- Traefik module
- Zeek (Bro) Module
- Exported fields
- ActiveMQ fields
- Apache fields
- Auditd fields
- AWS fields
- Azure fields
- Beat fields
- Decode CEF processor fields fields
- CEF fields
- Cisco fields
- Cloud provider metadata fields
- Coredns fields
- Docker fields
- ECS fields
- Elasticsearch fields
- Envoyproxy fields
- Google Cloud fields
- HAProxy fields
- Host fields
- ibmmq fields
- Icinga fields
- IIS fields
- iptables fields
- Jolokia Discovery autodiscover provider fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- MISP fields
- mongodb fields
- mssql fields
- MySQL fields
- NATS fields
- NetFlow fields
- Nginx fields
- Office 365 fields
- Okta fields
- Osquery fields
- panw fields
- PostgreSQL fields
- Process fields
- RabbitMQ fields
- Redis fields
- s3 fields
- Google Santa fields
- Suricata fields
- System fields
- Traefik fields
- Zeek fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Common problems
- Can’t read log files from network volumes
- Filebeat isn’t collecting lines from a file
- Too many open file handlers
- Registry file is too large
- Inode reuse causes Filebeat to skip lines
- Log rotation results in lost or duplicate events
- Open file handlers cause issues with Windows file rotation
- Filebeat is using too much CPU
- Dashboard in Kibana is breaking up data fields incorrectly
- Fields are not indexed or usable in Kibana visualizations
- Filebeat isn’t shipping the last line of a file
- Filebeat keeps open file handlers of deleted files for a long time
- Filebeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- Contribute to Beats
Grant users access to secured resources
editGrant users access to secured resources
editYou can use role-based access control to grant users access to secured resources. The roles that you set up depend on your organization’s security requirements and the minimum privileges required to use specific features.
Typically you need the create the following separate roles:
- setup role for setting up index templates and other dependencies
- monitoring role for sending monitoring information
- writer role for publishing events collected by Filebeat
- reader role for Kibana users who need to view and create visualizations that access Filebeat data
X-Pack security provides built-in roles that grant a subset of the privileges needed by Filebeat users. When possible, use the built-in roles to minimize the affect of future changes on your security strategy.
Grant privileges and roles needed for setup
editSetting up Filebeat is an admin-level task that requires extra privileges. As a best practice, grant the setup role to administrators only, and use a more restrictive role for event publishing.
Administrators who set up Filebeat typically need to load mappings, dashboards, and other objects used to index data into Elasticsearch and visualize it in Kibana.
To grant users the required privileges:
-
Create a setup role, called something like
filebeat_setup
, that has the following privileges:Type Privilege Purpose Cluster
monitor
Retrieve cluster details (e.g. version)
Cluster
manage_ilm
Set up and manage index lifecycle management (ILM) policy
Cluster
manage_ml
Set up Machine Learning job configurations
Index
manage
onfilebeat-*
indicesSet up aliases used by ILM
Index
read
onfilebeat-*
indicesRead Filebeat indices in order to set up Machine Learning jobs
Omit any privileges that aren’t relevant in your environment.
These instructions assume that you are using the default name for Filebeat indices. If you are using a custom name, modify the privileges to match your index naming pattern.
-
Assign the setup role, along with the following built-in roles, to users who need to set up Filebeat:
Role Purpose kibana_admin
Load dependencies, such as example dashboards, if available, into Kibana
ingest_admin
Set up index templates and, if available, ingest pipelines
beats_admin
Enroll and manage configurations in Beats central management
Omit any roles that aren’t relevant in your environment.
Grant privileges and roles needed for monitoring
editX-Pack security provides built-in users and roles for monitoring. The privileges and roles needed depend on the method used to collect monitoring data.
Important note for Elastic Cloud users
Built-in users are not available when running our hosted Elasticsearch Service on Elastic Cloud. To send monitoring data securely, create a monitoring user and grant it the roles described in the following sections.
-
If you’re using internal collection to collect metrics about Filebeat, X-Pack security provides the
beats_system
built-in user andbeats_system
built-in role to send monitoring information. You can use the built-in user, if it’s available in your environment, or create a user who has the privileges needed to send monitoring information.If you use the
beats_system
user, make sure you set the password.If you don’t use the
beats_system
user:-
Create a monitoring role, called something like
filebeat_monitoring
, that has the following privileges:Type Privilege Purpose Cluster
monitor
Retrieve cluster details (e.g. version)
Index
create_index
on.monitoring-beats-*
indicesCreate monitoring indices in Elasticsearch
Index
create_doc
on.monitoring-beats-*
indicesWrite monitoring events into Elasticsearch
-
Assign the monitoring role, along with the following built-in roles, to users who need to monitor Filebeat:
Role Purpose kibana_user
Use Kibana
monitoring_user
Use Stack Monitoring in Kibana to monitor Filebeat
-
-
If you’re using Metricbeat to collect metrics about Filebeat, X-Pack security provides the
remote_monitoring_user
built-in user, and theremote_monitoring_collector
andremote_monitoring_agent
built-in roles for collecting and sending monitoring information. You can use the built-in user, if it’s available in your environment, or create a user who has the privileges needed to collect and send monitoring information.If you use the
remote_monitoring_user
user, make sure you set the password.If you don’t use the
remote_monitoring_user
user:- Create a user on the production cluster who will collect and send monitoring information.
-
Assign the following roles to the user:
Role Purpose remote_monitoring_collector
Collect monitoring metrics from Filebeat
remote_monitoring_agent
Send monitoring data to the monitoring cluster
-
Assign the following role to users who will view the monitoring data in Kibana:
Role Purpose monitoring_user
Use Stack Monitoring in Kibana to monitor Filebeat
Grant privileges and roles needed for publishing
editUsers who publish events to Elasticsearch need to create and write to Filebeat indices. To minimize the privileges required by the writer role, use the setup role to pre-load dependencies. This section assumes that you’ve pre-loaded dependencies.
When using ILM, turn off the ILM setup check in the Filebeat config file before running Filebeat to publish events:
setup.ilm.check_exists: false
To grant the required privileges:
-
Create a writer role, called something like
filebeat_writer
, that has the following privileges:The
monitor
cluster privilege and thecreate_doc
privilege onfilebeat-*
indices are required in every configuration.Type Privilege Purpose Cluster
monitor
Retrieve cluster details (e.g. version)
Cluster
read_ilm
Read the ILM policy when connecting to clusters that support ILM. Not needed when
setup.ilm.check_exists
isfalse
.Cluster
cluster:admin/ingest/pipeline/get
Check for ingest pipelines used by modules. Needed when using modules.
Index
create_doc
onfilebeat-*
indicesWrite events into Elasticsearch
Index
view_index_metadata
onfilebeat-*
indicesCheck for alias when connecting to clusters that support ILM. Not needed when
setup.ilm.check_exists
isfalse
.Index
create_index
onfilebeat-*
indicesCreate daily indices when connecting to clusters that do not support ILM. Not needed when using ILM.
Omit any privileges that aren’t relevant in your environment.
- Assign the writer role to users who will index events into Elasticsearch.
Grant privileges and roles needed to read Filebeat data from Kibana
editKibana users typically need to view dashboards and visualizations that contain Filebeat data. These users might also need to create and edit dashboards and visualizations. If you’re using Beats central management, some of these users might need to create and manage configurations.
To grant users the required privileges:
-
Create a reader role, called something like
filebeat_reader
, that has the following privilege:Type Privilege Purpose Index
read
onfilebeat-*
indicesRead data indexed by Filebeat
Spaces
Read
orAll
on Dashboards, Visualize, and DiscoverAllow the user to view, edit, and create dashboards, as well as browse data.
Spaces
Read
orAll
on Kibana LogsAllow the use of Kibana Logs
-
Assign the reader role, along with the following built-in roles, to users who need to read Filebeat data:
Role Purpose monitoring_user
Allow users to monitor the health of Filebeat itself. Only assign this role to users who manage Filebeat.
beats_admin
Create and manage configurations in Beats central management. Only assign this role to users who need to use Beats central management. +
Learn more about users and roles
editWant to learn more about creating users and roles? See Secure a cluster. Also see:
- Security privileges for a description of available privileges
- Built-in roles for a description of roles that you can assign to users
On this page
ElasticON events are back!
Learn about the Elastic Search AI Platform from the experts at our live events.
Register now