- Filebeat Reference: other versions:
- Filebeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade
- How Filebeat works
- Configure
- Inputs
- Multiline messages
- AWS CloudWatch
- AWS S3
- Azure Event Hub
- Azure Blob Storage
- Benchmark
- CEL
- Cloud Foundry
- CometD
- Container
- Entity Analytics
- ETW
- filestream
- GCP Pub/Sub
- Google Cloud Storage
- HTTP Endpoint
- HTTP JSON
- journald
- Kafka
- Log
- MQTT
- NetFlow
- Office 365 Management Activity API
- Redis
- Salesforce
- Stdin
- Streaming
- Syslog
- TCP
- UDP
- Unix
- winlog
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- append
- cache
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- parse_aws_vpc_flow_log
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- timestamp
- translate_ldap_attribute
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- Feature flags
- filebeat.reference.yml
- Inputs
- How to guides
- Override configuration settings
- Load the Elasticsearch index template
- Change the index name
- Load Kibana dashboards
- Load ingest pipelines
- Enrich events with geoIP information
- Deduplicate data
- Parse data using an ingest pipeline
- Use environment variables in the configuration
- Avoid YAML formatting problems
- Migrate
log
input configurations tofilestream
- Migrating from a Deprecated Filebeat Module
- Modules
- Modules overview
- ActiveMQ module
- Apache module
- Auditd module
- AWS module
- AWS Fargate module
- Azure module
- CEF module
- Check Point module
- Cisco module
- CoreDNS module
- CrowdStrike module
- Cyberark PAS module
- Elasticsearch module
- Envoyproxy Module
- Fortinet module
- Google Cloud module
- Google Workspace module
- HAproxy module
- IBM MQ module
- Icinga module
- IIS module
- Iptables module
- Juniper module
- Kafka module
- Kibana module
- Logstash module
- Microsoft module
- MISP module
- MongoDB module
- MSSQL module
- MySQL module
- MySQL Enterprise module
- NATS module
- NetFlow module
- Nginx module
- Office 365 module
- Okta module
- Oracle module
- Osquery module
- Palo Alto Networks module
- pensando module
- PostgreSQL module
- RabbitMQ module
- Redis module
- Salesforce module
- Santa module
- Snyk module
- Sophos module
- Suricata module
- System module
- Threat Intel module
- Traefik module
- Zeek (Bro) Module
- ZooKeeper module
- Zoom module
- Exported fields
- ActiveMQ fields
- Apache fields
- Auditd fields
- AWS fields
- AWS CloudWatch fields
- AWS Fargate fields
- Azure fields
- Beat fields
- Decode CEF processor fields fields
- CEF fields
- Checkpoint fields
- Cisco fields
- Cloud provider metadata fields
- Coredns fields
- Crowdstrike fields
- CyberArk PAS fields
- Docker fields
- ECS fields
- Elasticsearch fields
- Envoyproxy fields
- Fortinet fields
- Google Cloud Platform (GCP) fields
- google_workspace fields
- HAProxy fields
- Host fields
- ibmmq fields
- Icinga fields
- IIS fields
- iptables fields
- Jolokia Discovery autodiscover provider fields
- Juniper JUNOS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- Lumberjack fields
- Microsoft fields
- MISP fields
- mongodb fields
- mssql fields
- MySQL fields
- MySQL Enterprise fields
- NATS fields
- NetFlow fields
- Nginx fields
- Office 365 fields
- Okta fields
- Oracle fields
- Osquery fields
- panw fields
- Pensando fields
- PostgreSQL fields
- Process fields
- RabbitMQ fields
- Redis fields
- s3 fields
- Salesforce fields
- Google Santa fields
- Snyk fields
- sophos fields
- Suricata fields
- System fields
- threatintel fields
- Traefik fields
- Windows ETW fields
- Zeek fields
- ZooKeeper fields
- Zoom fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Understand logged metrics
- Common problems
- Error extracting container id while using Kubernetes metadata
- Can’t read log files from network volumes
- Filebeat isn’t collecting lines from a file
- Too many open file handlers
- Registry file is too large
- Inode reuse causes Filebeat to skip lines
- Log rotation results in lost or duplicate events
- Open file handlers cause issues with Windows file rotation
- Filebeat is using too much CPU
- Dashboard in Kibana is breaking up data fields incorrectly
- Fields are not indexed or usable in Kibana visualizations
- Filebeat isn’t shipping the last line of a file
- Filebeat keeps open file handlers of deleted files for a long time
- Filebeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Load the Elasticsearch index template
editLoad the Elasticsearch index template
editElasticsearch uses index templates to define:
- Settings that control the behavior of your data stream and backing indices. The settings include the lifecycle policy used to manage backing indices as they grow and age.
- Mappings that determine how fields are analyzed. Each mapping sets the Elasticsearch datatype to use for a specific data field.
The recommended index template file for Filebeat is installed by the
Filebeat packages. If you accept the default configuration in the
filebeat.yml
config file, Filebeat loads the template automatically
after successfully connecting to Elasticsearch. If the template already exists,
it’s not overwritten unless you configure Filebeat to do so.
A connection to Elasticsearch is required to load the index template. If the output is not Elasticsearch (or Elasticsearch Service), you must load the template manually.
This page shows how to change the default template loading behavior to:
For a full list of template setup options, see Elasticsearch index template.
Load your own index template
editTo load your own index template, set the following options:
setup.template.name: "your_template_name" setup.template.fields: "path/to/fields.yml"
If the template already exists, it’s not overwritten unless you configure Filebeat to do so.
You can load templates for both data streams and indices.
Overwrite an existing index template
editDo not enable this option for more than one instance of Filebeat. If you start multiple instances at the same time, it can overload your Elasticsearch with too many template update requests.
To overwrite a template that’s already loaded into Elasticsearch, set:
setup.template.overwrite: true
Disable automatic index template loading
editYou may want to disable automatic template loading if you’re using an output other than Elasticsearch and need to load the template manually. To disable automatic template loading, set:
setup.template.enabled: false
If you disable automatic template loading, you must load the index template manually.
Load the index template manually
editTo load the index template manually, run the setup
command.
A connection to Elasticsearch is required. If another output is enabled, you need to
temporarily disable that output and enable Elasticsearch by using the -E
option.
The examples here assume that Logstash output is enabled.
You can omit the -E
flags if Elasticsearch output is already enabled.
If you are connecting to a secured Elasticsearch cluster, make sure you’ve configured credentials as described in the Quick start: installation and configuration.
If the host running Filebeat does not have direct connectivity to Elasticsearch, see Load the index template manually (alternate method).
To load the template, use the appropriate command for your system.
deb and rpm:
filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
mac:
./filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
linux:
./filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
docker:
docker run --rm docker.elastic.co/beats/filebeat:8.17.0 setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
win:
Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select Run As Administrator).
From the PowerShell prompt, change to the directory where you installed Filebeat, and run:
PS > .\filebeat.exe setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
Force Kibana to look at newest documents
editIf you’ve already used Filebeat to index data into Elasticsearch,
the index may contain old documents. After you load the index template,
you can delete the old documents from filebeat-*
to force Kibana to look
at the newest documents.
Use this command:
deb and rpm:
curl -XDELETE 'http://localhost:9200/filebeat-*'
mac:
curl -XDELETE 'http://localhost:9200/filebeat-*'
linux:
curl -XDELETE 'http://localhost:9200/filebeat-*'
win:
PS > Invoke-RestMethod -Method Delete "http://localhost:9200/filebeat-*"
This command deletes all indices that match the pattern filebeat
.
Before running this command, make sure you want to delete all indices that match
the pattern.
Load the index template manually (alternate method)
editIf the host running Filebeat does not have direct connectivity to Elasticsearch, you can export the index template to a file, move it to a machine that does have connectivity, and then install the template manually.
To export the index template, run:
deb and rpm:
filebeat export template > filebeat.template.json
mac:
./filebeat export template > filebeat.template.json
linux:
./filebeat export template > filebeat.template.json
win:
PS > .\filebeat.exe export template --es.version 8.17.0 | Out-File -Encoding UTF8 filebeat.template.json
To install the template, run:
deb and rpm:
curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_index_template/filebeat-8.17.0 -d@filebeat.template.json
mac:
curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_index_template/filebeat-8.17.0 -d@filebeat.template.json
linux:
curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_index_template/filebeat-8.17.0 -d@filebeat.template.json
win:
PS > Invoke-RestMethod -Method Put -ContentType "application/json" -InFile filebeat.template.json -Uri http://localhost:9200/_index_template/filebeat-8.17.0
Once you have loaded the index template, load the data stream as well. If you
do not load it, you have to give the publisher user manage
permission on
filebeat-8.17.0 index.
deb and rpm:
curl -XPUT http://localhost:9200/_data_stream/filebeat-8.17.0
mac:
curl -XPUT http://localhost:9200/_data_stream/filebeat-8.17.0
linux:
curl -XPUT http://localhost:9200/_data_stream/filebeat-8.17.0
win:
PS > Invoke-RestMethod -Method Put -Uri http://localhost:9200/_data_stream/filebeat-8.17.0
On this page
ElasticON events are back!
Learn about the Elastic Search AI Platform from the experts at our live events.
Register now