- Filebeat Reference: other versions:
- Filebeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade
- How Filebeat works
- Configure
- Inputs
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_json_fields
- decompress_gzip_field
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- registered_domain
- rename
- script
- timestamp
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Load balancing
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- filebeat.reference.yml
- How to guides
- Override configuration settings
- Load the Elasticsearch index template
- Change the index name
- Load Kibana dashboards
- Load ingest pipelines
- Enrich events with geoIP information
- Deduplicate data
- Parse data by using ingest node
- Use environment variables in the configuration
- Avoid YAML formatting problems
- Beats central management
- Modules
- Modules overview
- ActiveMQ module
- Apache module
- Auditd module
- AWS module
- Azure module
- Barracuda module
- Bluecoat module
- CEF module
- Check Point module
- Cisco module
- CoreDNS module
- Crowdstrike module
- Cylance module
- Elasticsearch module
- Envoyproxy Module
- F5 module
- Fortinet module
- Google Cloud module
- GSuite module
- haproxy module
- IBM MQ module
- Icinga module
- IIS module
- Imperva module
- Infoblox module
- Iptables module
- Juniper module
- Kafka module
- Kibana module
- Logstash module
- Microsoft module
- MISP module
- MongoDB module
- MSSQL module
- MySQL module
- nats module
- NetFlow module
- Netscout module
- Nginx module
- Office 365 module
- Okta module
- Osquery module
- Palo Alto Networks module
- PostgreSQL module
- RabbitMQ module
- Radware module
- Redis module
- Santa module
- Sonicwall module
- Sophos module
- Squid module
- Suricata module
- System module
- Tomcat module
- Traefik module
- Zeek (Bro) Module
- Zscaler module
- Exported fields
- ActiveMQ fields
- Apache fields
- Auditd fields
- AWS fields
- awscloudwatch fields
- Azure fields
- Barracuda Web Application Firewall fields
- Beat fields
- Blue Coat Director fields
- Decode CEF processor fields fields
- CEF fields
- Checkpoint fields
- Cisco fields
- Cloud provider metadata fields
- Coredns fields
- Crowdstrike fields
- CylanceProtect fields
- Docker fields
- ECS fields
- Elasticsearch fields
- Envoyproxy fields
- Big-IP Access Policy Manager fields
- Fortinet fields
- Google Cloud fields
- gsuite fields
- HAProxy fields
- Host fields
- ibmmq fields
- Icinga fields
- IIS fields
- Imperva SecureSphere fields
- Infoblox NIOS fields
- iptables fields
- Jolokia Discovery autodiscover provider fields
- Juniper JUNOS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- Microsoft fields
- MISP fields
- mongodb fields
- mssql fields
- MySQL fields
- NATS fields
- NetFlow fields
- Arbor Peakflow SP fields
- Nginx fields
- Office 365 fields
- Okta fields
- Osquery fields
- panw fields
- PostgreSQL fields
- Process fields
- RabbitMQ fields
- Radware DefensePro fields
- Redis fields
- s3 fields
- Google Santa fields
- Sonicwall-FW fields
- sophos fields
- Squid fields
- Suricata fields
- System fields
- Apache Tomcat fields
- Traefik fields
- Zeek fields
- Zscaler NSS fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Common problems
- Can’t read log files from network volumes
- Filebeat isn’t collecting lines from a file
- Too many open file handlers
- Registry file is too large
- Inode reuse causes Filebeat to skip lines
- Log rotation results in lost or duplicate events
- Open file handlers cause issues with Windows file rotation
- Filebeat is using too much CPU
- Dashboard in Kibana is breaking up data fields incorrectly
- Fields are not indexed or usable in Kibana visualizations
- Filebeat isn’t shipping the last line of a file
- Filebeat keeps open file handlers of deleted files for a long time
- Filebeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- Contribute to Beats
Add cloud metadata
editAdd cloud metadata
editThe add_cloud_metadata
processor enriches each event with instance metadata
from the machine’s hosting provider. At startup it will query a list of hosting
providers and cache the instance metadata.
The following cloud providers are supported:
- Amazon Web Services (AWS)
- Digital Ocean
- Google Compute Engine (GCE)
- Tencent Cloud (QCloud)
- Alibaba Cloud (ECS)
- Azure Virtual Machine
- Openstack Nova
The Alibaba Cloud and Tencent cloud providers are disabled by default, because
they require to access a remote host. The providers
setting allows users to
select a list of default providers to query.
The simple configuration below enables the processor.
processors: - add_cloud_metadata: ~
The add_cloud_metadata
processor has three optional configuration settings.
The first one is timeout
which specifies the maximum amount of time to wait
for a successful response when detecting the hosting provider. The default
timeout value is 3s
.
If a timeout occurs then no instance metadata will be added to the events. This makes it possible to enable this processor for all your deployments (in the cloud or on-premise).
The second optional setting is providers
. The providers
settings accepts a
list of cloud provider names to be used. If providers
is not configured, then
all providers that do not access a remote endpoint are enabled by default.
List of names the providers
setting supports:
- "alibaba", or "ecs" for the Alibaba Cloud provider (disabled by default).
- "azure" for Azure Virtual Machine (enabled by default).
- "digitalocean" for Digital Ocean (enabled by default).
- "aws", or "ec2" for Amazon Web Services (enabled by default).
- "gcp" for Google Copmute Enging (enabled by default).
- "openstack", or "nova" for Openstack Nova (enabled by default).
- "tencent", or "qcloud" for Tencent Cloud (disabled by default).
The third optional configuration setting is overwrite
. When overwrite
is
true
, add_cloud_metadata
overwrites existing cloud.*
fields (false
by
default).
The metadata that is added to events varies by hosting provider. Below are examples for each of the supported providers.
AWS
{ "cloud": { "account.id": "123456789012", "availability_zone": "us-east-1c", "instance.id": "i-4e123456", "machine.type": "t2.medium", "image.id": "ami-abcd1234", "provider": "aws", "region": "us-east-1" } }
Digital Ocean
{ "cloud": { "instance.id": "1234567", "provider": "digitalocean", "region": "nyc2" } }
GCP
{ "cloud": { "availability_zone": "us-east1-b", "instance.id": "1234556778987654321", "machine.type": "f1-micro", "project.id": "my-dev", "provider": "gcp" } }
Tencent Cloud
{ "cloud": { "availability_zone": "gz-azone2", "instance.id": "ins-qcloudv5", "provider": "qcloud", "region": "china-south-gz" } }
Alibaba Cloud
This metadata is only available when VPC is selected as the network type of the ECS instance.
{ "cloud": { "availability_zone": "cn-shenzhen", "instance.id": "i-wz9g2hqiikg0aliyun2b", "provider": "ecs", "region": "cn-shenzhen-a" } }
Azure Virtual Machine
{ "cloud": { "provider": "azure", "instance.id": "04ab04c3-63de-4709-a9f9-9ab8c0411d5e", "instance.name": "test-az-vm", "machine.type": "Standard_D3_v2", "region": "eastus2" } }
Openstack Nova
{ "cloud": { "instance.name": "test-998d932195.mycloud.tld", "instance.id": "i-00011a84", "availability_zone": "xxxx-az-c", "provider": "openstack", "machine.type": "m2.large" } }