- Filebeat Reference: other versions:
- Filebeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade
- How Filebeat works
- Configure
- Inputs
- Multiline messages
- AWS CloudWatch
- AWS S3
- Azure Event Hub
- Azure Blob Storage
- Benchmark
- CEL
- Cloud Foundry
- CometD
- Container
- Entity Analytics
- ETW
- filestream
- GCP Pub/Sub
- Google Cloud Storage
- HTTP Endpoint
- HTTP JSON
- journald
- Kafka
- Log
- MQTT
- NetFlow
- Office 365 Management Activity API
- Redis
- Salesforce
- Stdin
- Streaming
- Syslog
- TCP
- UDP
- Unix
- winlog
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- append
- cache
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- parse_aws_vpc_flow_log
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- timestamp
- translate_ldap_attribute
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- Feature flags
- filebeat.reference.yml
- Inputs
- How to guides
- Override configuration settings
- Load the Elasticsearch index template
- Change the index name
- Load Kibana dashboards
- Load ingest pipelines
- Enrich events with geoIP information
- Deduplicate data
- Parse data using an ingest pipeline
- Use environment variables in the configuration
- Avoid YAML formatting problems
- Migrate
log
input configurations tofilestream
- Migrating from a Deprecated Filebeat Module
- Modules
- Modules overview
- ActiveMQ module
- Apache module
- Auditd module
- AWS module
- AWS Fargate module
- Azure module
- CEF module
- Check Point module
- Cisco module
- CoreDNS module
- CrowdStrike module
- Cyberark PAS module
- Elasticsearch module
- Envoyproxy Module
- Fortinet module
- Google Cloud module
- Google Workspace module
- HAproxy module
- IBM MQ module
- Icinga module
- IIS module
- Iptables module
- Juniper module
- Kafka module
- Kibana module
- Logstash module
- Microsoft module
- MISP module
- MongoDB module
- MSSQL module
- MySQL module
- MySQL Enterprise module
- NATS module
- NetFlow module
- Nginx module
- Office 365 module
- Okta module
- Oracle module
- Osquery module
- Palo Alto Networks module
- pensando module
- PostgreSQL module
- RabbitMQ module
- Redis module
- Salesforce module
- Santa module
- Snyk module
- Sophos module
- Suricata module
- System module
- Threat Intel module
- Traefik module
- Zeek (Bro) Module
- ZooKeeper module
- Zoom module
- Exported fields
- ActiveMQ fields
- Apache fields
- Auditd fields
- AWS fields
- AWS CloudWatch fields
- AWS Fargate fields
- Azure fields
- Beat fields
- Decode CEF processor fields fields
- CEF fields
- Checkpoint fields
- Cisco fields
- Cloud provider metadata fields
- Coredns fields
- Crowdstrike fields
- CyberArk PAS fields
- Docker fields
- ECS fields
- Elasticsearch fields
- Envoyproxy fields
- Fortinet fields
- Google Cloud Platform (GCP) fields
- google_workspace fields
- HAProxy fields
- Host fields
- ibmmq fields
- Icinga fields
- IIS fields
- iptables fields
- Jolokia Discovery autodiscover provider fields
- Juniper JUNOS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- Lumberjack fields
- Microsoft fields
- MISP fields
- mongodb fields
- mssql fields
- MySQL fields
- MySQL Enterprise fields
- NATS fields
- NetFlow fields
- Nginx fields
- Office 365 fields
- Okta fields
- Oracle fields
- Osquery fields
- panw fields
- Pensando fields
- PostgreSQL fields
- Process fields
- RabbitMQ fields
- Redis fields
- s3 fields
- Salesforce fields
- Google Santa fields
- Snyk fields
- sophos fields
- Suricata fields
- System fields
- threatintel fields
- Traefik fields
- Windows ETW fields
- Zeek fields
- ZooKeeper fields
- Zoom fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Understand logged metrics
- Common problems
- Error extracting container id while using Kubernetes metadata
- Can’t read log files from network volumes
- Filebeat isn’t collecting lines from a file
- Too many open file handlers
- Registry file is too large
- Inode reuse causes Filebeat to skip lines
- Log rotation results in lost or duplicate events
- Open file handlers cause issues with Windows file rotation
- Filebeat is using too much CPU
- Dashboard in Kibana is breaking up data fields incorrectly
- Fields are not indexed or usable in Kibana visualizations
- Filebeat isn’t shipping the last line of a file
- Filebeat keeps open file handlers of deleted files for a long time
- Filebeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Hints based autodiscover
editHints based autodiscover
editFilebeat supports autodiscover based on hints from the provider. The hints system looks for
hints in Kubernetes Pod annotations or Docker labels that have the prefix co.elastic.logs
. As soon as
the container starts, Filebeat will check if it contains any hints and launch the proper config for
it. Hints tell Filebeat how to get logs for the given container. By default logs will be retrieved
from the container using the filestream
input. You can use hints to modify this behavior. This is the full
list of supported hints:
co.elastic.logs/enabled
editFilebeat gets logs from all containers by default, you can set this hint to false
to ignore
the output of the container. Filebeat won’t read or send logs from it. If default config is
disabled, you can use this annotation to enable log retrieval only for containers with this
set to true
. If you are aiming to use this with Kubernetes, have in mind that annotation
values can only be of string type so you will need to explicitly define this as "true"
or "false"
accordingly.
co.elastic.logs/multiline.*
editMultiline settings. See Multiline messages for a full list of all supported options.
co.elastic.logs/json.*
editJSON settings. In case of filestream
input(default) see ndjson
for a full list of all supported options.
In case of container
or log
input see json
for a full list of all supported options.
For example, the following hints with json options:
co.elastic.logs/json.message_key: "log" co.elastic.logs/json.add_error_key: "true"
will lead to the following input configuration:
-
filestream
parsers: - ndjson: message_key: "log" add_error_key: "true"
-
log
json.message_key: "log" json.add_error_key: "true"
keys_under_root
json option of log
input is replaced with target
option in filestream input. Read the documentation(ndjson
) on how to use it correctly.
co.elastic.logs/include_lines
editA list of regular expressions to match the lines that you want Filebeat to include. See Inputs for more info.
co.elastic.logs/exclude_lines
editA list of regular expressions to match the lines that you want Filebeat to exclude. See Inputs for more info.
co.elastic.logs/module
editInstead of using raw docker
input, specifies the module to use to parse logs from the container. See
Modules for the list of supported modules.
co.elastic.logs/fileset
editWhen module is configured, map container logs to module filesets. You can either configure a single fileset like this:
co.elastic.logs/fileset: access
Or configure a fileset per stream in the container (stdout and stderr):
co.elastic.logs/fileset.stdout: access co.elastic.logs/fileset.stderr: error
co.elastic.logs/raw
editWhen an entire input/module configuration needs to be completely set the raw
hint can be used. You can provide a
stringified JSON of the input configuration. raw
overrides every other hint and can be used to create both a single or
a list of configurations.
co.elastic.logs/raw: "[{\"containers\":{\"ids\":[\"${data.container.id}\"]},\"multiline\":{\"negate\":\"true\",\"pattern\":\"^test\"},\"type\":\"docker\"}]"
co.elastic.logs/processors
editDefine a processor to be added to the Filebeat input/module configuration. See Processors for the list of supported processors.
If processors configuration uses list data structure, object fields must be enumerated.
For example, hints for the rename
processor configuration below
processors: - rename: fields: - from: "a.g" to: "e.d" fail_on_error: true
will look like:
co.elastic.logs/processors.rename.fields.0.from: "a.g" co.elastic.logs/processors.rename.fields.1.to: "e.d" co.elastic.logs/processors.rename.fail_on_error: 'true'
If processors configuration uses map data structure, enumeration is not needed. For example, the equivalent to the add_fields
configuration below
processors: - add_fields: target: project fields: name: myproject
is
co.elastic.logs/processors.1.add_fields.target: "project" co.elastic.logs/processors.1.add_fields.fields.name: "myproject"
In order to provide ordering of the processor definition, numbers can be provided. If not, the hints builder will do arbitrary ordering:
co.elastic.logs/processors.1.dissect.tokenizer: "%{key1} %{key2}" co.elastic.logs/processors.dissect.tokenizer: "%{key2} %{key1}"
In the above sample the processor definition tagged with 1
would be executed first.
co.elastic.logs/pipeline
editDefine an ingest pipeline ID to be added to the Filebeat input/module configuration.
co.elastic.logs/pipeline: custom-pipeline
When hints are used along with templates, then hints will be evaluated only in case there is no template’s condition that resolves to true. For example:
filebeat.autodiscover.providers: - type: docker hints.enabled: true hints.default_config: type: container paths: - /var/lib/docker/containers/${data.container.id}/*.log templates: - condition: equals: docker.container.labels.type: "pipeline" config: - type: container paths: - "/var/lib/docker/containers/${data.docker.container.id}/*.log" pipeline: my-pipeline
In this example first the condition docker.container.labels.type: "pipeline"
is evaluated
and if not matched the hints will be processed and if there is again no valid config
the hints.default_config
will be used.
Kubernetes
editKubernetes autodiscover provider supports hints in Pod annotations. To enable it just set hints.enabled
:
filebeat.autodiscover: providers: - type: kubernetes hints.enabled: true
You can configure the default config that will be launched when a new container is seen, like this:
filebeat.autodiscover: providers: - type: kubernetes hints.enabled: true hints.default_config: type: container paths: - /var/log/containers/*-${data.container.id}.log # CRI path
You can also disable default settings entirely, so only Pods annotated like co.elastic.logs/enabled: true
will be retrieved:
filebeat.autodiscover: providers: - type: kubernetes hints.enabled: true hints.default_config.enabled: false
You can annotate Kubernetes Pods with useful info to spin up Filebeat inputs or modules:
annotations: co.elastic.logs/multiline.pattern: '^\[' co.elastic.logs/multiline.negate: true co.elastic.logs/multiline.match: after
Multiple containers
editWhen a pod has multiple containers, the settings are shared unless you put the container name in the
hint. For example, these hints configure multiline settings for all containers in the pod, but set a
specific exclude_lines
hint for the container called sidecar
.
annotations: co.elastic.logs/multiline.pattern: '^\[' co.elastic.logs/multiline.negate: true co.elastic.logs/multiline.match: after co.elastic.logs.sidecar/exclude_lines: '^DBG'
Multiple sets of hints
editWhen a container needs multiple inputs to be defined on it, sets of annotations can be provided with numeric prefixes. If there are hints that don’t have a numeric prefix then they get grouped together into a single configuration.
annotations: co.elastic.logs/exclude_lines: '^DBG' co.elastic.logs/1.include_lines: '^DBG' co.elastic.logs/1.processors.dissect.tokenizer: "%{key2} %{key1}"
The above configuration would generate two input configurations. The first input handles only debug logs and passes it through a dissect tokenizer. The second input handles everything but debug logs.
Namespace Defaults
editHints can be configured on the Namespace’s annotations as defaults to use when Pod level annotations are missing.
The resultant hints are a combination of Pod annotations and Namespace annotations with the Pod’s taking precedence. To
enable Namespace defaults configure the add_resource_metadata
for Namespace objects as follows:
filebeat.autodiscover: providers: - type: kubernetes hints.enabled: true add_resource_metadata: namespace: include_annotations: ["nsannotation1"]
Docker
editDocker autodiscover provider supports hints in labels. To enable it just set hints.enabled
:
filebeat.autodiscover: providers: - type: docker hints.enabled: true
You can configure the default config that will be launched when a new container is seen, like this:
filebeat.autodiscover: providers: - type: docker hints.enabled: true hints.default_config: type: container paths: - /var/log/containers/*-${data.container.id}.log # CRI path
You can also disable default settings entirely, so only containers labeled with co.elastic.logs/enabled: true
will be retrieved:
filebeat.autodiscover: providers: - type: docker hints.enabled: true hints.default_config.enabled: false
You can label Docker containers with useful info to spin up Filebeat inputs, for example:
co.elastic.logs/module: nginx co.elastic.logs/fileset.stdout: access co.elastic.logs/fileset.stderr: error
The above labels configure Filebeat to use the Nginx module to harvest logs for this container. Access logs will be retrieved from stdout stream, and error logs from stderr.
You can label Docker containers with useful info to decode logs structured as JSON messages, for example:
co.elastic.logs/json.keys_under_root: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.message_key: log
Nomad
editNomad autodiscover provider supports hints using the
meta
stanza. To
enable it just set hints.enabled
:
filebeat.autodiscover: providers: - type: nomad hints.enabled: true
You can configure the default config that will be launched when a new job is seen, like this:
filebeat.autodiscover: providers: - type: nomad hints.enabled: true hints.default_config: type: log paths: - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
You can also disable the default config such that only logs from jobs explicitly
annotated with "co.elastic.logs/enabled" = "true"
will be collected:
filebeat.autodiscover: providers: - type: nomad hints.enabled: true hints.default_config: enabled: false type: log paths: - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
You can annotate Nomad Jobs using the meta
stanza with useful info to spin up
Filebeat inputs or modules:
meta { "co.elastic.logs/enabled" = "true" "co.elastic.logs/multiline.pattern" = "^\\[" "co.elastic.logs/multiline.negate" = "true" "co.elastic.logs/multiline.match" = "after" }
If you are using autodiscover then in most cases you will want to use the
add_nomad_metadata
processor to enrich events with
Nomad metadata. This example configures {Filebeat} to connect to the local
Nomad agent over HTTPS and adds the Nomad allocation ID to all events from the
input. Later in the pipeline the add_nomad_metadata
processor will use that ID
to enrich the event.
filebeat.autodiscover: providers: - type: nomad address: https://localhost:4646 hints.enabled: true hints.default_config: enabled: false type: log paths: - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.* processors: - add_fields: target: nomad fields: allocation.id: ${data.nomad.allocation.id} processors: - add_nomad_metadata: when.has_fields.fields: [nomad.allocation.id] address: https://localhost:4646 default_indexers.enabled: false default_matchers.enabled: false indexers: - allocation_uuid: matchers: - fields: lookup_fields: - 'nomad.allocation.id'
The default config is disabled meaning any task without the
|
|
The |
|
The |
On this page
co.elastic.logs/enabled
co.elastic.logs/multiline.*
co.elastic.logs/json.*
co.elastic.logs/include_lines
co.elastic.logs/exclude_lines
co.elastic.logs/module
co.elastic.logs/fileset
co.elastic.logs/raw
co.elastic.logs/processors
co.elastic.logs/pipeline
- Kubernetes
- Multiple containers
- Multiple sets of hints
- Namespace Defaults
- Docker
- Nomad