- Functionbeat Reference:
- Functionbeat overview
- Quick start: installation and configuration
- Set up and deploy
- Configure
- AWS functions
- General settings
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- append
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- rate_limit
- registered_domain
- rename
- replace
- syslog
- translate_sid
- truncate_fields
- urldecode
- Internal queue
- Logging
- Regular expression support
- Instrumentation
- functionbeat.reference.yml
- How to guides
- Exported fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Understand logged metrics
- Common problems
- Deployment to AWS fails with "failed to create the stack"
- Deployment to AWS fails with "resource limit exceeded"
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- High RSS memory usage due to MADV settings
Functionbeat overview
editFunctionbeat overview
editBeginning with version 8.10.4 the Functionbeat documentation is no longer being updated. We recommend instead to use Elastic Serverless Forwarder to ships logs from your AWS environment to Elastic.
Functionbeat is an Elastic Beat that you deploy as a function in your serverless environment to collect data from cloud services and ship it to the Elastic Stack.
Version 8.10.4 supports deploying Functionbeat as an AWS Lambda service. It responds to triggers defined for the following event sources:
Functionbeat is an Elastic Beat. It’s
based on the libbeat framework. For more information, see the
Beats Platform Reference.
The following sections explore some common use cases for Functionbeat:
Want to ship logs from Google Cloud? Use our Google Cloud Dataflow templates to ship Google Pub/Sub and Google Cloud Storage logs directly from the Google Cloud Console. To learn more, refer to GCP Dataflow templates.
Monitor cloud deployments
editYou can deploy Functionbeat on your serverless environment to collect logs and metrics generated by cloud services and stream the data to the Elastic Stack for centralized analytics.
Monitor AWS services with CloudWatch logs
editYou can deploy Functionbeat as a Lambda function on AWS to receive events from a Cloudwatch Log group, extract and structure the relevant fields, then stream the events to Elasticsearch.
The processing pipeline for this use case typically looks like this:
- Functionbeat runs as a Lambda function on AWS and reads the data stream from a Cloudwatch Log group.
- Beats processors, such as dissect and drop_fields, filter and structure the events.
- Optional ingest pipelines in Elasticsearch further enhance the data.
- The structured events are indexed into an Elasticsearch cluster.
Perform event-driven processing
editYou can use Functionbeat to implement event-driven processing workflows with cloud messaging queues and the Elastic Stack. Functionbeat responds to event triggers from AWS Kinesis and SQS.
Analyze application data from SQS
editFor applications that send JSON-encoded events to an SQS queue, Functionbeat can listen for, ingest, and decode JSON events prior to shipping them to Elasticsearch, where you can analyze the streaming data.
The processing pipeline for this use case typically looks like this:
- Functionbeat runs as a serverless shipper and listens to an SQS queue for application events.
- The Beats decode_json_fields processor decodes JSON strings and replaces them with valid JSON objects.
- Optional ingest pipelines in Elasticsearch further enhance the data.
- The events are indexed into an Elasticsearch cluster.
On this page