IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

› › ›

APM agent ECS reformatting

edit

APM agent ECS reformatting

edit

Elastic APM agents can automatically reformat application logs to Elastic Common Schema (ECS) format without needing to add an ECS logger dependency or modify the application.

Requirements

The Elastic APM agent for your programming language
Filebeat configured to monitor and capture application logs

Pros

All the benefits of using ECS logging, without having to modify the application or its configuration:

Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
Decently human-readable JSON structure
APM log correlation

Cons

Requires an Elastic APM agent
Not all APM agents support this feature

Supported APM agents/languages

Ruby
Python
Java

Step 1: Enable APM agent reformatting

edit

Enable APM agent ECS reformatting in the supported APM agent:

See the Ruby or Java agent docs.

Step 2: Set up Filebeat

edit

Follow the Filebeat quick start to learn how to install Filebeat and connect to the Elastic Stack.

Add the following configuration to your filebeat.yaml file to start collecting log data.

filebeat.yaml.

filebeat.inputs:
- type: filestream 
  paths: /path/to/logs.json
  parsers:
    - ndjson:
        overwrite_keys: true 
        add_error_key: true 
        expand_keys: true 
  fields:
    service.name: your_service_name 
    service.version: your_service_version 
    service.environment: your_service_environment 

processors: 
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

	Use the filestream input to read lines from active log files.
	Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.
	Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
	Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.
	The `service.name` (required), `service.version` (optional) and `service.environment` (optional) of the service you’re collecting logs from, used for Log correlation.
	Processors enhance your data. See processors to learn more.

Make sure your application logs to stdout/stderr.
Follow the Run Filebeat on Kubernetes guide.
Enable hints-based autodiscover (uncomment the corresponding section in filebeat-kubernetes.yaml).

Add these annotations to your pods that log using ECS-compatible JSON. This will make sure the logs are parsed appropriately.

annotations:
  co.elastic.logs/json.overwrite_keys: true 
  co.elastic.logs/json.add_error_key: true 
  co.elastic.logs/json.expand_keys: true

	Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.
	Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
	Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.

Make sure your application logs to stdout/stderr.
Follow the Run Filebeat on Docker guide.
Enable hints-based autodiscover.

Add these labels to your containers that log using ECS-compatible JSON. This will make sure the logs are parsed appropriately.

docker-compose.yml.

labels:
  co.elastic.logs/json.overwrite_keys: true 
  co.elastic.logs/json.add_error_key: true 
  co.elastic.logs/json.expand_keys: true

	Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.
	Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
	Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.

Step 3: View your logs in Kibana

edit

Use the APM or Logs UI to search, filter, and visualize your logs.

« ECS logging APM agent log sending »