APM agent ECS reformatting with Filebeat

edit

APM agent ECS reformatting with Filebeat

edit

Elastic APM agents can automatically reformat application logs to Elastic Common Schema (ECS) format without needing to add an ECS logger dependency or modify the application.

Requirements

  • The Elastic APM agent for your programming language
  • Filebeat configured to monitor and capture application logs

Pros

All the benefits of using ECS logging, without having to modify the application or its configuration:

  • Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
  • Decently human-readable JSON structure
  • APM log correlation

Cons

  • Requires an Elastic APM agent
  • Not all APM agents support this feature

Supported APM agents/languages

Step 1: Enable APM agent reformatting

edit

Enable APM agent ECS reformatting in the supported APM agent:

See the Ruby or Java agent docs.

Step 2: Set up Filebeat

edit
  1. Follow the Filebeat quick start to learn how to install Filebeat and connect to the Elastic Stack.
  2. Add the following configuration to your filebeat.yaml file to start collecting log data.

    filebeat.yaml.

    filebeat.inputs:
    - type: filestream 
      paths: /path/to/logs.json
      parsers:
        - ndjson:
          overwrite_keys: true 
          add_error_key: true 
          expand_keys: true 
      fields:
        service.name: your_service_name 
    
    processors: 
      - add_host_metadata: ~
      - add_cloud_metadata: ~
      - add_docker_metadata: ~
      - add_kubernetes_metadata: ~

    Use the filestream input to read lines from active log files.

    Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.

    Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.

    Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.

    The service.name of the service you’re collecting logs from. This is required for Log correlation.

    Processors enhance your data. See processors to learn more.

Step 3: View your logs in Kibana

edit

Use the APM or Logs UI to search, filter, and visualize your logs.