Stream application logs

Application logs provide valuable insight into events that have occurred within your services and applications.

The format of your logs (structured or plaintext) influences your log ingestion strategy.

Logs are typically produced as either plaintext or structured. Plaintext logs contain only text and have no special formatting, for example:

2019-08-06T12:09:12.375Z INFO:spring-petclinic: Tomcat started on port(s): 8080 (http) with context path, org.springframework.boot.web.embedded.tomcat.TomcatWebServer
2019-08-06T12:09:12.379Z INFO:spring-petclinic: Started PetClinicApplication in 7.095 seconds (JVM running for 9.082), org.springframework.samples.petclinic.PetClinicApplication
2019-08-06T14:08:40.199Z DEBUG:spring-petclinic: init find form, org.springframework.samples.petclinic.owner.OwnerController

Structured logs follow a predefined, repeatable pattern or structure. This structure is applied at write time — preventing the need for parsing at ingest time. The Elastic Common Schema (ECS) defines a common set of fields to use when structuring logs. This structure allows logs to be easily ingested, and provides the ability to correlate, search, and aggregate on individual fields within your logs.

For example, the previous example logs might look like this when structured with ECS-compatible JSON:

{"@timestamp":"2019-08-06T12:09:12.375Z", "log.level": "INFO", "message":"Tomcat started on port(s): 8080 (http) with context path ''", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer"}
{"@timestamp":"2019-08-06T12:09:12.379Z", "log.level": "INFO", "message":"Started PetClinicApplication in 7.095 seconds (JVM running for 9.082)", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.samples.petclinic.PetClinicApplication"}
{"@timestamp":"2019-08-06T14:08:40.199Z", "log.level":"DEBUG", "message":"init find form", "service.name":"spring-petclinic","process.thread.name":"http-nio-8080-exec-8","log.logger":"org.springframework.samples.petclinic.owner.OwnerController","transaction.id":"28b7fb8d5aba51f1","trace.id":"2869b25b5469590610fea49ac04af7da"}

There are several ways to ingest application logs into your project. Your specific situation helps determine the method that’s right for you.

With Filebeat or Elastic Agent, you can ingest plaintext logs, including existing logs, from any programming language or framework without modifying your application or its configuration.

For plaintext logs to be useful, you need to use Filebeat or Elastic Agent to parse the log data.

Learn more in Plaintext logs

Logs formatted in ECS don’t require manual parsing and the configuration can be reused across applications. They also include log correlation. You can format your logs in ECS by using ECS logging plugins or APM agent ECS reformatting.

Add ECS logging plugins to your logging libraries to format your logs into ECS-compatible JSON that doesn’t require parsing.

To use ECS logging, you need to modify your application and its log configuration.

Learn more in ECS formatted logs

Some Elastic APM agents can automatically reformat application logs to ECS format without adding an ECS logger dependency or modifying the application.

This feature is supported for the following APM agents:

Learn more in ECS formatted logs

Automatically capture and send logs directly to the managed intake service using the APM agent without using Filebeat or Elastic Agent.

Log sending is supported in the Java APM agent.

Learn more in APM agent log sending

Correlate your application logs with trace events to:

Learn more about log correlation in the agent-specific ingestion guides: