Installation
editInstallation
edit$ python -m pip install ecs-logging
Getting Started
editecs-logging-python
has formatters for the standard library
logging
module
and the structlog
package.
Standard Library logging
Module
editimport logging import ecs_logging # Get the Logger logger = logging.getLogger("app") logger.setLevel(logging.DEBUG) # Add an ECS formatter to the Handler handler = logging.StreamHandler() handler.setFormatter(ecs_logging.StdlibFormatter()) logger.addHandler(handler) # Emit a log! logger.debug("Example message!", extra={"http.request.method": "get"})
{ "@timestamp": "2020-03-20T18:11:37.895Z", "log.level": "debug", "message": "Example message!", "ecs": { "version": "1.6.0" }, "http": { "request": { "method": "get" } }, "log": { "logger": "app", "origin": { "file": { "line": 14, "name": "test.py" }, "function": "func" }, "original": "Example message!" } }
Excluding Fields
editYou can exclude fields from being collected by using the exclude_fields
option
in the StdlibFormatter
constructor:
from ecs_logging import StdlibFormatter formatter = StdlibFormatter( exclude_fields=[ # You can specify individual fields to ignore: "log.original", # or you can also use prefixes to ignore # whole categories of fields: "process", "log.origin", ] )
Limiting Stack Traces
editThe StdlibLogger
automatically gathers exc_info
into ECS error.*
fields.
If you’d like to control the number of stack frames that are included
in error.stack_trace
you can use the stack_trace_limit
parameter
(by default all frames are collected):
from ecs_logging import StdlibFormatter formatter = StdlibFormatter( # Only collects 3 stack frames stack_trace_limit=3, ) formatter = StdlibFormatter( # Disable stack trace collection stack_trace_limit=0, )
Structlog Example
editNote that the structlog processor should be the last processor in the list, as it handles the conversion to JSON as well as the ECS field enrichment.
import structlog import ecs_logging # Configure Structlog structlog.configure( processors=[ecs_logging.StructlogFormatter()], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(), ) # Get the Logger logger = structlog.get_logger("app") # Add additional context logger = logger.bind(**{ "http": { "version": "2", "request": { "method": "get", "bytes": 1337, }, }, "url": { "domain": "example.com", "path": "/", "port": 443, "scheme": "https", "registered_domain": "example.com", "top_level_domain": "com", "original": "https://example.com", } }) # Emit a log! logger.debug("Example message!")
{ "@timestamp": "2020-03-26T13:08:11.728Z", "ecs": { "version": "1.6.0" }, "http": { "request": { "bytes": 1337, "method": "get" }, "version": "2" }, "log": { "level": "debug" }, "message": "Example message!", "url": { "domain": "example.com", "original": "https://example.com", "path": "/", "port": 443, "registered_domain": "example.com", "scheme": "https", "top_level_domain": "com" } }
Elastic APM Log Correlation
editecs-logging-python
supports automatically collecting ECS tracing fields
from the Elastic APM Python agent in order to
correlate logs to spans, transactions and traces in Elastic APM.
You can also quickly turn on ECS-formatted logs in your python app by setting
LOG_ECS_REFORMATTING=override
in the Elastic APM Python agent.
Install Filebeat
editThe best way to collect the logs once they are ECS-formatted is with Filebeat:
- Follow the Filebeat quick start
-
Add the following configuration to your
filebeat.yaml
file.
For Filebeat 7.16+
filebeat.yaml.
filebeat.inputs: - type: filestream paths: /path/to/logs.json parsers: - ndjson: overwrite_keys: true add_error_key: true expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~
Use the filestream input to read lines from active log files. |
|
Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts. |
|
Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors. |
|
Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure. |
|
Processors enhance your data. See processors to learn more. |
For Filebeat < 7.16
filebeat.yaml.
filebeat.inputs: - type: log paths: /path/to/logs.json json.keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~
- Make sure your application logs to stdout/stderr.
- Follow the Run Filebeat on Kubernetes guide.
-
Enable hints-based autodiscover (uncomment the corresponding section in
filebeat-kubernetes.yaml
). - Add these annotations to your pods that log using ECS loggers. This will make sure the logs are parsed appropriately.
annotations: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true
Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts. |
|
Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors. |
|
Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure. |
- Make sure your application logs to stdout/stderr.
- Follow the Run Filebeat on Docker guide.
- Enable hints-based autodiscover.
- Add these labels to your containers that log using ECS loggers. This will make sure the logs are parsed appropriately.
docker-compose.yml.
labels: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true
Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts. |
|
Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors. |
|
Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure. |
For more information, see the Filebeat reference.