- Logstash Reference: other versions:
- Logstash Introduction
- Getting Started with Logstash
- How Logstash Works
- Setting Up and Running Logstash
- Setting Up X-Pack
- Breaking changes
- Upgrading Logstash
- Configuring Logstash
- Data Resiliency
- Working with Filebeat Modules
- Transforming Data
- Deploying and Scaling Logstash
- Performance Tuning
- Monitoring Logstash
- Working with plugins
- Input plugins
- Beats input plugin
- Cloudwatch input plugin
- Couchdb_changes input plugin
- Dead_letter_queue input plugin
- Drupal_dblog input plugin
- Elasticsearch input plugin
- Eventlog output plugin
- Exec input plugin
- File input plugin
- Ganglia input plugin
- Gelf input plugin
- Gemfire input plugin
- Generator input plugin
- Github input plugin
- Google_pubsub input plugin
- Graphite input plugin
- Heartbeat input plugin
- heroku input plugin
- Http input plugin
- Http_poller input plugin
- Imap input plugin
- Irc input plugin
- Jdbc input plugin
- Jms input plugin
- Jmx input plugin
- Kafka input plugin
- Kinesis input plugin
- Log4j input plugin
- Lumberjack input plugin
- Meetup input plugin
- Pipe input plugin
- Puppet_facter input plugin
- Rabbitmq input plugin
- rackspace input plugin
- Redis input plugin
- Relp input plugin
- Rss input plugin
- S3 input plugin
- Salesforce input plugin
- Snmptrap input plugin
- Sqlite input plugin
- Sqs input plugin
- Stdin input plugin
- Stomp input plugin
- Syslog input plugin
- Tcp input plugin
- Twitter input plugin
- Udp input plugin
- Unix input plugin
- Varnishlog input plugin
- Websocket input plugin
- Wmi input plugin
- Xmpp input plugin
- Zenoss input plugin
- Zeromq input plugin
- Output plugins
- Boundary output plugin
- Circonus output plugin
- Cloudwatch output plugin
- Csv output plugin
- Datadog output plugin
- Datadog_metrics output plugin
- Elasticsearch output plugin
- Email output plugin
- Exec output plugin
- File output plugin
- Ganglia output plugin
- Gelf output plugin
- Google_bigquery output plugin
- Google_cloud_storage output plugin
- Graphite output plugin
- Graphtastic output plugin
- Hipchat output plugin
- Http output plugin
- Influxdb output plugin
- Irc output plugin
- Jira output plugin
- Jms output plugin
- Juggernaut output plugin
- Kafka output plugin
- Librato output plugin
- Loggly output plugin
- Lumberjack output plugin
- Metriccatcher output plugin
- Mongodb output plugin
- Nagios output plugin
- Nagios_nsca output plugin
- Newrelic output plugin
- Opentsdb output plugin
- Pagerduty output plugin
- Pipe output plugin
- Rabbitmq output plugin
- Rackspace output plugin
- Redis output plugin
- Redmine output plugin
- Riak output plugin
- Riemann output plugin
- S3 output plugin
- Sns output plugin
- Solr_http output plugin
- Sqs output plugin
- Statsd output plugin
- Stdout output plugin
- Stomp output plugin
- Syslog output plugin
- Tcp output plugin
- Udp output plugin
- Webhdfs output plugin
- Websocket output plugin
- Xmpp output plugin
- Zabbix output plugin
- Zeromq output plugin
- Filter plugins
- Aggregate filter plugin
- Alter filter plugin
- Anonymize filter plugin
- Cidr filter plugin
- Cipher filter plugin
- Clone filter plugin
- Collate filter plugin
- Csv filter plugin
- Date filter plugin
- De_dot filter plugin
- Dissect filter plugin
- Dns filter plugin
- Drop filter plugin
- Elapsed filter plugin
- Elasticsearch filter plugin
- Environment filter plugin
- Extractnumbers filter plugin
- Fingerprint filter plugin
- Geoip filter plugin
- Grok filter plugin
- I18n filter plugin
- Jdbc_streaming filter plugin
- Json filter plugin
- Json_encode filter plugin
- Kv filter plugin
- Metaevent filter plugin
- Metricize filter plugin
- Metrics filter plugin
- Mutate filter plugin
- Oui filter plugin
- Prune filter plugin
- Punct filter plugin
- Range filter plugin
- Ruby filter plugin
- Sleep filter plugin
- Split filter plugin
- Syslog_pri filter plugin
- Throttle filter plugin
- Tld filter plugin
- Translate filter plugin
- Truncate filter plugin
- Urldecode filter plugin
- Useragent filter plugin
- Uuid filter plugin
- Xml filter plugin
- Yaml filter plugin
- Zeromq filter plugin
- Codec plugins
- Avro codec plugin
- Cef codec plugin
- Cloudfront codec plugin
- Cloudtrail codec plugin
- Collectd codec plugin
- Compress_spooler codec plugin
- Dots codec plugin
- Edn codec plugin
- Edn_lines codec plugin
- Es_bulk codec plugin
- Fluent codec plugin
- Graphite codec plugin
- Gzip_lines codec plugin
- Json codec plugin
- Json_lines codec plugin
- Line codec plugin
- Msgpack codec plugin
- Multiline codec plugin
- Netflow codec plugin
- Nmap codec plugin
- Oldlogstashjson codec plugin
- Plain codec plugin
- Protobuf codec plugin
- Rubydebug codec plugin
- Contributing to Logstash
- How to write a Logstash input plugin
- How to write a Logstash input plugin
- How to write a Logstash codec plugin
- How to write a Logstash filter plugin
- Contributing a Patch to a Logstash Plugin
- Logstash Plugins Community Maintainer Guide
- Submitting your plugin to RubyGems.org and the logstash-plugins repository
- Glossary of Terms
- Release Notes
Running Logstash on Docker
editRunning Logstash on Docker
editDocker images for Logstash are available from the Elastic Docker registry.
The base image is centos:7 and the source code can be found on GitHub.
The images are shipped with X-Pack installed.
Pulling the image
editObtaining Logstash for Docker is as simple as issuing a docker
pull
command against the Elastic Docker registry.
The Docker image for Logstash 5.5.3 can be retrieved with the following command:
docker pull docker.elastic.co/logstash/logstash:5.5.3
Configuring Logstash for Docker
editLogstash differentiates between two types of configuration: Settings and Pipeline Configuration.
Pipeline Configuration
editIt is essential to place your pipeline configuration where it can be
found by Logstash. By default, the container will look in
/usr/share/logstash/pipeline/
for pipeline configuration files.
In this example we use a bind-mounted volume to provide the
configuration via the docker run
command:
docker run --rm -it -v ~/pipeline/:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:5.5.3
Every file in the host directory ~/pipeline/
will then be parsed
by Logstash as pipeline configuration.
If you don’t provide configuration to Logstash, it will run with a
minimal config that listens for messages from the
Beats input plugin and echoes any that are
received to stdout
. In this case, the startup logs will be similar
to the following:
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties. [2016-10-26T05:11:34,992][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"} [2016-10-26T05:11:35,068][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500} [2016-10-26T05:11:35,078][INFO ][org.logstash.beats.Server] Starting server on port: 5044 [2016-10-26T05:11:35,078][INFO ][logstash.pipeline ] Pipeline main started [2016-10-26T05:11:35,105][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
This is the default configuration for the image, defined in
/usr/share/logstash/pipeline/logstash.conf
. If this is the
behaviour that you are observing, ensure that your pipeline
configuration is being picked up correctly, and that you are replacing
either logstash.conf
or the entire pipeline
directory.
Settings
editThe image provides several methods for configuring settings. The conventional
approach is to provide a custom logstash.yml
file, but it’s
also possible to use environment variables to define settings.
Bind-mounted settings files
editSettings files can also be provided through bind-mounts. Logstash
expects to find them at /usr/share/logstash/config/
.
It’s possible to provide an entire directory containing all needed files:
docker run --rm -it -v ~/settings/:/usr/share/logstash/config/ docker.elastic.co/logstash/logstash:5.5.3
Alternatively, a single file can be mounted:
docker run --rm -it -v ~/settings/logstash.yml:/usr/share/logstash/config/logstash.yml docker.elastic.co/logstash/logstash:5.5.3
Bind-mounted configuration files will retain the same permissions and
ownership within the container that they have on the host system. Be sure
to set permissions such that the files will be readable and, ideally, not
writeable by the container’s logstash
user (UID 1000).
Custom Images
editBind-mounted configuration is not the only option, naturally. If you
prefer the Immutable Infrastructure approach, you can prepare a
custom image containing your configuration by using a Dockerfile
like this one:
FROM docker.elastic.co/logstash/logstash:5.5.3 RUN rm -f /usr/share/logstash/pipeline/logstash.conf ADD pipeline/ /usr/share/logstash/pipeline/ ADD config/ /usr/share/logstash/config/
Be sure to replace or delete logstash.conf
in your custom image, so
that you don’t retain the example config from the base image.
Environment variable configuration
editUnder Docker, Logstash settings can be configured via environment
variables. When the container starts, a helper process checks the environment
for variables that can be mapped to Logstash settings. Settings that are found
in the environment are merged into logstash.yml
as the container starts up.
For compatibility with container orchestration systems, these environment variables are written in all capitals, with underscores as word separators
Some example translations are shown here:
Table 1. Example Docker Environment Variables
Environment Variable |
Logstash Setting |
|
|
|
|
|
|
In general, any setting listed in the settings documentation can be configured with this technique.
Defining settings with environment variables causes logstash.yml
to
be modified in place. This behaviour is likely undesirable if logstash.yml
was
bind-mounted from the host system. Thus, it is not reccomended to
combine the bind-mount technique with the environment variable technique. It
is best to choose a single method for defining Logstash settings.
Logging Configuration
editUnder Docker, Logstash logs go to standard output by default. To
change this behaviour, use any of the techniques above to replace the
file at /usr/share/logstash/config/log4j2.properties
.
On this page