- Logstash Reference: other versions:
- Logstash Introduction
- Getting Started with Logstash
- Breaking Changes
- Upgrading Logstash
- Configuring Logstash
- Performance Troubleshooting Guide
- Working with plugins
- Input plugins
- beats
- couchdb_changes
- drupal_dblog
- elasticsearch
- exec
- eventlog
- file
- ganglia
- gelf
- generator
- graphite
- github
- heartbeat
- heroku
- http
- http_poller
- irc
- imap
- jdbc
- jmx
- kafka
- log4j
- lumberjack
- meetup
- pipe
- puppet_facter
- relp
- rss
- rackspace
- rabbitmq
- redis
- salesforce
- snmptrap
- stdin
- sqlite
- s3
- sqs
- stomp
- syslog
- tcp
- unix
- udp
- varnishlog
- wmi
- websocket
- xmpp
- zenoss
- zeromq
- Output plugins
- boundary
- circonus
- csv
- cloudwatch
- datadog
- datadog_metrics
- elasticsearch
- elasticsearch_java
- exec
- file
- google_bigquery
- google_cloud_storage
- ganglia
- gelf
- graphtastic
- graphite
- hipchat
- http
- irc
- influxdb
- juggernaut
- jira
- kafka
- lumberjack
- librato
- loggly
- mongodb
- metriccatcher
- nagios
- null
- nagios_nsca
- opentsdb
- pagerduty
- pipe
- riemann
- redmine
- rackspace
- rabbitmq
- redis
- riak
- s3
- sqs
- stomp
- statsd
- solr_http
- sns
- syslog
- stdout
- tcp
- udp
- webhdfs
- websocket
- xmpp
- zabbix
- zeromq
- Filter plugins
- aggregate
- alter
- anonymize
- collate
- csv
- cidr
- clone
- cipher
- checksum
- date
- de_dot
- dns
- drop
- elasticsearch
- extractnumbers
- environment
- elapsed
- fingerprint
- geoip
- grok
- i18n
- json
- json_encode
- kv
- mutate
- metrics
- multiline
- metaevent
- prune
- punct
- ruby
- range
- syslog_pri
- sleep
- split
- throttle
- translate
- uuid
- urldecode
- useragent
- xml
- zeromq
- Codec plugins
- Contributing to Logstash
- How to write a Logstash input plugin
- How to write a Logstash input plugin
- How to write a Logstash codec plugin
- How to write a Logstash filter plugin
- Contributing a Patch to a Logstash Plugin
- Logstash Plugins Community Maintainer Guide
- Submitting your plugin to RubyGems.org and the logstash-plugins repository
- Glossary of Terms
- Release Notes
Stalled Shutdown Detection
editStalled Shutdown Detection
editShutting down a running Logstash instance involves the following steps:
- Stop all input, filter and output plugins
- Process all in-flight events
- Terminate the Logstash process
The following conditions affect the shutdown process:
- An input plugin receiving data at a slow pace.
-
A slow filter, like a Ruby filter executing
sleep(10000)
or an Elasticsearch filter that is executing a very heavy query. - A disconnected output plugin that is waiting to reconnect to flush in-flight events.
These situations make the duration and success of the shutdown process unpredictable.
Logstash has a stall detection mechanism that analyzes the behavior of the pipeline and plugins during shutdown. This mechanism produces periodic information about the count of inflight events in internal queues and a list of busy worker threads.
To enable Logstash to forcibly terminate in the case of a stalled shutdown, use the --allow-unsafe-shutdown
flag when
you start Logstash.
Stall Detection Example
editIn this example, slow filter execution prevents the pipeline from clean shutdown. By starting Logstash with the
--allow-unsafe-shutdown
flag, quitting with Ctrl+C results in an eventual shutdown that loses events.
bin/logstash -e 'input { generator { } } filter { ruby { code => "sleep 10000" } } output { stdout { codec => dots } }' -w 1 --allow-unsafe-shutdown Settings: User set pipeline workers: 1, Default pipeline workers: 8 Pipeline main started ^CSIGINT received. Shutting down the agent. {:level=>:warn} stopping pipeline {:id=>"main"} Received shutdown signal, but pipeline is still waiting for in-flight events to be processed. Sending another ^C will force quit Logstash, but this may cause data loss. {:level=>:warn} {"inflight_count"=>125, "stalling_thread_info"=>{["LogStash::Filters::Ruby", {"code"=>"sleep 10000"}]=>[{"thread_id"=>17, "name"=>"[main]>worker0", "current_call"=>"(ruby filter code):1:in `sleep'"}]}} {:level=>:warn} The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information. {:level=>:error} {"inflight_count"=>125, "stalling_thread_info"=>{["LogStash::Filters::Ruby", {"code"=>"sleep 10000"}]=>[{"thread_id"=>17, "name"=>"[main]>worker0", "current_call"=>"(ruby filter code):1:in `sleep'"}]}} {:level=>:warn} {"inflight_count"=>125, "stalling_thread_info"=>{["LogStash::Filters::Ruby", {"code"=>"sleep 10000"}]=>[{"thread_id"=>17, "name"=>"[main]>worker0", "current_call"=>"(ruby filter code):1:in `sleep'"}]}} {:level=>:warn} Forcefully quitting logstash.. {:level=>:fatal}
When --allow-unsafe-shutdown
isn’t enabled, Logstash continues to run and produce these reports periodically.
On this page