Logstash shutdown stalling when starting from bash script

Logstash tries to performs various steps when you try to shutdown such as,

• It stop all input, filter and output plugins
• Process all in-flight events
• Terminate the Logstash process

and there are various factors which makes the shutdown process very unpredictable such as,

• An input plugin receiving data at a slow pace.
• A slow filter, like a Ruby filter executing sleep(10000) or an Elasticsearch filter that is executing a very heavy query.
• A disconnected output plugin that is waiting to reconnect to flush in-flight events.

From Logstash documentation,

Logstash has a stall detection mechanism that analyzes the behavior of the pipeline and plugins during shutdown. This mechanism produces periodic information about the count of inflight events in internal queues and a list of busy worker threads.

You can use --pipeline.unsafe_shutdown flag while starting logstash to force terminate the process in case of stalled shutdown. When --pipeline.unsafe_shutdown isn’t enabled, Logstash continues to run and produce these reports periodically, this is why the problem appears to be random in your case.

Remember that Unsafe shutdowns, force-kills of the Logstash process, or crashes ofthe Logstash process for any other reason may result in data loss(unless you’ve enabled Logstash to use persistent queues).