Elasticsearch 5 stuck reading from disk
You probably need to reduce the number of threads for searching. Try going with 2x the number of processors. In the elasticsearch.yaml:
threadpool.search.size:<size>Also, that sounds like too many shards for a 6 node cluster. If possible, I would try reducing that.
The max content of an HTTP request. Defaults to 100mb
servers start reading from disk at 250 MB/s making the cluster unresponsive - The max content of an HTTP request. Defaults to 100mb. . If set to greater than Integer.MAX_VALUE, it will be reset to 100mb.
This will become unresponsive and you might see the logs related this. Check with the max read size of the indices.
Check with Elasticsearch HTTP
a few things;
- 5.x has been EOL for years now, please upgrade as a matter of urgency
- you are heavily oversharded
for point 2 - you either need to;
- upgrade to handle that amount of shards, the memory management in 7.X is far superior
- reduce your shard count by reindexing
- add more nodes to deal with the load