Failing to start shard in ElasticSearch IndexShardGatewayRecoveryException "sending failed"
It seems that I had a messed up shard, that needed fixing.It's a Lucene thing, where you tell Lucene to fix the shard.
For Ubuntu, the solution was to go to the /usr/share/elasticsearch/lib
directory and find out which Lucene core version is was running (running ls
will show you a file named something like lucene-core-4.8.1.jar) and then type:
java -cp lucene-core-x.x.x.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /var/lib/elasticsearch/<clustername>/nodes/0/indices/<index>/<shard>/index/ -fix
Replace the x.x.x with the Lucene core version, with your clustername, index with the name of the index and of course with the failing shard number.
This can potentially give a loss of documents
But it fixed our issue.
I faced this problem multiple times. As my set up is to read clickstream data (12-20M hits a day), I could not afford data losses.
So this was my solution and it runs beautifully:
Solution:
- Stop elasticsearch from running
- go to /path/to/my/data/mycluster_name/nodes/0/indices/myindex_name/index
- delete segments.gen file
- Start elasticsearch
Problem Root Cause
Shards fail for various reasons, especially when shards are not able to fulfill Kibana requests.
Lucene is not directly connected to this process. So when there are issues, elasticsearch is not able to efficiently pick the shards value from Lucene segments references that are stored in segments.gen
Lucene sets this value fresh again in the next run. So elasticsearch is able to reference the values correctly. And the shard issue is resolved.
Taking hints from Repox. In Centos 6.5 with the built-in ElasticSearch in logstash provisioned using Chef.
- https://github.com/lusis/chef-logstash (v0.10.0)
- logstash 1.4.2
- ES/logstash working directory eg /opt/logstash/forwarder
java -cp /opt/logstash/forwarder/vendor/jar/elasticsearch-1.1.1/lib/lucene-core-4.7.2.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /opt/logstash/forwarder/data/elasticsearch/nodes/0/indices/logstash-2014.11.01/3/index/ -fix
But even with fixing I still see Failed to start shard, message ... failed to recover shard. I have to destructively delete using eg curator delete --older-than 3
.