Elasticsearch on AWS: How to fix unassigned shards?
I had a similar issue with AWS Elasticsearch version 6.3, namely 2 shards failed to be assigned, and the cluster had status RED. Running GET _cluster/allocation/explain showed that the reason was that they had exceeded the default maximum allocation retries of 5.
Running the query GET <my-index-name>/_settings revealed the few settings that can be changed per index. Note that all queries are in Kibana format which you have out of the box if you are using AWS Elasticsearch service. The following solved my problem:
PUT <my-index-name>/_settings{ "index.allocation.max_retries": 6}Running GET _cluster/allocation/explain immediately afterwards returned an error with the following: "reason": "unable to find any unassigned shards to explain...", and after some time the problem was resolved.
There might be an alternative solution when the other solutions fail. If you have a managed Elasticsearch Instance on AWS the chances are high that you can "just" restore a snapshot.
Check for failed indexes.
You can use for e.g.:
curl -X GET "https://<es-endpoint>/_cat/shards"or
curl -X GET "https://<es-endpoint>/_cluster/allocation/explain"Check for snapshots.
To find snapshot repositories execute the following query:
curl -X GET "https://<es-endpoint>/_snapshot?pretty"Next let's have a look at all the snapshots in the cs-automated repository:
curl -X GET "https://<es-endpoint>/_snapshot/cs-automated/_all?pretty"Find a snapshot where failures: [ ] is empty or the index you want to restore is NOT in a failed state. Then delete the index you want to restore:
curl -XDELETE 'https://<es-endpoint>/<index-name>'... and restore the deleted index like this:
curl -XPOST 'https://<es-endpoint>/_snapshot/cs-automated/<snapshot-name>/_restore' -d '{"indices": "<index-name>"}' -H 'Content-Type: application/json'There is also some good documentation here: