ElasticSearch: Unassigned Shards, how to fix?

By default, Elasticsearch will re-assign shards to nodes dynamically. However, if you've disabled shard allocation (perhaps you did a rolling restart and forgot to re-enable it), you can re-enable shard allocation.

# v0.90.x and earliercurl -XPUT 'localhost:9200/_settings' -d '{    "index.routing.allocation.disable_allocation": false}'# v1.0+curl -XPUT 'localhost:9200/_cluster/settings' -d '{    "transient" : {        "cluster.routing.allocation.enable" : "all"    }}'

Elasticsearch will then reassign shards as normal. This can be slow, consider raising indices.recovery.max_bytes_per_sec and cluster.routing.allocation.node_concurrent_recoveries to speed it up.

If you're still seeing issues, something else is probably wrong, so look in your Elasticsearch logs for errors. If you see EsRejectedExecutionException your thread pools may be too small.

Finally, you can explicitly reassign a shard to a node with the reroute API.

# Suppose shard 4 of index "my-index" is unassigned, so you want to# assign it to node search03:curl -XPOST 'localhost:9200/_cluster/reroute' -d '{    "commands": [{        "allocate": {            "index": "my-index",            "shard": 4,            "node": "search03",            "allow_primary": 1        }    }]}'

OK, I've solved this with some help from ES support. Issue the following command to the API on all nodes (or the nodes you believe to be the cause of the problem):

curl -XPUT 'localhost:9200/<index>/_settings' \    -d '{"index.routing.allocation.disable_allocation": false}'

where <index> is the index you believe to be the culprit. If you have no idea, just run this on all nodes:

curl -XPUT 'localhost:9200/_settings' \    -d '{"index.routing.allocation.disable_allocation": false}'

I also added this line to my yaml config and since then, any restarts of the server/service have been problem free. The shards re-allocated back immediately.

FWIW, to answer an oft sought after question, set MAX_HEAP_SIZE to 30G unless your machine has less than 60G RAM, in which case set it to half the available memory.

References

• Shard Allocation Awareness

This little bash script will brute force reassign, you may lose data.

NODE="YOUR NODE NAME"IFS=$'\n'for line in $(curl -s 'localhost:9200/_cat/shards' | fgrep UNASSIGNED); do  INDEX=$(echo $line | (awk '{print $1}'))  SHARD=$(echo $line | (awk '{print $2}'))  curl -XPOST 'localhost:9200/_cluster/reroute' -d '{     "commands": [        {            "allocate": {                "index": "'$INDEX'",                "shard": '$SHARD',                "node": "'$NODE'",                "allow_primary": true          }        }    ]  }'done