How can I restore etcd cluster from snapshot in docker image on CoreOS?

according to the Etcd Disaster Recovery document, you need restore all three etcd nodes from snapshot with commands like yours, then run three node with commands like this:

etcd \  --name m1 \  --listen-client-urls http://host1:2379 \  --advertise-client-urls http://host1:2379 \  --listen-peer-urls http://host1:2380 &

Also, you can extract etcdctl from the image, like this:

docker run --rm -v /opt/bin:/opt/bin registry:5000/quay.io/coreos/etcd:v3.1.5 cp /usr/local/bin/etcdctl /opt/bin

Then use etcdctl to restore snapshot:

# ETCDCTL_API=3 ./etcdctl snapshot restore snapshot.db \  --name m1 \  --initial-cluster m1=http://host1:2380,m2=http://host2:2380,m3=http://host3:2380 \  --initial-cluster-token etcd-cluster-1 \  --initial-advertise-peer-urls http://host1:2380 \  --data-dir /var/lib/etcd

This will restore snapshot to the /var/lib/etcd directory. Then start etcd with docker, don't forget mount /var/lib/etcd into your container, and specify --data-dir to it .

Ectd in kubernetes is running in Docker containers, here was what I did to recovery the cluster:

• retrieve Etcd cluster metedata

  docker inspect etcd1

  you'd got something like below:

  "Binds": [    "/etc/ssl/certs:/etc/ssl/certs:ro",    "/etc/ssl/etcd/ssl:/etc/ssl/etcd/ssl:ro",    "/var/lib/etcd:/var/lib/etcd:rw"],..."Env": [    "ETCD_DATA_DIR=/var/lib/etcd",    "ETCD_ADVERTISE_CLIENT_URLS=https://172.16.60.1:2379",    "ETCD_INITIAL_ADVERTISE_PEER_URLS=https://172.16.60.1:2380",    "ETCD_INITIAL_CLUSTER_STATE=existing",    "ETCD_METRICS=basic",    "ETCD_LISTEN_CLIENT_URLS=https://172.16.60.1:2379,https://127.0.0.1:2379",    "ETCD_ELECTION_TIMEOUT=5000",    "ETCD_HEARTBEAT_INTERVAL=250",    "ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd",    "ETCD_LISTEN_PEER_URLS=https://172.16.60.1:2380",    "ETCD_NAME=etcd1",    "ETCD_PROXY=off",    "ETCD_INITIAL_CLUSTER=etcd1=https://172.16.60.1:2380,etcd2=https://172.16.60.2:2380,etcd3=https://172.16.60.2:2380",    "ETCD_AUTO_COMPACTION_RETENTION=8",    "ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem",    "ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-node01.pem",    "ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-node01-key.pem",    "ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem",    "ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-node01.pem",    "ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-node01-key.pem",    "ETCD_PEER_CLIENT_CERT_AUTH=true",    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd": [    "/usr/local/bin/etcd"],

• copy etcd snapshotdb to other etcd nodes

  scp snapshotdb_20180913 node02:/root/  scp snapshotdb_20180913 node03:/root/  

• rebuild a new cluster with original info

  # etcd1docker stop etcd1rm -rf /var/lib/etcdETCDCTL_API=3 etcdctl snapshot restore snapshotdb_20180913 \  --cacert /etc/ssl/etcd/ssl/ca.pem \  --cert /etc/ssl/etcd/ssl/member-node01.pem \  --key /etc/ssl/etcd/ssl/member-node01-key.pem \  --name etcd1 \  --initial-cluster etcd1=https://node01:2380,etcd2=https://node02:2380,etcd3=https://node03:2380 \  --initial-cluster-token k8s_etcd \  --initial-advertise-peer-urls https://node01:2380 \  --data-dir /var/lib/etcd# etcd2docker stop etcd2rm -rf /var/lib/etcdETCDCTL_API=3 etcdctl snapshot restore snapshotdb_20180913 \  --cacert /etc/ssl/etcd/ssl/ca.pem \  --cert /etc/ssl/etcd/ssl/member-node02.pem \  --key /etc/ssl/etcd/ssl/member-node02-key.pem \  --name etcd2 \  --initial-cluster etcd1=https://node01:2380,etcd2=https://node02:2380,etcd3=https://node03:2380 \  --initial-cluster-token k8s_etcd \  --initial-advertise-peer-urls https://node02:2380 \  --data-dir /var/lib/etcd# etcd3docker stop etcd3rm -rf /var/lib/etcdETCDCTL_API=3 etcdctl snapshot restore snapshotdb_20180913 \  --cacert /etc/ssl/etcd/ssl/ca.pem \  --cert /etc/ssl/etcd/ssl/member-node03.pem \  --key /etc/ssl/etcd/ssl/member-node03-key.pem \  --name etcd3 \  --initial-cluster etcd1=https://node01:2380,etcd2=https://node02:2380,etcd3=https://node03:2380 \  --initial-cluster-token k8s_etcd \  --initial-advertise-peer-urls https://node03:2380 \  --data-dir /var/lib/etcd

• start containers and check cluster status

  cd /etc/ssl/etcd/ssletcdctl \  --endpoints=https://node01:2379 \  --ca-file=./ca.pem \  --cert-file=./member-node01.pem \  --key-file=./member-node01-key.pem \  member list