This issue has been already discussed on Github question - HA to single master migration.

There is already prepared solution for you.

Since etcd-manager was introduced in kops 1.12, and main and events etcd clusters are backup to S3 (same bucket for KOPS_STATE_STORE) automatically and regularly.

So if you have a k8s cluster newer than 1.12 version, maybe you need the following steps:

1. Delete etcd zones in cluster

$ kops edit cluster

In etcdCluster section, remove etcdMembers items to keep only one instanceGroup for main and events. e.g.

  etcdClusters:  - etcdMembers:    - instanceGroup: master-ap-southeast-1a      name: a    name: main  - etcdMembers:    - instanceGroup: master-ap-southeast-1a      name: a    name: events

2. Apply the changes

$ kops update cluster --yes$ kops rolling-update cluster --yes

3. Remove 2 master instance groups

$ kops delete ig master-xxxxxx-1b$ kops delete ig master-xxxxxx-1c

This action cannot be undone, and it will delete the 2 master nodes immediately.

Now 2 out of 3 of your master nodes are deleted, k8s etcd services might be failed and the kube-api service will be unreachable. It is normal that your kops and kubectl commands do not work anymore after this step.

4. Restart the ectd cluster with single master node
   This is the tricky part. ssh into the remaining master node, then

$ sudo systemctl stop protokube$ sudo systemctl stop kubelet

Download the etcd-manager-ctl tool. If using a different etcd-manager version, adjust the download link accordingly

$ wget https://github.com/kopeio/etcd-manager/releases/download/3.0.20190930/etcd-manager-ctl-linux-amd64$ mv etcd-manager-ctl-linux-amd64 etcd-manager-ctl$ chmod +x etcd-manager-ctl$ mv etcd-manager-ctl /usr/local/bin/

Restore backups from S3. See the official docs

$ etcd-manager-ctl -backup-store=s3://<kops s3 bucket name>/<cluster name>/backups/etcd/main list-backups$ etcd-manager-ctl -backup-store=s3://<kops s3 bucket name>/<cluster name>/backups/etcd/main restore-backup 2019-10-16T09:42:37Z-000001# do the same for events$ etcd-manager-ctl -backup-store=s3://<kops s3 bucket name>/<cluster name>/backups/etcd/events list-backups$ etcd-manager-ctl -backup-store=s3://<kops s3 bucket name>/<cluster name>/backups/etcd/events restore-backup 2019-10-16T09:42:37Z-000001

This does not start the restore immediately; you need to restart etcd: kill related containers and start kubelet

$ sudo systemctl start kubelet$ sudo systemctl start protokube

Wait for the restore to finish, then kubectl get nodes and kops validate cluster should be working. If not, you can just terminate the EC2 instance of the remaining master node in AWS console, a new master node will be created by Auto Scaling Groups, and etcd cluster will be restored.

These are the steps to be taken in order to downsize count of master node in a KOPS deployed cluster

Note: Before you even attempt to follow steps described here consider first it it is possible to re-create the cluster.Following steps, although allowed me to reduce from 3 master to 1 in the end, on different occasions required additional troubleshooting. All I learned from the process is below but your case may be different and therefore success is not guaranteed.

Prerequisites

Go to AWS console and determine the private IP (MASTER_IP variable later on) and availability zone of the master node that is meant to be become a single-master after this procedure is over (AZ).

You will need AWS CLI access configured to S3 for kops to be able to work.You will need kubectl configured to work with the cluster we are to operate on.If, for any reason things go bad, you may need SSH key allowing you to get to remaining master node to recover ETCD there (as the kubectl wouldn't be available any more in such case). This case is currently not covered by this doc.Provide values to MASTER_IP, AZ, KOPS_STATE_BUCKET and CLUSTER_NAME to match your environment.

# MASTER_IP is the IP of master node in availability zone AZ (so "c" in this example)export MASTER_IP="172.20.115.115"export AZ="c"export KOPS_STATE_BUCKET="mironq-prod-eu-central-1-state-store"export CLUSTER_NAME="mironq.prod.eu-central-1.aws.svc.example.com"# no need to change following command unless you use different version of Etcdexport BACKUP_MAIN="s3://${KOPS_STATE_BUCKET}/${CLUSTER_NAME}/backups/etcd/main"export BACKUP_EVENT="s3://${KOPS_STATE_BUCKET}/${CLUSTER_NAME}/backups/etcd/events"export ETCD_CMD="/opt/etcd-v3.4.3-linux-amd64/etcdctl --cacert=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt --cert=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.crt --key=/rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.key --endpoints=https://127.0.0.1:4001"CONTAINER=$(kubectl get pod -l k8s-app=etcd-manager-main -o=jsonpath='{.items[*].metadata.name}'|tr ' ' '\n'|grep ${MASTER_IP})

Notice: your CONTAINER variable should contains now name of the pod with master that is going to remain, i.e:

$ echo $CONTAINER etcd-manager-main-ip-172-20-109-104.eu-central-1.compute.internal

Now confirm Etcd backups exist and are very recent (~ 15min ago) and current number of members of the cluster.

kubectl exec -it -n kube-system ${CONTAINER} -- /etcd-manager-ctl -backup-store=${BACKUP_MAIN} list-backups|sort -nkubectl exec -it -n kube-system ${CONTAINER} -- /etcd-manager-ctl -backup-store=${BACKUP_EVENT} list-backups|sort -n# Confirm current members of existing Etcd clusterkubectl exec -it -n kube-system ${CONTAINER} -- ${ETCD_CMD} member list

Remove Etcd nodes

Get IDs of Etcd nodes that are going to be removed

MASTERS2DELETE=$(kubectl exec -it -n kube-system ${CONTAINER} -- ${ETCD_CMD} member list|grep -v etcd-${AZ}|cut -d, -f1)#$ echo $MASTERS2DELETEefb9893f347468eb ffea6e819b91a131

Now you are ready to remove the not needed Etcd nodes

for MASTER in ${MASTERS2DELETE};do echo "Deleting ETCD node ${MSTR}"; kubectl exec -it -n kube-system ${CONTAINER} -- ${ETCD_CMD} member remove ${MASTER}; done# a few minutes may be needed after this has been executed# Confirm only one member is leftkubectl exec -it -n kube-system ${CONTAINER} -- ${ETCD_CMD} member list

Also you will see that some master nodes became not ready

$ kubectl get node

Schedule backup restore

!!! IMPORTANT !!!Now we need to ensure a new backup is taken before we proceed. By default etcd-manager takes a backup every 15 minutes. Wait until a new back is taken as it will contain information about expected number of node (=1)

Now that we have a new backup created of this single node cluster we can schedule a recovery of it after restarting.Below code contain commented out responses to help you determine if your command executed as expected.

Schedule restore of "main" cluster.

BACKUP_LIST=$(kubectl exec -it -n kube-system ${CONTAINER} -- /etcd-manager-ctl -backup-store=${BACKUP_MAIN} list-backups|sort -n)#echo "$BACKUP_LIST"#[...]#2020-12-17T14:55:55Z-000001#2020-12-17T15:11:05Z-000002#2020-12-17T15:26:13Z-000003#2020-12-17T15:41:14Z-000001#2020-12-17T15:56:20Z-000001#2020-12-17T16:11:35Z-000004#2020-12-17T16:26:41Z-000005LATEST_BACKUP=$(echo -n "${BACKUP_LIST}"|tail -1)# confirm that latest backup has been selected#$ echo $LATEST_BACKUP #2020-12-17T16:26:41Z-000005kubectl exec -it -n kube-system ${CONTAINER} -- /etcd-manager-ctl -backup-store=${BACKUP_MAIN} restore-backup "${LATEST_BACKUP/%[$'\t\r\n']}"#Backup Store: s3://mironq-prod-eu-central-1-state-store/mironq.prod.eu-central-1.aws.svc.example.com/backups/etcd/main#I1217 16:41:59.101078   11608 vfs.go:60] Adding command at s3://mironq-prod-eu-central-1-state-store/mironq.prod.eu-central-1.aws.svc.example.com/backups/etcd/main/control/#2020-12-17T16:41:59Z-000000/_command.json: timestamp:1608223319100999598 restore_backup:<cluster_spec:<member_count:3 etcd_version:"3.4.3" > backup:"2020-12-17T16:26:41Z-000005\r" #> #added restore-backup command: timestamp:1608223319100999598 restore_backup:<cluster_spec:<member_count:3 etcd_version:"3.4.3" > backup:"2020-12-17T16:26:41Z-000005\r" > 

Schedule restore of "main" cluster.

BACKUP_LIST=$(kubectl exec -it -n kube-system ${CONTAINER} -- /etcd-manager-ctl -backup-store=${BACKUP_EVENT} list-backups|sort -n)#$ echo "$BACKUP_LIST"#Backup Store: s3://mironq-prod-eu-central-1-state-store/mironq.prod.eu-central-1.aws.svc.example.com/backups/etcd/events#I1217 16:48:41.230896   17761 vfs.go:102] listed backups in s3://mironq-prod-eu-central-1-state-store/mironq.prod.eu-central-1.aws.svc.example.com/backups/etcd/events: #[2020-12-17T14:56:08Z-000001 2020-12-17T15:11:17Z-000001 2020-12-17T15:26:26Z-000002 2020-12-17T15:41:27Z-000002 2020-12-17T15:56:32Z-000003 2020-12-17T16:11:41Z-000003 #2020-12-17T16:26:48Z-000004 2020-12-17T16:41:56Z-000001]#2020-12-17T14:56:08Z-000001#2020-12-17T15:11:17Z-000001#2020-12-17T15:26:26Z-000002#2020-12-17T15:41:27Z-000002#2020-12-17T15:56:32Z-000003#2020-12-17T16:11:41Z-000003#2020-12-17T16:26:48Z-000004#2020-12-17T16:41:56Z-000001LATEST_BACKUP=$(echo -n "${BACKUP_LIST}"|tail -1)# confirm that latest backup has been selected$ echo $LATEST_BACKUP 2020-12-17T16:41:56Z-000001kubectl exec -it -n kube-system ${CONTAINER} -- /etcd-manager-ctl -backup-store=${BACKUP_EVENT} restore-backup "${LATEST_BACKUP/%[$'\t\r\n']}"#Backup Store: s3://mironq-prod-eu-central-1-state-store/mironq.prod.eu-central-1.aws.svc.example.com/backups/etcd/events#I1217 16:53:17.876318   21958 vfs.go:60] Adding command at s3://mironq-prod-eu-central-1-state-store/mironq.prod.eu-central-1.aws.svc.example.com/backups/etcd/events/control/#2020-12-17T16:53:17Z-000000/_command.json: timestamp:1608223997876256810 restore_backup:<cluster_spec:<member_count:3 etcd_version:"3.4.3" > backup:"2020-12-17T16:41:56Z-000001\r" #> #added restore-backup command: timestamp:1608223997876256810 restore_backup:<cluster_spec:<member_count:3 etcd_version:"3.4.3" > backup:"2020-12-17T16:41:56Z-000001\r" > 

Check if endpoint is still healthy (as it should be)

# check if endpoint is healthykubectl exec -it -n kube-system ${CONTAINER} -- ${ETCD_CMD} endpoint health#https://127.0.0.1:4001 is healthy: successfully committed proposal: took = 7.036109ms

Remove instance groups

Example list of instance groups

kops --name ${CLUSTER_NAME} --state s3://${KOPS_STATE_BUCKET} get ig#NAME           ROLE    MACHINETYPE MIN MAX ZONES#bastions       Bastion t3.micro    1   1   eu-central-1a,eu-central-1b,eu-central-1c#master-eu-central-1a   Master  t3.medium   1   1   eu-central-1a#master-eu-central-1b   Master  t3.medium   1   1   eu-central-1b#master-eu-central-1c   Master  t3.medium   1   1   eu-central-1c#nodes          Node    t3.medium   2   6   eu-central-1a,eu-central-1b

Delete master instance groups in availability zones where we disabled Etcd nodes (a and b in this example as we want leave c running as the only master).Edit command below (replace the [AZ-letter] to match you case.

kops --name ${CLUSTER_NAME} --state s3://${KOPS_STATE_BUCKET} delete ig master-eu-central-1[AZ-letter]#InstanceGroup "master-eu-central-1a" found for deletion#I1217 17:01:39.035294 2538280 delete.go:54] Deleting "master-eu-central-1a"#Deleted InstanceGroup: "master-eu-central-1a"

Calling edit mode with command below manually edit cluster. The goal here is to match remaining Etcd node with cluster configuration: by the example below, you need to remove entries for nodes that don't exist any more.

change this:

  etcdClusters:  - cpuRequest: 200m    etcdMembers:    - instanceGroup: master-eu-central-1a      name: a    - instanceGroup: master-eu-central-1b      name: b    - instanceGroup: master-eu-central-1c      name: c    memoryRequest: 100Mi    name: main  - cpuRequest: 100m    etcdMembers:    - instanceGroup: master-eu-central-1a      name: a    - instanceGroup: master-eu-central-1b      name: b    - instanceGroup: master-eu-central-1c      name: c    memoryRequest: 100Mi    name: events

to this (leave only the zone in which there still is a master):

  etcdClusters:  - cpuRequest: 200m    etcdMembers:    - instanceGroup: master-eu-central-1c      name: c    memoryRequest: 100Mi    name: main  - cpuRequest: 100m    etcdMembers:    - instanceGroup: master-eu-central-1c      name: c    memoryRequest: 100Mi    name: events

This will open editor allowing to make the changes.

kops --name ${CLUSTER_NAME} --state s3://${KOPS_STATE_BUCKET} edit cluster

Apply KOPS changes

Apply changes and force master node re-creation (second command will render cluster unresponsive until new master is created and brought back online.

kops --name ${CLUSTER_NAME} --state s3://${KOPS_STATE_BUCKET} update cluster --yeskops --name ${CLUSTER_NAME} --state s3://${KOPS_STATE_BUCKET} rolling-update cluster --cloudonly --yes

Troubleshooting

Both Etcd clusters, "main" and "events" have to be back on-line to get API up again. If API server log complains on being unable to connect to port 4001 then your "main" Etcd cluster isn't starting and it's "events" if port number is 4002.Just above you have instructed the Etcd clusters to import back a backup and this has to complete to get the cluster up