Re-attach volume claim on deployment update

The issue here is that EBS volumes are ReadWriteOnce and can only be mounted to a single pod, so when you do the rolling update the old pod holds the volume. For this to work you would either have to use StatefulSet or you can use any of the ReadWriteMany PV types.

A Kubernetes Deployment is sometimes better used for stateless pods.

You can always go with the brute force approach which force delete the pod that is holding the volume. Make sure that the Reclaim Policy is set to Retain.

From the context you provided in your question, I can't tell if your intention was to run a single instance stateful application, or a clustered stateful application.

I ran into this problem recently and from this section in the docs, here's how to go about this...

If you're running a single instance stateful app:

• You should should not scale the app, that is, leave the default value of spec.replicas as 1 if you're using a Deployment
• You should instruct Kubernetes to not use rolling updates, that is, you should set spec.strategy.type to Recreate in your Deployment

Sample Deployment (from the docs):

# application/mysql/mysql-deployment.yamlapiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2kind: Deploymentmetadata:  name: mysqlspec:  selector:    matchLabels:      app: mysql  strategy:    type: Recreate  template:    metadata:      labels:        app: mysql    spec:      containers:      - image: mysql:5.6        name: mysql        env:          # Use secret in real usage        - name: MYSQL_ROOT_PASSWORD          value: password        ports:        - containerPort: 3306          name: mysql        volumeMounts:        - name: mysql-persistent-storage          mountPath: /var/lib/mysql      volumes:      - name: mysql-persistent-storage        persistentVolumeClaim:          claimName: mysql-pv-claim

And the sample PersistentVolume & PersistentVolumeClaim (from the docs):

# application/mysql/mysql-pv.yamlapiVersion: v1kind: PersistentVolumemetadata:  name: mysql-pv-volume  labels:    type: localspec:  storageClassName: manual  capacity:    storage: 20Gi  accessModes:    - ReadWriteOnce  hostPath:    path: "/mnt/data"---apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: mysql-pv-claimspec:  storageClassName: manual  accessModes:    - ReadWriteOnce  resources:    requests:      storage: 20Gi

The obvious underlying matter here is that a rolling update will not work, because there can be no more than one pod running at any time. Setting spec.strategy.type to Recreate tells Kubernetes to stop the running pod before deploying a new one, so presumably there will be some downtime, even if minimal.

If you need a clustered stateful application, then using the already mentioned StatefulSet as a controller type or ReadWriteMany as a storage type would probably be the way to go.

Not sure, RollingUpdate may solve the problem. Since "Rolling Update" is safe way to update the containers images, according to docs. I assume, K8s can handle PV/PVC too.