Adding a volume to a Kubernetes StatefulSet using kubectl patch

The answer is in your first log:

The StatefulSet "cassandra" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy'

You can't change some fields in a statefulset after creation. You will likely need to delete and recreate the statefulset to add a new volumeClaimTemplate.

edit:It can many times be useful to leave your pods running even when you delete the statefulset. To accomplish this use the --cascade=false flag on the delete operation.

kubectl delete statefulset <name> --cascade=false

Then your workload will stay running while you recreate your statefulset with the updated VPC.

As mentioned by switchboard.op, deleting is the answer.

but

Watch out for deleting these objects:

• PersistentVolumeClaim (kubectl get pvc)
• PersistentVolume (kubectl get pv)

which for example in case you'd want to do just helm uninstall instead of kubectl delete statefulset/<item> will be deleted thus unless there's any other reference for the volumes and in case you don't have backups of the previous YAMLs that contain the IDs (i.e. not just generated from Helm templates, but from the orchestrator) you might have a painful day ahead of you.

PVCs and PVs hold IDs and other reference properties for the underlying (probably/mostly?) vendor specific volume referencing by e.g. S3 or other object or file storage implementation used in the background as a volume in a Pod or other resources.

Deleting or otherwise modifying a StatefulSet if you preserve the PVC name within the spec doesn't affect mounting of the correct resource.

If in doubt, always just copy locally the whole volume prior to doing destructive action to PVCs and PVs if you need them in the future or running commands without knowing the underlying source code e.g. by:

kubectl cp <some-namespace>/<some-pod>:/var/lib/something /tmp/backup-something

and then just load it back by reversing the arguments.

Also for Helm usage, delete the StatefulSet, then issue helm upgrade command and it'll fix the missing StatefulSet without touching PVCs and PVs.