Cassandra on Kubernetes podManagementPolicy - OrderedReady vs. Parallel

As far as I know, this is a bad idea. I tried it and got the last node going into CrashLoopBackoff. It seems like the reason is that joining nodes crash if they see another node is trying to join at the same time.

podManagementPolicy: OrderedReady should be the way to go.

Yes that works fine. We are using podManagementPolicy: Parallel in our every statefulsets which includes cassandra cluster also. This really helped us in the entire cluster restart scenario where all the pods comes up at the same time and sync.

Use case of podManagementPolicy: Parallel in our cluster:

We have 3 node baremetal K8s cluster and 3 node cassandra cluster on top of it leveraging the local-storage of the node for PV. In case of local-storage PV is bound to node. So if we set podManagementPolicy: OrderedReady then the issue is, if we bring down 2 node of cluster which have lets say cds-pod-1 and cds-pod-2, both of them goes into unknown state. Now lets say we bring the node up where cds-pod-2 reside then it doesn't bring that pod up because it needs the cds-pod-1 to be in running state to bring cds-pod-2 to running state. Hence we have to change the podManagementPolicy: Paralleland then you can bring up the pod in any fashion and do not depends on the order.

PodManagement parallel is still a safe operation. As Cassandra forms a ring and does not have master/slave architecture, starting the pods in parallel is valid. However as long as the seed pods are not started, the rest of the replicas would join a crash loop until seed pods are up and healthy. Might take a few restarts but they should be up.