Issue while updating a record in 3 node cassandra cluster deployed using kubernetes

Thanks guys and sorry for delayed reply.

I found the root cause for this behavior. Actually much later I found out that the Cassandra relies (for column timestamp) on client timestamp. Client means the different pod's (instances of microservice).In my case there were 3 containers running on different hosts. Finally after a lot of struggle and research I figured out that there was slight clock drift among these containers running on different hosts. Later I installed the NTP server on all these hosts which helped us keeping the time in sync across these nodes. Similar to NTP you can also install any time syn server/utility and get away from the problem of nodes clock drift issue.

Though this helped me and will also help other in keeping node clock in sync. But in certain corner cases I found based on the sync time configured with NTP server there could be instances where you can find 2-3 seconds drift across nodes (as in my case the NTP sync time was 2 seconds). Which can be further reduced by reducing the sync time across nodes.

But eventually the root cause was only the clock drift across nodes running microservices.