Scaling Down Replicas On Completion Without Termination
You can add the termination grace period for all pods so when container terminates it's wait for some time.
you can also add the SIG term in lifecycle also that particular pod donot take next item from queue.
lifecycle: preStop: exec: # SIGTERM triggers a quick exit; gracefully terminate instead command: ["/usr/sbin/nginx","-s","quit"]you can also have look at this : https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-terminating-with-grace
If you find yourself in the same situation as me, have a look at Argo: https://argoproj.github.io/argo/