Downscaling of pods is not graceful when using Kafka-connect on kubernetes

It could be that the default grace period is not enough for you aaplication to finish its tasks after recevied SIGTERM singal.

SIGTERM signal is sent the the main process in the container and once the signal is recevied container should start a graceful shutdown of the running application and exit.

There is a very good explanation/flow described in kubernetes official documentation about Termination of Pods.

You could try to extend the terminationGracePeriodSeconds in your deployment to see if that helps (The default is 30):

spec:    replicas:     template:        spec:            containers:              - name: test                image: ...            terminationGracePeriodSeconds: 60

The other way is to use preStop hook. preStop hook is executed immediately before a container is terminated. How it works is when container needs to be terminated, Kubelet will run the pre-stop hook and only then send SIGTERM to the process. This can be used to initate a graceful shutdown of the container.

It can be also used to perform some other operations before shutdown without having to implement those in the app itself.

This is an simple example how it works (It is a HTTP GET request that will be sent to `http:///shutdown):

lifecycle:   preStop:      httpGet:         port: 80         path: shutdown

Here is also link to kubernetes documentation about Container hooks. Let me know if this was heplful.

just increase shutdown timeout "task.shutdown.graceful.timeout.ms" in kafka-connect config. Also use preStop hook pods hooks and use some sleep command in hook so that kubelet wait for hook to complete and run SIGTERM after that.