Kafka Connect: Multiple DB2 JDBC Source Connectors fail Kafka Connect: Multiple DB2 JDBC Source Connectors fail docker docker

Kafka Connect: Multiple DB2 JDBC Source Connectors fail


I eventually figured out the problem:I'm using the JDBC connector in the timestamp mode and not timestamp+incrementing because I cannot (always) specify an incrementing column. I was aware that this might lead to the problem, that Connect cannot know which entries have already been read when there's multiple with the same timestamp.

A big part of my data rows have the same timestamp. When I added the second connector, the current timestamp of the first connector was stored and Connect started the rebalancing and hence loosing the information which rows for that stimestamp had been read already. When the connectors were up and running again the first connector continued with "the next timestamp" and hence only loading the newest rows (which are only a small part).

My mistake was assuming, that in a situation like this, the first connector would restart working with the previous timestamp than continuing with the "next timestamp". It would've made more sense to me to rather risking duplicates than potentially missing data.