Unable to make requests to Kafka container from another container using kafka-python
Quoting Robin Moffet’s excellent blog post on Kafka listeners and docker:
If you are using docker you need to set KAFKA_ADVERTISED_LISTENERS to the external address (host or IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.
https://rmoff.net/2018/08/02/kafka-listeners-explained/
Kafka client connections are actually a two step process that involved connecting to the bootstrap server first to request metadata about the entire cluster, and then connecting to one or many of the clusters nodes using the advertised listeners names and ports.
I appear to solved the problem by using a different container version:
kafka: image: wurstmeister/kafka:2.11-0.11.0.3My consumer object can now retrieve the topic list from the broker. Something I was unable to do previously.