Unable to run console consumer using a Kafka broker inside a Docker container

Make sure to configure the following:

KAFKA_LISTENERS: LISTENER_INTERNAL://kafka-host:29092,LISTENER_EXTERNAL://kafka-host:9092KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka-host:29092,LISTENER_EXTERNAL://localhost:9092KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXTKAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL

• Now all your clients that run within the Docker Network, should use

  LISTENER_INTERNAL://kafka-host:29092

• All clients outside the Docker Network should use

  LISTENER_EXTERNAL://kafka-host:9092

For a more comprehensive guide, read @RobinMoffatt article.

So, I have solved the problem and I now have a working Docker/Kafka environment: I have a zookeeper, a broker and a consumer running in docker, and I can produce messages both from a producer running in docker and a consumer running from a command line.

As @GiorgosMyrianthous suspected, the problem was with the listeners and advertised listeners. I entered the following values in the Kafka broker properties file

listeners=PLAINTEXT://:29092,PLAINTEXT_HOST://:9092advertised.listeners=PLAINTEXT://server:29092,PLAINTEXT_HOST://localhost:9092listener.security.protocol.map=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

and I started the container with port 9092 exposed to the outside world.

I ran the Docker consumer with the bootstrap server set to server:29092.

I ran the Docker producer with the bootstrap server set to server:29092.

I ran the command line producer with the bootstrap server set to localhost:9092.

Messages typed in either of the producers appeared in the consumer.