kafka docker stuck on configuration and not working on mac

Seems like you missed an important part from your docker-compose log. I ran your docker-compose file on my local and would like to highlight this part from the logs.

kafka        | [2021-04-19 09:20:19,209] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)kafka        | [2021-04-19 09:20:19,253] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)kafka        | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9092kafka        |  at kafka.utils.CoreUtils$.validate$1(CoreUtils.scala:265)kafka        |  at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:276)kafka        |  at kafka.server.KafkaConfig.$anonfun$listeners$1(KafkaConfig.scala:1680)kafka        |  at kafka.server.KafkaConfig.listeners(KafkaConfig.scala:1679)kafka        |  at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1779)kafka        |  at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1756)kafka        |  at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1312)kafka        |  at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)kafka        |  at kafka.Kafka$.main(Kafka.scala:68)kafka        |  at kafka.Kafka.main(Kafka.scala)kafka exited with code 1

So you were using the same port for accessing Kafka from inside and outside the docker network. I changed the below setting (the INSIDE port) and it worked for me. I was able to create and list topics. I was also able to produce and consume simple messages from the kafka container.

 KAFKA_LISTENERS: INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092 KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:29092,OUTSIDE://localhost:9092

This is some of the testing I did from the kafka container

docker exec -it kafka bash    bash-4.4# cd /opt/kafka_2.13-2.7.0/bash-4.4# bin/kafka-topics.sh --create --topic test-topic --partitions 16 --replication-factor 1 --zookeeper zookeeper:2181Created topic test-topic.bash-4.4# bin/kafka-topics.sh --list --zookeeper zookeeper:2181test-topic

This is an excellent post that I would like to refer you. Please let me know if this works for you or not.

Try to use another image of kafka, to example - bitnami. I had the same problem as you on Apple M1 Big Shur, and i used bitnami and it's working!! this is my docker-compose.yml:

version: "2"services:    zookeeper:        image: docker.io/bitnami/zookeeper:3        ports:            - "2181:2181"        volumes:            - "zookeeper_data:/bitnami"        environment:            - ALLOW_ANONYMOUS_LOGIN=yes    kafka:        image: docker.io/bitnami/kafka:2        ports:            - "9092:9092"        volumes:            - "kafka_data:/bitnami"        environment:            - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181            - ALLOW_PLAINTEXT_LISTENER=yes            - KAFKA_LISTENERS=PLAINTEXT://:9092            - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092        depends_on:            - zookeepervolumes:    zookeeper_data:        driver: local    kafka_data:        driver: local