kafka on kubernetes cannot produce/consume topics (ClosedChannelException, ErrorLoggingCallback)
Kafka register to zookeeper with its service's name. And consuming/producing messages need access to the service names(here is dns records on zookeeper-1, zookeeper-2, zookeeper-3), which are only accessible through kubernetes' dns. So only application running on kubernetes can access my kafka. Therefore I cannot use the external IP of kafka-service or port-forwarding the kafka pod to localhost and then access it.
But why can I create, list and describe topics outside kubernetes cluster? I guess it is because zookeepers can do those operation by themselves. But consuming/producing message will need access to the ADVERTISED_HOST_NAME provided by kafka.