Check my Kafka and Zookeeper functionality and connections

kafkacat -L -b kafka-statefulset-0.kafka headless.default.svc.cluster.local:9093

Doesn't work. There is a space in there after the first -b flag

kafkacat -b 172.17.0.10:9093 -t second_topic -P

Without the correct advertised.listeners that return the resolvable addresses of brokers, this won't work.

kafkacat -L -b 172.17.0.10:9093

This is a step in the right direction, but you're using the Docker network IP, and not a Cluster / Node Port of any of the services, still isn't portable to other machines outside your cluster.

installing Kafkacat inside the pod's container and trying to talk with other Kafka's broker

That's a good test to make sure that replication will at least work, but doesn't solve the external client issue.

I got the right metadata but with error in the end

That could be a sign that your healthcheck on the broker or zookeeper is failing and the pod is restarting. Notice that the error cycles from 0,2,4...

I also built headless service for talking with each Kafka's Broker through port 9093

Okay, great, but you need to set advertised.listeners of each broker to now accept connections on 9093.

All in all, I think just using existing Kafka Helm charts that allow for external connectivity would be your best option.

As You mentioned in comment:

I ran - / # nslookup headless.default.svc.cluster.local and got - Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local nslookup: can't resolve 'headless.default.svc.cluster.local

Problem is related with DNS in your environment as your environment cannot resolve headless service.DNS component should provide Pod's DNS names.

You should receive something like:

/ # nslookup my-kafka-headlessServer:    10.122.0.10Address 1: 10.122.0.10 kube-dns.kube-system.svc.cluster.localName:      my-kafka-headlessAddress 1: 10.56.0.5 my-kafka-0.my-kafka-headless.default.svc.cluster.local

Treat is as sort of prerequisite if you want to refer to Pod's backing the Statefulset in Kubernetes based on DNS name (in your case: headless.default.svc.cluster.local).

Verify if your services have set .sepec.clusterIP: None or if everything is ok with kube-dns-XXXX pod in kube-system namespace. Here you can find some information about troubleshooting your DNS issue.

Also as @cricket_007 advised you can use helm for deploying kafka. For example helm chart from this source which also contains HowTo.

I had a similar issue and I fixed it by adding the following changes in the client OS hostfile. (in mac it's in private/etc/host with root access. Then you should sudo dscacheutil -flushcache)

127.0.0.1 my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svclocalhost my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc

Kafkacat is already telling you what's the resolved address looks like its:

kafka-statefulset-0.kafka-headless.default.svc.cluster.local

you can also add any IP that resolves the domain(if it's routed to the domain)