Kafka in Kubernetes - Marking the coordinator dead for group

I had the same problem as you last week and solved it, soit's possible to expose Kafka outside Kubernetes!

Solution:In your Kafka broker-config.yaml you should map cluster external IP to your local DNS

kafka-I.kafka-hs.default.svc.cluster.local:9093

How To:

add those to your server.properties file:

listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXTinter.broker.listener.name=INTERNAL_PLAINTEXT

if you have an init which run before server.properties you should add those:

# add unique label to each podkubectl label pods ${HOSTNAME} kafka-set-component=${HOSTNAME}EXTERNAL_LISTENER_IP=<YOUR_KUBERNETES_CLUSTER_EXTERNAL_IP>EXTERNAL_LISTENER_PORT=$((30093 + ${HOSTNAME##*-}))sed -i "s/#listeners=PLAINTEXT:\/\/:9092/listeners=INTERNAL_PLAINTEXT:\/\/0.0.0.0:9092,EXTERNAL_PLAINTEXT:\/\/0.0.0.0:9093/" /etc/kafka/server.propertiessed -i "s/#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092/advertised.listeners=INTERNAL_PLAINTEXT:\/\/$HOSTNAME.broker.kafka.svc.cluster.local:9092,EXTERNAL_PLAINTEXT:\/\/$EXTERNAL_LISTENER_IP:$EXTERNAL_LISTENER_PORT/" /etc/kafka/server.properties

otherwise you should find a way to add replace configurations in your server.properties at runtime.

Notice that you must have those lines commented in your server.properties file

#listeners=PLAINTEXT://:9092#advertised.listeners=PLAINTEXT://your.host.name:9092

Services:Create headless service to map local DNS and a service for each broker you have:

# A headless service to create DNS records---apiVersion: v1kind: Servicemetadata:  name: broker  namespace: kafkaspec:  ports:  - port: 9092  # [podname].broker.kafka.svc.cluster.local  clusterIP: None  selector:    app: kafka---apiVersion: v1kind: Servicemetadata:  name: broker-0  namespace: kafkaspec:  type: NodePort  ports:  - port: 9093    nodePort: 30093  selector:    kafka-set-component: kafka-0---apiVersion: v1kind: Servicemetadata:  name: broker-1  namespace: kafkaspec:  type: NodePort  ports:  - port: 9093    nodePort: 30094  selector:    kafka-set-component: kafka-1---apiVersion: v1kind: Servicemetadata:  name: broker-2  namespace: kafkaspec:  type: NodePort  ports:  - port: 9093    nodePort: 30095  selector:    kafka-set-component: kafka-2

Notes: - If you are running on GKE:

1. YOUR_KUBERNETES_CLUSTER_EXTERNAL_IP which declared in the server.properties init can be found viagcloud compute instances list
2. Also you must give permission to the firewall gcloud compute firewall-rules create kafka-external --allow tcp:30093,tcp:30094,tcp:30095