Kafka listeners are wrong in the confluentic kubernetes setup

Kafka has a lot of components like Headless Services, Statefulsets and each one has a distinctive role.For that reason I'd suggest too the usage of Kafka Confluentic Helm Chart.

This guide is based on the helm chart since you mentioned you'd use it in the comments but the concepts here can be extended to any application that uses headless services and need external access.

For what you provided, I believe you are facing some difficulties because you are referencing a headless service externally, which will not work since the headless service does not have an internal operational IP.

The Headless Service is created alongside the StatefulSet. The created service will not be given a clusterIP, but will instead simply include a list of Endpoints. These Endpoints are then used to generate instance-specific DNS records in the form of: <StatefulSet>-<Ordinal>.<Service>.<Namespace>.svc.cluster.local

It creates a DNS name for each pod, e.g:

[ root@curl:/ ]$ nslookup my-confluent-cp-kafka-headlessServer:    10.0.0.10Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.localName:      my-confluent-cp-kafka-headlessAddress 1: 10.8.0.23 my-confluent-cp-kafka-1.my-confluent-cp-kafka-headless.default.svc.cluster.localAddress 2: 10.8.1.21 my-confluent-cp-kafka-0.my-confluent-cp-kafka-headless.default.svc.cluster.localAddress 3: 10.8.3.7 my-confluent-cp-kafka-2.my-confluent-cp-kafka-headless.default.svc.cluster.local

• This is what makes this services connect to each other inside the cluster.

• You can't, therefore, expose cp-kafka:9092 which is the headless service, also only used internally, as I explained above.

• In order to get outside access you have to set the parameters nodeport.enabled to true as stated here: External Access Parameters.
• It adds one service to each kafka-N pod during chart deployment.
• Note that the service created has the selector statefulset.kubernetes.io/pod-name: demo-cp-kafka-0 this is how the service identifies the pod it is intended to connect to.

Reproduction:

• git clone https://github.com/confluentinc/cp-helm-charts.git
• edit the file cp-helm-charts/cp-kafka/values.yaml changing the nodeport from false to true and change the ports as you'd like:

nodeport:  enabled: true  servicePort: 19092  firstListenerPort: 31090

• Deploy the chart:

$ helm install demo cp-helm-charts$ kubectl get podsNAME                                       READY   STATUS    RESTARTS   AGEdemo-cp-control-center-6d79ddd776-ktggw    1/1     Running   3          113sdemo-cp-kafka-0                            2/2     Running   1          113sdemo-cp-kafka-1                            2/2     Running   0          94sdemo-cp-kafka-2                            2/2     Running   0          84sdemo-cp-kafka-connect-79689c5c6c-947c4     2/2     Running   2          113sdemo-cp-kafka-rest-56dfdd8d94-79kpx        2/2     Running   1          113sdemo-cp-ksql-server-c498c9755-jc6bt        2/2     Running   2          113sdemo-cp-schema-registry-5f45c498c4-dh965   2/2     Running   3          113sdemo-cp-zookeeper-0                        2/2     Running   0          112sdemo-cp-zookeeper-1                        2/2     Running   0          93sdemo-cp-zookeeper-2                        2/2     Running   0          74s$ kubectl get svcNAME                         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGEdemo-cp-control-center       ClusterIP   10.0.13.134   <none>        9021/TCP            50mdemo-cp-kafka                ClusterIP   10.0.15.71    <none>        9092/TCP            50mdemo-cp-kafka-0-nodeport     NodePort    10.0.7.101    <none>        19092:31090/TCP     50mdemo-cp-kafka-1-nodeport     NodePort    10.0.4.234    <none>        19092:31091/TCP     50mdemo-cp-kafka-2-nodeport     NodePort    10.0.3.194    <none>        19092:31092/TCP     50mdemo-cp-kafka-connect        ClusterIP   10.0.3.217    <none>        8083/TCP            50mdemo-cp-kafka-headless       ClusterIP   None          <none>        9092/TCP            50mdemo-cp-kafka-rest           ClusterIP   10.0.14.27    <none>        8082/TCP            50mdemo-cp-ksql-server          ClusterIP   10.0.7.150    <none>        8088/TCP            50mdemo-cp-schema-registry      ClusterIP   10.0.7.84     <none>        8081/TCP            50mdemo-cp-zookeeper            ClusterIP   10.0.9.119    <none>        2181/TCP            50mdemo-cp-zookeeper-headless   ClusterIP   None          <none>        2888/TCP,3888/TCP   50m

• My Node is on IP 35.226.189.123 and I'll connect to the demo-cp-kafka-0-nodeport nodeport service which is on port 31090, now let's try to connect from outside the cluster. For that I'll connect to another VM where I have a minikube, so I can use kafka-client pod to test:

user@minikube:~$ kubectl get podsNAME           READY   STATUS    RESTARTS   AGEkafka-client   1/1     Running   0          17huser@minikube:~$ kubectl exec kafka-client -it -- bin/bashroot@kafka-client:/# kafka-console-consumer --bootstrap-server 35.226.189.123:31090 --topic demo-topic --from-beginning --timeout-ms 8000 --max-messages 1Wed Apr 15 18:19:48 UTC 2020Processed a total of 1 messagesroot@kafka-client:/# 

As you can see, I was able to access the kafka from outside.

• Using this method, the helm chart will create 1 external service for each replica you define.
• If you need external access to Zookeeper it's not automatically provisioned like the kafka agent, but I'll leave a service model for you:

zookeeper-external-0.yaml

apiVersion: v1kind: Servicemetadata:  labels:    app: cp-zookeeper    pod: demo-cp-zookeeper-0  name: demo-cp-zookeeper-0-nodeport  namespace: defaultspec:  externalTrafficPolicy: Cluster  ports:  - name: external-broker    nodePort: 31181    port: 12181    protocol: TCP    targetPort: 31181  selector:    app: cp-zookeeper    statefulset.kubernetes.io/pod-name: demo-cp-zookeeper-0  sessionAffinity: None  type: NodePort

• It will create a service for it:

NAME                           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGEdemo-cp-zookeeper-0-nodeport   NodePort    10.0.5.67     <none>        12181:31181/TCP     2s

• Test it with your external IP:

pod/zookeeper-client createduser@minikube:~$ kubectl exec -it zookeeper-client -- /bin/bashroot@zookeeper-client:/# zookeeper-shell 35.226.189.123:31181Connecting to 35.226.189.123:31181Welcome to ZooKeeper!JLine support is disabled

If you have any doubts, let me know in the comments!

If you used the Confluent Helm Charts and read through the docs there, then you can configure different, functional options for remote listeners.

Also, I'd suggest using an Operator rather than a simple Deployment https://operatorhub.io/?keyword=kafka

Alternatively, use Docker Compose if you are just on a single machine