HPA using Kafka Exporter in on premise Kubernetes cluster

In order to scale based on custom metrics, Kubernetes needs to query an API for metrics to check for those metrics. That API needs to implement the custom metrics interface.

So for Prometheus, you need to setup an API that exposes Prometheus metrics through the custom metrics API. Luckily, there already is an adapter.

When I implemented Kubernetes HPA using Metrics from Kafka-exporter I had a few setbacks which I solved doing the following:

1. I deployed the kafka-exporter container as a sidecar to the pods Iwanted to scale. I found that the HPA scales the pod it gets themetrics from.

2. I used annotations to make Prometheus scrape the metrics from the pods with exporter.

3. Then I verified that the kafka-exporter metrics are getting to Prometheus. If it's not there you can't advance further.

4. I deployed prometheus adapter using its helm chart. The adapter will "translate" Prometheus's metrics into custom MetricsApi, which will make it visible to HPA.
5. I made sure that the metrics are visible in k8s by executing kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 from one ofthe master nodes.
6. I created an hpa with the matching metric name.

Here is a complete guide explaining how to implement Kubernetes HPA using Metrics from Kafka-exporter

Please comment if you have more questions