Grafana HTTP Error Bad Gateway and Templating init failed errors

In the HTTP settings of Grafana you set Access to Proxy, which means that Grafana wants to access Prometheus. Since Kubernetes uses an overlay network, it is a different IP.

There are two ways of solving this:

1. Set Access to Direct, so the browser directly connects to Prometheus.
2. Use the Kubernetes-internal IP or domain name. I don't know about the Prometheus Helm-chart, but assuming there is a Service named prometheus, something like http://prometheus:9090 should work.

I turned off the firewall on appliance, post that adding http://prometheus:9090 on URL did not throw bad gateway error.

I was never able to find a "proper" fix, but I found a workaround:

apiVersion: v1kind: Servicemetadata:  labels:    prometheus: k8s  name: prometheus-k8s  namespace: monitoringspec:  selector:    app: prometheus    prometheus: k8s  sessionAffinity: ClientIP  clusterIP: None

By setting the clusterIP to None, the service changes to "Headless" mode, which means that requests are sent directly to a random one of the pods in that service/cluster. More info here: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

There's probably a better solution, but this is the only one I've found that actually works for me, with kube-prometheus. (I've tried docker-desktop, k3d, and kind, and all of them have the same issue, so I doubt it's the emulator's fault; and I stripped my config down to basically just kube-prometheus, so it's hard to understand where the problem lies, but oh well.)