How to get a pod's labels in Prometheus when pulling the metrics from Kube State Metrics
Solution: Using PromQL you can do group by. So in my prometheus-rules.yaml, I changed this:
expr: kube_pod_status_phase{phase="Failed"} > 0to this:
expr: kube_pod_status_phase{phase="Failed"} * on (pod,namespace) group_right kube_pod_labels > 0So my new alert rule looks like this:
- name: Pod_Failed rules: - alert: pod_failed expr: kube_pod_status_phase{phase="Failed"} * on (pod,namespace) group_right kube_pod_labels > 0 labels: appname: '{{ $labels.label_APP }}' # This is what I wanted to capture teamname: '{{ $labels.label_TEAM }}' # This is what I wanted to capture annotations: summary: 'Pod: {{ $labels.pod }} is down' description: 'Pod: {{ $labels.pod }} is down in {{ $labels.namespace }} namespace.'