AlertManager is not forwarding alerts to webhook receiver AlertManager is not forwarding alerts to webhook receiver kubernetes kubernetes

AlertManager is not forwarding alerts to webhook receiver


I was able to find the root cause of the issue. actually root causes. There were two problems:

  1. I was using webhook to integrate with a Discord channel, which I later learned is not straightforward. A middle layer is required to parse and forward webhook alerts to Discord in a compatible template. A good solution is already mentioned in the Prometheus documentation, which points to alertmanager-discord application. I used the docker image for it to create a deployment and a service which bridged alertmanager to discord.

  2. The operator was adding an additional namepsace label matcher in the top most alert route. So I added the same label to the alerts I created. I used this Routing Tree editor to visualize the routes and make sure the given set of labels matches a route.

apiVersion: apps/v1kind: Deploymentmetadata:  name: alertmanager-discordspec:  selector:    matchLabels:      app: alertmanager-discord  replicas: 1  template:    metadata:      labels:        app: alertmanager-discord    spec:      containers:      - name: alertmanager-discord        image: benjojo/alertmanager-discord        resources:          limits:            memory: "128Mi"            cpu: "500m"        ports:        - containerPort: 9094        env:          - name: DISCORD_WEBHOOK            value: {{ .Values.webhookURL }}---apiVersion: v1kind: Servicemetadata:  name: alertmanager-discordspec:  selector:    app: alertmanager-discord  ports:  - port: 9094    targetPort: 9094  type: ClusterIP---apiVersion: monitoring.coreos.com/v1alpha1kind: AlertmanagerConfigmetadata:  name: alertmanagerspec:  receivers:  - name: discord    webhookConfigs:    - url: 'http://alertmanager-discord:9094'      sendResolved: true...