AlertManager is not forwarding alerts to webhook receiver
I was able to find the root cause of the issue. actually root causes. There were two problems:
I was using webhook to integrate with a Discord channel, which I later learned is not straightforward. A middle layer is required to parse and forward webhook alerts to Discord in a compatible template. A good solution is already mentioned in the Prometheus documentation, which points to alertmanager-discord application. I used the docker image for it to create a deployment and a service which bridged alertmanager to discord.
The operator was adding an additional
namepsace
label matcher in the top most alert route. So I added the same label to the alerts I created. I used this Routing Tree editor to visualize the routes and make sure the given set of labels matches a route.
apiVersion: apps/v1kind: Deploymentmetadata: name: alertmanager-discordspec: selector: matchLabels: app: alertmanager-discord replicas: 1 template: metadata: labels: app: alertmanager-discord spec: containers: - name: alertmanager-discord image: benjojo/alertmanager-discord resources: limits: memory: "128Mi" cpu: "500m" ports: - containerPort: 9094 env: - name: DISCORD_WEBHOOK value: {{ .Values.webhookURL }}---apiVersion: v1kind: Servicemetadata: name: alertmanager-discordspec: selector: app: alertmanager-discord ports: - port: 9094 targetPort: 9094 type: ClusterIP---apiVersion: monitoring.coreos.com/v1alpha1kind: AlertmanagerConfigmetadata: name: alertmanagerspec: receivers: - name: discord webhookConfigs: - url: 'http://alertmanager-discord:9094' sendResolved: true...