Matrix-Alertmanager

A bot to receive Prometheus Alertmanager webhook events and forward them to chosen rooms.

Main features:

• Uses pre-created Matrix user to send alerts using token auth.
• Configurable room per alert receiver.
• Automatic joining of configured rooms. Private rooms require an invite.
• Secret key authentication with Alertmanager.
• HTML formatted messages.
• Optionally mentions @room on firing alerts

How to use

Configuration

Whether running manually or via the Docker image, the configuration is set via environment variables. When running manually, copy .env.default into .env, set the values and they will be loaded automatically. When using the Docker image, set the environment variables when running the container.

Docker

The Docker image jaywink/matrix-alertmanager:latest is the easiest way to get the service running. Ensure you set the required environment variables listed in .env.default in this repository.

Alertmanager

You will need to configure a webhook receiver in Alertmanager. It should looks something like this:

receivers:
- name: 'myreceiver'
  webhook_configs:
  - url: 'https://my-matrix-alertmanager.tld/alerts'
    http_config:
      authorization:
        type: Bearer
        credentials: 'veryverysecretkeyhere'

The secret key obviously should match the one in the alertmanager configuration.

The configuration above will pass the secret as an Authorization header bearer token, alternatively you can pass it as a query parameter secret, but if you do it that way then it is not redacted from the Alertmanager web UI so this is not really recommended.

Prometheus rules

Add some styling to your prometheus rules

rules:
- alert: High Memory Usage of Container
  annotations:
    description: Container named <strong>{{\$labels.container_name}}</strong> in <strong>{{\$labels.pod_name}}</strong> in <strong>{{\$labels.namespace}}</strong> is using more than 75% of Memory Limit
  expr: |
    ((( sum(container_memory_usage_bytes{image!=\"\",container_name!=\"POD\", namespace!=\"kube-system\"}) by (namespace,container_name,pod_name, instance)  / sum(container_spec_memory_limit_bytes{image!=\"\",container_name!=\"POD\",namespace!=\"kube-system\"}) by (namespace,container_name,pod_name, instance) ) * 100 ) < +Inf ) > 75
  for: 5m
  labels:
    team: dev

NOTE! Currently the bot cannot talk HTTPS, so you need to have a reverse proxy in place to terminate SSL, or use unsecure unencrypted connections.

TODO

• Registering an account instead of having to use an existing account

Tech

Node 14, Express, Matrix JS SDK

Help

Come chat in the https://matrix.to/#/#matrix-alertmanager:federator.dev room!

Author

Jason Robinson / https://jasonrobinson.me / @jaywink:federator.dev

License

MIT