feat: implement phantom_threading to group email alerts into threads #4623
+33
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stapelberg
force-pushed
the
threading
branch
2 times, most recently
from
ba09364 to
dd8d44b
Compare
Some email clients such as Gmail apparently use their own heuristics for threading and already implement this behavior based on the subject. But for users of other email clients that only implement threading based on the relevant headers (e.g. notmuch), those users currently get one email thread for each newly firing alert. With phantom_threading enabled, all alert emails (of the same alert) on the same day are grouped into the same thread. Much nicer :) Signed-off-by: Michael Stapelberg <[email protected]>
sysadmind
reviewed
Oct 24, 2025
| if n.conf.PhantomThreading && len(as) > 0 { | ||
| // Add threading headers. All notifications for the same alert | ||
| // (identified by fingerprint) on the same day are threaded together. | ||
| // The thread root ID is a phantom Message-ID that doesn't correspond to |
There was a problem hiding this comment.
I assume this is where the name "Phantom Threading" comes from? I'm not sure people will understand that based on the name. Maybe "daily threading" would make more sense, although in that case what if we want to do a different threading strategy in the future?
| } | ||
|
|
||
| // Use first alert as representative of the alert group. | ||
| alert := alerts[0] |
There was a problem hiding this comment.
Doesn't that mean even if we have 30 alerts, whatever the first one is, that's what will be used for fingerprinting? What if that alert resolves, but the other 29 remain?
Spaceman1701
reviewed
Nov 3, 2025
There was a problem hiding this comment.
Hi, this is an interesting change! Threading for alertmanager notifications would be very useful for teams using email!
I have a couple questions:
- Did you consider using the
GroupKeyhas to source for the thread root id? If I understand the rest of this change correct, this would result in one thread per group. It's also consistent with how we typically group/dedup notifications. For example, the PagerDuty integration uses the hash of the group key as the PagerDuty dedup key: https://github.com/prometheus/alertmanager/blob/main/notify/pagerduty/pagerduty.go#L241. This would also make threading configured directly by routing config. - It looks like we end up just setting some headers on the email. Is this something that could be implemented by templated header? I'm also wondering if this has any strange interaction with the user's config if they set headers.
- The comment mentions email clients that use "
(commonly used) JWZ" - do you know how this change interacts with email clients that don't behave that way?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some email clients such as Gmail apparently use their own heuristics for threading and already implement this behavior based on the subject.
But for users of other email clients that only implement threading based on the relevant headers (e.g. notmuch), those users currently get one email thread for each newly firing alert.
With phantom_threading enabled, all alert emails (of the same alert) on the same day are grouped into the same thread. Much nicer :)
I have tested this manually and you can see the effect start to work in this screenshot:
(Monday morning, I got one thread per alert email notification; in the evening, the threading change was effective and emails are grouped into the daily thread.)