feat: add support for alert auto-resolve + Incident.io integration

[wrn14897]

Plus fixed 'group-by' alert state issues (alert histories)

Ref: HDX-2661
Ref: HDX-2660

[changeset-bot]

🦋 Changeset detected

Latest commit: 3f3231c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/common-utils	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
hyperdx-v2-oss-app	Ready	Preview	Comment	Nov 4, 2025 10:53pm

[claude]

Code Review

Critical Issues

❌ Missing index for group-by queries → Add compound index { alert: 1, group: 1, createdAt: -1 } to AlertHistory model

• The getPreviousAlertHistories aggregation groups by both alert AND group (line 627-631 in index.ts)
• Current index only covers { alert: 1, createdAt: -1 } (alertHistory.ts:58)
• For group-by alerts with many groups, MongoDB will need to scan all alert histories and do an in-memory group, degrading performance

Important Issues

⚠️ Potential race condition in skip logic → Consider adding alert-level locking or idempotency keys

• Lines 209-222 check if any history exists in current window to skip processing
• If multiple workers process the same alert concurrently, both could pass the skip check before either creates history
• This could result in duplicate notifications being sent

⚠️ AlertInput.id field lacks validation → Add Zod schema validation or document usage

• Added optional id?: string field to AlertInput type (controllers/alerts.ts:21)
• No validation ensuring it is a valid ObjectId format
• Usage in template.ts:267 assumes it exists without null check

Minor Observations

✅ Test coverage is extensive (1845 lines added to checkAlerts.test.ts)
✅ Auto-resolve logic for missing groups is well-implemented
✅ Good separation of concerns with composite keys for grouped alerts
✅ IncidentIO integration follows existing webhook pattern

Recommendation

Fix the missing compound index before merge - this is a performance issue that will impact production workloads with group-by alerts.

[github-actions]

E2E Test Results

✅ All tests passed • 39 passed • 3 skipped • 304s

Status	Count
✅ Passed	39
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	3

View full report →

[wrn14897]

need to remap the status (https://api-docs.incident.io/tag/Alert-Events-V2#operation/Alert%20Events%20V2_CreateHTTP!path=status&t=request)

[pulpdrew]

Besides the other comments, I also notice that since we create an alert history for each group, the Alerts page has an alert status per group, which is probably not correct:

For example, if there are three group keys, you'll get three red bars here for the same time and with different counts.

Just a thought - would it make more sense to save one alert history with all the groups, and maintain the state of each group within the one history entry? Sort of like how we have lastValues already? That could cut down on requests to mongo, help avoid making changes to the UI to fix the above ^^ issue, and maybe simplify some of the other code (particularly the stuff that handles partial mongo save failures)?

[pulpdrew]

LGTM, with the understanding that there are a few followup items listed in various comments

[wrn14897]

LGTM, with the understanding that there are a few followup items listed in various comments

Thanks. I will file tickets for those followup items