Skip to content

feat(seer): Record prompt version on night shift deliveries - #123407

Merged
trevor-e merged 1 commit into
masterfrom
feat/night-shift-prompt-version
Sep 2, 2026
Merged

feat(seer): Record prompt version on night shift deliveries#123407
trevor-e merged 1 commit into
masterfrom
feat/night-shift-prompt-version

Conversation

@trevor-e

@trevor-e trevor-e commented Sep 2, 2026

Copy link
Copy Markdown
Member

Night shift triage results can now be attributed to the prompt revision that produced them. deliver_feature_result accepts an optional Seer-reported prompt_version string, and the night shift handler records it in the shard's extras — on every delivery, errors included, since failed runs write no result rows — and denormalized onto each SeerNightShiftRunResult row, so verdict quality and error rates can be grouped by prompt version without a join. The smart_assignment and autofix handlers accept the kwarg and ignore it for now.

This ships ahead of the Seer-side change, which will add a PROMPT_VERSION constant to the night shift feature and pass it on result push. Deploy order matters: Seer must not send the kwarg until this is deployed, as the RPC dispatch would reject the unknown argument and lose the delivered result. Older Seer versions simply omit it and nothing is recorded.

deliver_feature_result now accepts an optional Seer-reported prompt_version string. The night shift handler records it on the shard's extras (every delivery, errors included, since failed runs write no result rows) and denormalized onto each verdict row so quality and error rates can be grouped by prompt version. Other feature handlers accept the kwarg and ignore it.
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Sep 2, 2026
@trevor-e
trevor-e marked this pull request as ready for review September 2, 2026 13:57
@trevor-e
trevor-e requested a review from a team as a code owner September 2, 2026 13:57
Comment on lines +75 to +82
if prompt_version or error:
extras = {**(shard.extras or {})}
if prompt_version:
extras["prompt_version"] = prompt_version
if error:
extras["error_type"] = SeerNightShiftRunErrorType.SHARD_DELIVERY_FAILED.value
extras["error_message"] = error
shard.update(extras=extras)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: A stale read of shard.extras between two update() calls can cause the prompt_version to be lost when clearing a pre-existing error message on a shard.
Severity: MEDIUM

Suggested Fix

Refresh the shard object from the database after the first update() call by calling shard.refresh_from_db(). This will ensure that the in-memory shard.extras is up-to-date before the logic that decides whether to perform the second update, preventing the stale data from overwriting the prompt_version.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/sentry/seer/night_shift/delivery.py#L75-L82

Potential issue: The function `deliver_night_shift_result` can perform two separate
`shard.update()` calls on the `extras` field. The first update, which may add a
`prompt_version`, modifies the database directly but does not refresh the in-memory
`shard` object. If the shard had a pre-existing error, a second update is triggered to
clear it. This second update reads the stale `shard.extras` from the in-memory object,
which lacks the new `prompt_version`. It then writes a modified version back to the
database, overwriting the `extras` field and silently deleting the `prompt_version` that
was just added. This occurs during retry scenarios where a failed delivery is followed
by a successful one.

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My clanker doesn't think this is an issue with Sentry's version of BaseModel.

@trevor-e
trevor-e merged commit a02a40a into master Sep 2, 2026
67 checks passed
@trevor-e
trevor-e deleted the feat/night-shift-prompt-version branch September 2, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants