Skip to content

detect-backport-regressions ci plugin command.#431

Open
jogeo wants to merge 2 commits into
openshift-eng:mainfrom
jogeo:detect-backport-regressions
Open

detect-backport-regressions ci plugin command.#431
jogeo wants to merge 2 commits into
openshift-eng:mainfrom
jogeo:detect-backport-regressions

Conversation

@jogeo
Copy link
Copy Markdown
Contributor

@jogeo jogeo commented Apr 22, 2026

What this PR does / why we need it:

Detect regressions that have cascaded from newer releases to older releases due to problematic backports

Summary by CodeRabbit

  • New Features

    • Added CI command /ci:detect-backport-regressions with options to constrain release range and analysis window.
    • Added cascade analysis capabilities: potential-cascade detection, similarity-based confirmation, and interactive report generation (HTML/Markdown/JSON).
  • Bug Fixes

    • Fixed regression filtering to preserve test identification data in outputs.
  • Documentation

    • Added comprehensive docs for the backport-regression detection workflow and analysis skills.
  • Chores

    • Bumped CI and Teams plugin versions.

@openshift-ci openshift-ci Bot requested review from bryan-cox and enxebre April 22, 2026 20:52
@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented Apr 22, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jogeo
Once this PR has been reviewed and has the lgtm label, please assign cblecker for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 22, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 426847fe-fe8a-4f8b-977c-a9aea3ae04cc

📥 Commits

Reviewing files that changed from the base of the PR and between 1b06dfb and f336f9f.

📒 Files selected for processing (2)
  • plugins/ci/.claude-plugin/plugin.json
  • plugins/teams/.claude-plugin/plugin.json
✅ Files skipped from review due to trivial changes (2)
  • plugins/teams/.claude-plugin/plugin.json
  • plugins/ci/.claude-plugin/plugin.json

Walkthrough

Adds a new CI command /ci:detect-backport-regressions and four supporting CI skills to detect and confirm backport-driven regressions, plus report-generation docs; updates two plugin version manifests and a small teams helper to stop removing test_id.

Changes

Cohort / File(s) Summary
Command doc
plugins/ci/commands/detect-backport-regressions.md
New documentation for the /ci:detect-backport-regressions command describing the 3-step orchestration, CLI flags, outputs, severity/similarity thresholds, and prerequisites.
CI Skills (detection & analysis)
plugins/ci/skills/detect-potential-cascades/SKILL.md, plugins/ci/skills/analyze-cascade-similarity/SKILL.md, plugins/ci/skills/detect-backport-regressions/SKILL.md
New skill specifications for detecting potential cascades, validating similarity via Prow-job analyses, and orchestrating the overall detection workflow with subagent contracts and JSON outputs.
Report generation skill
plugins/ci/skills/generate-cascade-report/SKILL.md
New skill doc specifying HTML/Markdown/JSON report templates, severity/similarity presentation rules, and report generation control flow.
Plugin manifests / registry
docs/data.json, PLUGINS.md
Adds detect-backport-regressions command and four skills to docs/data.json; documents the new command in PLUGINS.md.
Plugin version bumps
plugins/ci/.claude-plugin/plugin.json, plugins/teams/.claude-plugin/plugin.json
Incremented plugin version fields: CI 0.0.380.0.39, Teams 0.0.130.0.14.
Teams helper tweak
plugins/teams/skills/list-regressions/list_regressions.py
Adjusted remove_unnecessary_fields to remove only links and preserve test_id; updated docstring accordingly.

Sequence Diagram(s)

sequenceDiagram
  participant User as User / CLI
  participant Orchestrator as CI Orchestrator (/ci:detect-backport-regressions)
  participant Detector as Detect Potential Cascades skill
  participant Analyzer as Analyze Cascade Similarity skill
  participant Prow as Prow Job Analyzer (subagents)
  participant Reporter as Generate Cascade Report skill
  User->>Orchestrator: invoke /ci:detect-backport-regressions [flags]
  Orchestrator->>Detector: run detect-potential-cascades (write potential_cascades.json)
  Detector-->>Orchestrator: potential_cascades.json
  Orchestrator->>Analyzer: run analyze-cascade-similarity (spawn subagents)
  Analyzer->>Prow: launch prow-job-analyze-test-failure subagents (parallel)
  Prow-->>Analyzer: ANALYSIS_RESULT blocks
  Analyzer-->>Orchestrator: confirmed_cascades.json
  Orchestrator->>Reporter: run generate-cascade-report (--format)
  Reporter-->>User: HTML/Markdown/JSON report
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 10
✅ Passed checks (10 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and directly describes the main change: a new CI plugin command for detecting backport regressions, which aligns with the PR's primary objective and the majority of the changeset.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
No Real People Names In Style References ✅ Passed Thorough examination of all added and modified files found no references to real people's names used as style references in documentation, commands, skills, or code.
No Assumed Git Remote Names ✅ Passed Comprehensive search found no hardcoded git remote names (origin/upstream) used without discovery in PR files.
Git Push Safety Rules ✅ Passed The pull request introduces documentation-only changes (SKILL.md files) and a minor Python code modification to list_regressions.py that only changes field filtering logic. No git push operations, force pushes, or pushes to main/master branches are present in any of the files added or modified by this PR. The existing git push references found in the codebase (in other files like address-reviews.md, coderabbit-rules-from-pr-reviews.md, etc.) are pre-existing in the repository and not introduced by this pull request.
No Untrusted Mcp Servers ✅ Passed PR introduces only documentation and metadata updates with no MCP server installations, npm packages, or untrusted dependencies.
Ai-Helpers Overlap Detection ✅ Passed All five new files introduce genuinely new functionality with no overlapping content or competing implementations in the repository.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/data.json`:
- Around line 363-367: The command metadata for detect-backport-regressions is
missing the documented flags; update the JSON entry for
"detect-backport-regressions" (the object containing "name":
"detect-backport-regressions", "argument_hint" and "synopsis") to include the
missing options --format, --analyze-similarity and --similarity-threshold in
both the "argument_hint" and "synopsis" strings so generated help matches the
documented flags and usage.

In `@plugins/ci/commands/detect-backport-regressions.md`:
- Line 3: Update the CLI docs in
plugins/ci/commands/detect-backport-regressions.md so the argument-hint,
synopsis, examples, and per-argument definitions are consistent: add the missing
options (--format, --analyze-similarity, --similarity-threshold) to the
argument-hint and synopsis, and normalize the representation of
--exclude-install and --exclude-monitor across synopsis/examples/argument text
(choose either flag style or value-style and apply it everywhere); verify the
examples and the argument descriptions reflect the chosen style and the same
default/expected values for these options.
- Around line 134-136: Update the fenced code blocks in
plugins/ci/commands/detect-backport-regressions.md for each example of the
/ci:detect-backport-regressions command (including variants like "--component
kube-apiserver --lookback 6", "--min-cascade 2", etc.) to include a language
identifier (use "bash") after the opening triple backticks so the examples
comply with MD040; add ```bash instead of ``` for the blocks referenced (lines
around the shown examples and the additional ranges 139-141, 145-146, 150-151,
155-156, 160-161, 165-166, 170-171).

In `@plugins/ci/skills/analyze-cascade-similarity/SKILL.md`:
- Around line 243-248: The current early-return for low confidence incorrectly
sets "same_root_cause": False; change this branch to avoid auto-classifying as
false-positive by returning an indeterminate value (e.g., "same_root_cause":
None or omitting the key) while preserving "similarity_score" and a descriptive
"reason" like "Low confidence in one or both analyses" so semantics match the
later non-classification behavior (see the handling around lines 421-424);
update the low-confidence branch that produces the dict with keys
same_root_cause, similarity_score, and reason to reflect this indeterminate
result instead of False.
- Around line 209-211: Replace the brittle selection of origin_release =
list(results.keys())[0] with an explicit determination of the origin: either
accept an explicit origin identifier/parameter or compute the earliest release
from results by sorting keys using a proper chronological ordering (e.g.,
semantic-version or timestamp from each results[release] entry) and then assign
origin_release to that computed value; update origin_result =
results[origin_release] accordingly so origin selection no longer depends on map
iteration order.
- Around line 92-113: The fenced prompt block in SKILL.md (the Analyze the test
failure ... ANALYSIS_RESULT block including placeholders like {test_name},
{prow_url}, and {release}) is missing a language identifier; update the opening
triple-backtick to include a language token (e.g., ```text) so the block becomes
"```text" and keep the rest of the block unchanged, ensuring the closing "```"
remains present.

In `@plugins/ci/skills/detect-backport-regressions/SKILL.md`:
- Around line 41-45: The prerequisites list omits the `fetch-test-runs` skill
required by the workflow; update the "Installed plugins" bullet under the
prerequisites to include `fetch-test-runs` alongside `ci` and `teams` so lines
that call fetch-test-runs (e.g., references around "Line 80" / the
fetch-test-runs requirement) are satisfied; specifically add "`-
`fetch-test-runs` (for fetch-test-runs)`" to the installed plugins block so
users following the doc end-to-end won't miss this dependency.
- Around line 92-100: The three fenced code blocks containing the
ANALYSIS_RESULT block, the "ERROR: Could not auto-detect current development
release" message, and the "✅ No cascading regressions detected - system is
healthy!" message should include a language identifier (e.g., text) after the
opening triple backticks; update the fences around the ANALYSIS_RESULT block,
the auto-detect error block, and the healthy status block in SKILL.md to use
```text (or another appropriate language) so markdownlint rule MD040 is
satisfied.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5a6ea192-8e56-4073-bb55-6b9ae9a2d7c8

📥 Commits

Reviewing files that changed from the base of the PR and between ef00907 and 1b06dfb.

📒 Files selected for processing (8)
  • PLUGINS.md
  • docs/data.json
  • plugins/ci/commands/detect-backport-regressions.md
  • plugins/ci/skills/analyze-cascade-similarity/SKILL.md
  • plugins/ci/skills/detect-backport-regressions/SKILL.md
  • plugins/ci/skills/detect-potential-cascades/SKILL.md
  • plugins/ci/skills/generate-cascade-report/SKILL.md
  • plugins/teams/skills/list-regressions/list_regressions.py

Comment thread docs/data.json
Comment on lines +363 to +367
"argument_hint": "[--current-release <version>] [--lookback N] [--days N]",
"description": "Detect regressions that have cascaded from newer releases to older releases due to problematic backports",
"name": "detect-backport-regressions",
"synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install] [--exclude-monitor] [--component <name>] [--min-cascade N] [--include-resolved]"
},
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Keep command metadata in sync with documented options.
The detect-backport-regressions metadata omits documented flags (--format, --analyze-similarity, --similarity-threshold), which makes generated command help incomplete.

Suggested manifest update
-          "argument_hint": "[--current-release <version>] [--lookback N] [--days N]",
+          "argument_hint": "[--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]",
@@
-          "synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install] [--exclude-monitor] [--component <name>] [--min-cascade N] [--include-resolved]"
+          "synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"argument_hint": "[--current-release <version>] [--lookback N] [--days N]",
"description": "Detect regressions that have cascaded from newer releases to older releases due to problematic backports",
"name": "detect-backport-regressions",
"synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install] [--exclude-monitor] [--component <name>] [--min-cascade N] [--include-resolved]"
},
"argument_hint": "[--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]",
"description": "Detect regressions that have cascaded from newer releases to older releases due to problematic backports",
"name": "detect-backport-regressions",
"synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]"
},
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/data.json` around lines 363 - 367, The command metadata for
detect-backport-regressions is missing the documented flags; update the JSON
entry for "detect-backport-regressions" (the object containing "name":
"detect-backport-regressions", "argument_hint" and "synopsis") to include the
missing options --format, --analyze-similarity and --similarity-threshold in
both the "argument_hint" and "synopsis" strings so generated help matches the
documented flags and usage.

@@ -0,0 +1,251 @@
---
description: Detect regressions that have cascaded from newer releases to older releases due to problematic backports
argument-hint: "[--current-release <version>] [--lookback N] [--days N]"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Unify the CLI contract across argument-hint, synopsis, and argument definitions.
The documented option set is internally inconsistent: Line 13 omits --format, --analyze-similarity, and --similarity-threshold despite Lines 208-228 documenting them; and --exclude-install/--exclude-monitor are shown as bare flags in synopsis but value-style in examples/argument text.

Suggested doc contract alignment
-argument-hint: "[--current-release <version>] [--lookback N] [--days N]"
+argument-hint: "[--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]"
-/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install] [--exclude-monitor] [--component <name>] [--min-cascade N] [--include-resolved]
+/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]

Also applies to: 13-14, 188-194, 208-228

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@plugins/ci/commands/detect-backport-regressions.md` at line 3, Update the CLI
docs in plugins/ci/commands/detect-backport-regressions.md so the argument-hint,
synopsis, examples, and per-argument definitions are consistent: add the missing
options (--format, --analyze-similarity, --similarity-threshold) to the
argument-hint and synopsis, and normalize the representation of
--exclude-install and --exclude-monitor across synopsis/examples/argument text
(choose either flag style or value-style and apply it everywhere); verify the
examples and the argument descriptions reflect the chosen style and the same
default/expected values for these options.

Comment on lines +134 to +136
```
/ci:detect-backport-regressions
```
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add language identifiers to command example code fences (MD040).
Several fenced blocks in Examples are missing a language specifier.

Suggested markdownlint-compliant fix
-   ```
+   ```bash
    /ci:detect-backport-regressions
    ```
-   ```
+   ```bash
    /ci:detect-backport-regressions --component kube-apiserver --lookback 6
    ```
-   ```
+   ```bash
    /ci:detect-backport-regressions --min-cascade 2
    ```
-   ```
+   ```bash
    /ci:detect-backport-regressions --exclude-install false --days 60
    ```
-   ```
+   ```bash
    /ci:detect-backport-regressions --current-release 4.22 --lookback 3
    ```
-   ```
+   ```bash
    /ci:detect-backport-regressions --include-resolved
    ```
-   ```
+   ```bash
    /ci:detect-backport-regressions --format markdown --days 45
    ```
-   ```
+   ```bash
    /ci:detect-backport-regressions --exclude-monitor false
    ```

Also applies to: 139-141, 145-146, 150-151, 155-156, 160-161, 165-166, 170-171

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@plugins/ci/commands/detect-backport-regressions.md` around lines 134 - 136,
Update the fenced code blocks in
plugins/ci/commands/detect-backport-regressions.md for each example of the
/ci:detect-backport-regressions command (including variants like "--component
kube-apiserver --lookback 6", "--min-cascade 2", etc.) to include a language
identifier (use "bash") after the opening triple backticks so the examples
comply with MD040; add ```bash instead of ``` for the blocks referenced (lines
around the shown examples and the additional ranges 139-141, 145-146, 150-151,
155-156, 160-161, 165-166, 170-171).

Comment on lines +92 to +113
```
Analyze the test failure "{test_name}" in this Prow job: {prow_url}

Use the ci:prow-job-analyze-test-failure skill to perform a thorough root cause analysis.

IMPORTANT:
- Trace to the actual root cause, not just symptoms
- Download log bundles, examine pod logs, cite specific error messages
- Never stop at symptoms like "0 nodes ready", "operator degraded", or "crash-looping"
- Look for the underlying reason (network issues, CNI config loss, registry failures, etc.)

At the end of your analysis, you MUST provide an ANALYSIS_RESULT block in this exact format:

ANALYSIS_RESULT:
- root_cause_summary: <one-line summary of the actual root cause>
- affected_components: <comma-separated list of affected operators/components>
- key_error_patterns: <comma-separated key error strings for matching>
- known_symptoms: <comma-separated symptom summaries, or "none">
- test_name: {test_name}
- confidence_level: <1-5, where 5 is highest confidence in the root cause>
- release: {release}
```
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add a language identifier for the prompt template fence (MD040).
The fenced block beginning at Line 92 should declare a language.

Suggested fix
-```
+```text
 Analyze the test failure "{test_name}" in this Prow job: {prow_url}
@@
-- release: {release}
-```
+- release: {release}
+```
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```
Analyze the test failure "{test_name}" in this Prow job: {prow_url}
Use the ci:prow-job-analyze-test-failure skill to perform a thorough root cause analysis.
IMPORTANT:
- Trace to the actual root cause, not just symptoms
- Download log bundles, examine pod logs, cite specific error messages
- Never stop at symptoms like "0 nodes ready", "operator degraded", or "crash-looping"
- Look for the underlying reason (network issues, CNI config loss, registry failures, etc.)
At the end of your analysis, you MUST provide an ANALYSIS_RESULT block in this exact format:
ANALYSIS_RESULT:
- root_cause_summary: <one-line summary of the actual root cause>
- affected_components: <comma-separated list of affected operators/components>
- key_error_patterns: <comma-separated key error strings for matching>
- known_symptoms: <comma-separated symptom summaries, or "none">
- test_name: {test_name}
- confidence_level: <1-5, where 5 is highest confidence in the root cause>
- release: {release}
```
🧰 Tools
🪛 markdownlint-cli2 (0.22.0)

[warning] 92-92: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@plugins/ci/skills/analyze-cascade-similarity/SKILL.md` around lines 92 - 113,
The fenced prompt block in SKILL.md (the Analyze the test failure ...
ANALYSIS_RESULT block including placeholders like {test_name}, {prow_url}, and
{release}) is missing a language identifier; update the opening triple-backtick
to include a language token (e.g., ```text) so the block becomes "```text" and
keep the rest of the block unchanged, ensuring the closing "```" remains
present.

Comment on lines +209 to +211
# Extract origin release result
origin_release = list(results.keys())[0]
origin_result = results[origin_release]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Select origin release explicitly, not by map iteration order.
Using the “first key” as origin can point comparisons at the wrong baseline when result assembly order differs from release chronology.

Suggested fix
-def compare_root_causes(results: dict) -> dict:
+def compare_root_causes(results: dict, origin_release: str) -> dict:
@@
-    origin_release = list(results.keys())[0]
-    origin_result = results[origin_release]
+    if origin_release not in results:
+        return {
+            "same_root_cause": False,
+            "similarity_score": 0.0,
+            "reason": "Origin release analysis missing"
+        }
+    origin_result = results[origin_release]
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@plugins/ci/skills/analyze-cascade-similarity/SKILL.md` around lines 209 -
211, Replace the brittle selection of origin_release = list(results.keys())[0]
with an explicit determination of the origin: either accept an explicit origin
identifier/parameter or compute the earliest release from results by sorting
keys using a proper chronological ordering (e.g., semantic-version or timestamp
from each results[release] entry) and then assign origin_release to that
computed value; update origin_result = results[origin_release] accordingly so
origin selection no longer depends on map iteration order.

Comment on lines +243 to +248
if conf1 < 3 or conf2 < 3:
return {
"same_root_cause": False,
"similarity_score": 0.0,
"reason": "Low confidence in one or both analyses"
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Do not auto-classify low-confidence comparisons as false positives.
Line 243 currently forces low-confidence cases into same_root_cause: False, which conflicts with Lines 421-424 (“Don’t automatically classify as false positive”).

Suggested fix for consistent classification semantics
     if conf1 < 3 or conf2 < 3:
         return {
             "same_root_cause": False,
             "similarity_score": 0.0,
+            "needs_manual_review": True,
             "reason": "Low confidence in one or both analyses"
         }

Also applies to: 421-424

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@plugins/ci/skills/analyze-cascade-similarity/SKILL.md` around lines 243 -
248, The current early-return for low confidence incorrectly sets
"same_root_cause": False; change this branch to avoid auto-classifying as
false-positive by returning an indeterminate value (e.g., "same_root_cause":
None or omitting the key) while preserving "similarity_score" and a descriptive
"reason" like "Low confidence in one or both analyses" so semantics match the
later non-classification behavior (see the handling around lines 421-424);
update the low-confidence branch that produces the dict with keys
same_root_cause, similarity_score, and reason to reflect this indeterminate
result instead of False.

Comment on lines +41 to +45
1. **Network Access**: Must be able to reach Sippy API endpoints and GCS (for Prow artifacts)
2. **Installed plugins**:
- `ci` plugin (for fetch-releases, fetch-regression-details, prow-job-analyze-test-failure)
- `teams` plugin (for list-regressions)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add missing prerequisite for fetch-test-runs.
Line 80 requires fetch-test-runs, but it is not listed in prerequisites, which can cause setup failures for users following this doc end-to-end.

Suggested doc fix
 2. **Installed plugins**:
-   - `ci` plugin (for fetch-releases, fetch-regression-details, prow-job-analyze-test-failure)
+   - `ci` plugin (for fetch-releases, fetch-regression-details, fetch-test-runs, prow-job-analyze-test-failure)
    - `teams` plugin (for list-regressions)

Also applies to: 79-81

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@plugins/ci/skills/detect-backport-regressions/SKILL.md` around lines 41 - 45,
The prerequisites list omits the `fetch-test-runs` skill required by the
workflow; update the "Installed plugins" bullet under the prerequisites to
include `fetch-test-runs` alongside `ci` and `teams` so lines that call
fetch-test-runs (e.g., references around "Line 80" / the fetch-test-runs
requirement) are satisfied; specifically add "`- `fetch-test-runs` (for
fetch-test-runs)`" to the installed plugins block so users following the doc
end-to-end won't miss this dependency.

Comment on lines +92 to +100
```
ANALYSIS_RESULT:
- root_cause_summary: <one-line summary>
- affected_components: <comma-separated list of affected operators/components>
- key_error_patterns: <comma-separated key error strings for matching>
- known_symptoms: <comma-separated symptom summaries, or "none">
- test_name: <the name of the failing test>
- confidence_level: <1-5, where 5 is highest confidence in the root cause>
```
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Specify languages for fenced code blocks (MD040).
The code fences at Lines 92, 153, and 161 are missing language identifiers.

Suggested markdownlint-compliant fix
-    ```
+    ```text
     ANALYSIS_RESULT:
     - root_cause_summary: <one-line summary>
     - affected_components: <comma-separated list of affected operators/components>
     - key_error_patterns: <comma-separated key error strings for matching>
     - known_symptoms: <comma-separated symptom summaries, or "none">
     - test_name: <the name of the failing test>
     - confidence_level: <1-5, where 5 is highest confidence in the root cause>
     ```
-```
+```text
 ERROR: Could not auto-detect current development release
 HINT: Use --current-release to manually specify (e.g., --current-release 4.22)

```diff
-```
+```text
 ✅ No cascading regressions detected - system is healthy!
</details>


Also applies to: 153-156, 161-163

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 markdownlint-cli2 (0.22.0)</summary>

[warning] 92-92: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

</details>

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against the current code and only fix it if needed.

In @plugins/ci/skills/detect-backport-regressions/SKILL.md around lines 92 -
100, The three fenced code blocks containing the ANALYSIS_RESULT block, the
"ERROR: Could not auto-detect current development release" message, and the "✅
No cascading regressions detected - system is healthy!" message should include a
language identifier (e.g., text) after the opening triple backticks; update the
fences around the ANALYSIS_RESULT block, the auto-detect error block, and the
healthy status block in SKILL.md to use ```text (or another appropriate
language) so markdownlint rule MD040 is satisfied.


</details>

<!-- fingerprinting:phantom:poseidon:hawk:d857237d-3e28-469d-8b04-71c3b6b69b09 -->

<!-- This is an auto-generated comment by CodeRabbit -->

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 13, 2026
@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented May 13, 2026

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant