detect-backport-regressions ci plugin command.#431
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: jogeo The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
✅ Files skipped from review due to trivial changes (2)
WalkthroughAdds a new CI command Changes
Sequence Diagram(s)sequenceDiagram
participant User as User / CLI
participant Orchestrator as CI Orchestrator (/ci:detect-backport-regressions)
participant Detector as Detect Potential Cascades skill
participant Analyzer as Analyze Cascade Similarity skill
participant Prow as Prow Job Analyzer (subagents)
participant Reporter as Generate Cascade Report skill
User->>Orchestrator: invoke /ci:detect-backport-regressions [flags]
Orchestrator->>Detector: run detect-potential-cascades (write potential_cascades.json)
Detector-->>Orchestrator: potential_cascades.json
Orchestrator->>Analyzer: run analyze-cascade-similarity (spawn subagents)
Analyzer->>Prow: launch prow-job-analyze-test-failure subagents (parallel)
Prow-->>Analyzer: ANALYSIS_RESULT blocks
Analyzer-->>Orchestrator: confirmed_cascades.json
Orchestrator->>Reporter: run generate-cascade-report (--format)
Reporter-->>User: HTML/Markdown/JSON report
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 10✅ Passed checks (10 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/data.json`:
- Around line 363-367: The command metadata for detect-backport-regressions is
missing the documented flags; update the JSON entry for
"detect-backport-regressions" (the object containing "name":
"detect-backport-regressions", "argument_hint" and "synopsis") to include the
missing options --format, --analyze-similarity and --similarity-threshold in
both the "argument_hint" and "synopsis" strings so generated help matches the
documented flags and usage.
In `@plugins/ci/commands/detect-backport-regressions.md`:
- Line 3: Update the CLI docs in
plugins/ci/commands/detect-backport-regressions.md so the argument-hint,
synopsis, examples, and per-argument definitions are consistent: add the missing
options (--format, --analyze-similarity, --similarity-threshold) to the
argument-hint and synopsis, and normalize the representation of
--exclude-install and --exclude-monitor across synopsis/examples/argument text
(choose either flag style or value-style and apply it everywhere); verify the
examples and the argument descriptions reflect the chosen style and the same
default/expected values for these options.
- Around line 134-136: Update the fenced code blocks in
plugins/ci/commands/detect-backport-regressions.md for each example of the
/ci:detect-backport-regressions command (including variants like "--component
kube-apiserver --lookback 6", "--min-cascade 2", etc.) to include a language
identifier (use "bash") after the opening triple backticks so the examples
comply with MD040; add ```bash instead of ``` for the blocks referenced (lines
around the shown examples and the additional ranges 139-141, 145-146, 150-151,
155-156, 160-161, 165-166, 170-171).
In `@plugins/ci/skills/analyze-cascade-similarity/SKILL.md`:
- Around line 243-248: The current early-return for low confidence incorrectly
sets "same_root_cause": False; change this branch to avoid auto-classifying as
false-positive by returning an indeterminate value (e.g., "same_root_cause":
None or omitting the key) while preserving "similarity_score" and a descriptive
"reason" like "Low confidence in one or both analyses" so semantics match the
later non-classification behavior (see the handling around lines 421-424);
update the low-confidence branch that produces the dict with keys
same_root_cause, similarity_score, and reason to reflect this indeterminate
result instead of False.
- Around line 209-211: Replace the brittle selection of origin_release =
list(results.keys())[0] with an explicit determination of the origin: either
accept an explicit origin identifier/parameter or compute the earliest release
from results by sorting keys using a proper chronological ordering (e.g.,
semantic-version or timestamp from each results[release] entry) and then assign
origin_release to that computed value; update origin_result =
results[origin_release] accordingly so origin selection no longer depends on map
iteration order.
- Around line 92-113: The fenced prompt block in SKILL.md (the Analyze the test
failure ... ANALYSIS_RESULT block including placeholders like {test_name},
{prow_url}, and {release}) is missing a language identifier; update the opening
triple-backtick to include a language token (e.g., ```text) so the block becomes
"```text" and keep the rest of the block unchanged, ensuring the closing "```"
remains present.
In `@plugins/ci/skills/detect-backport-regressions/SKILL.md`:
- Around line 41-45: The prerequisites list omits the `fetch-test-runs` skill
required by the workflow; update the "Installed plugins" bullet under the
prerequisites to include `fetch-test-runs` alongside `ci` and `teams` so lines
that call fetch-test-runs (e.g., references around "Line 80" / the
fetch-test-runs requirement) are satisfied; specifically add "`-
`fetch-test-runs` (for fetch-test-runs)`" to the installed plugins block so
users following the doc end-to-end won't miss this dependency.
- Around line 92-100: The three fenced code blocks containing the
ANALYSIS_RESULT block, the "ERROR: Could not auto-detect current development
release" message, and the "✅ No cascading regressions detected - system is
healthy!" message should include a language identifier (e.g., text) after the
opening triple backticks; update the fences around the ANALYSIS_RESULT block,
the auto-detect error block, and the healthy status block in SKILL.md to use
```text (or another appropriate language) so markdownlint rule MD040 is
satisfied.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 5a6ea192-8e56-4073-bb55-6b9ae9a2d7c8
📒 Files selected for processing (8)
PLUGINS.mddocs/data.jsonplugins/ci/commands/detect-backport-regressions.mdplugins/ci/skills/analyze-cascade-similarity/SKILL.mdplugins/ci/skills/detect-backport-regressions/SKILL.mdplugins/ci/skills/detect-potential-cascades/SKILL.mdplugins/ci/skills/generate-cascade-report/SKILL.mdplugins/teams/skills/list-regressions/list_regressions.py
| "argument_hint": "[--current-release <version>] [--lookback N] [--days N]", | ||
| "description": "Detect regressions that have cascaded from newer releases to older releases due to problematic backports", | ||
| "name": "detect-backport-regressions", | ||
| "synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install] [--exclude-monitor] [--component <name>] [--min-cascade N] [--include-resolved]" | ||
| }, |
There was a problem hiding this comment.
Keep command metadata in sync with documented options.
The detect-backport-regressions metadata omits documented flags (--format, --analyze-similarity, --similarity-threshold), which makes generated command help incomplete.
Suggested manifest update
- "argument_hint": "[--current-release <version>] [--lookback N] [--days N]",
+ "argument_hint": "[--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]",
@@
- "synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install] [--exclude-monitor] [--component <name>] [--min-cascade N] [--include-resolved]"
+ "synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "argument_hint": "[--current-release <version>] [--lookback N] [--days N]", | |
| "description": "Detect regressions that have cascaded from newer releases to older releases due to problematic backports", | |
| "name": "detect-backport-regressions", | |
| "synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install] [--exclude-monitor] [--component <name>] [--min-cascade N] [--include-resolved]" | |
| }, | |
| "argument_hint": "[--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]", | |
| "description": "Detect regressions that have cascaded from newer releases to older releases due to problematic backports", | |
| "name": "detect-backport-regressions", | |
| "synopsis": "/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]" | |
| }, |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/data.json` around lines 363 - 367, The command metadata for
detect-backport-regressions is missing the documented flags; update the JSON
entry for "detect-backport-regressions" (the object containing "name":
"detect-backport-regressions", "argument_hint" and "synopsis") to include the
missing options --format, --analyze-similarity and --similarity-threshold in
both the "argument_hint" and "synopsis" strings so generated help matches the
documented flags and usage.
| @@ -0,0 +1,251 @@ | |||
| --- | |||
| description: Detect regressions that have cascaded from newer releases to older releases due to problematic backports | |||
| argument-hint: "[--current-release <version>] [--lookback N] [--days N]" | |||
There was a problem hiding this comment.
Unify the CLI contract across argument-hint, synopsis, and argument definitions.
The documented option set is internally inconsistent: Line 13 omits --format, --analyze-similarity, and --similarity-threshold despite Lines 208-228 documenting them; and --exclude-install/--exclude-monitor are shown as bare flags in synopsis but value-style in examples/argument text.
Suggested doc contract alignment
-argument-hint: "[--current-release <version>] [--lookback N] [--days N]"
+argument-hint: "[--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]"-/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install] [--exclude-monitor] [--component <name>] [--min-cascade N] [--include-resolved]
+/ci:detect-backport-regressions [--current-release <version>] [--lookback N] [--days N] [--exclude-install <true|false>] [--exclude-monitor <true|false>] [--component <name>] [--min-cascade N] [--include-resolved] [--format <html|markdown|json>] [--analyze-similarity <true|false>] [--similarity-threshold N]Also applies to: 13-14, 188-194, 208-228
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@plugins/ci/commands/detect-backport-regressions.md` at line 3, Update the CLI
docs in plugins/ci/commands/detect-backport-regressions.md so the argument-hint,
synopsis, examples, and per-argument definitions are consistent: add the missing
options (--format, --analyze-similarity, --similarity-threshold) to the
argument-hint and synopsis, and normalize the representation of
--exclude-install and --exclude-monitor across synopsis/examples/argument text
(choose either flag style or value-style and apply it everywhere); verify the
examples and the argument descriptions reflect the chosen style and the same
default/expected values for these options.
| ``` | ||
| /ci:detect-backport-regressions | ||
| ``` |
There was a problem hiding this comment.
Add language identifiers to command example code fences (MD040).
Several fenced blocks in Examples are missing a language specifier.
Suggested markdownlint-compliant fix
- ```
+ ```bash
/ci:detect-backport-regressions
```- ```
+ ```bash
/ci:detect-backport-regressions --component kube-apiserver --lookback 6
```- ```
+ ```bash
/ci:detect-backport-regressions --min-cascade 2
```- ```
+ ```bash
/ci:detect-backport-regressions --exclude-install false --days 60
```- ```
+ ```bash
/ci:detect-backport-regressions --current-release 4.22 --lookback 3
```- ```
+ ```bash
/ci:detect-backport-regressions --include-resolved
```- ```
+ ```bash
/ci:detect-backport-regressions --format markdown --days 45
```- ```
+ ```bash
/ci:detect-backport-regressions --exclude-monitor false
```Also applies to: 139-141, 145-146, 150-151, 155-156, 160-161, 165-166, 170-171
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@plugins/ci/commands/detect-backport-regressions.md` around lines 134 - 136,
Update the fenced code blocks in
plugins/ci/commands/detect-backport-regressions.md for each example of the
/ci:detect-backport-regressions command (including variants like "--component
kube-apiserver --lookback 6", "--min-cascade 2", etc.) to include a language
identifier (use "bash") after the opening triple backticks so the examples
comply with MD040; add ```bash instead of ``` for the blocks referenced (lines
around the shown examples and the additional ranges 139-141, 145-146, 150-151,
155-156, 160-161, 165-166, 170-171).
| ``` | ||
| Analyze the test failure "{test_name}" in this Prow job: {prow_url} | ||
|
|
||
| Use the ci:prow-job-analyze-test-failure skill to perform a thorough root cause analysis. | ||
|
|
||
| IMPORTANT: | ||
| - Trace to the actual root cause, not just symptoms | ||
| - Download log bundles, examine pod logs, cite specific error messages | ||
| - Never stop at symptoms like "0 nodes ready", "operator degraded", or "crash-looping" | ||
| - Look for the underlying reason (network issues, CNI config loss, registry failures, etc.) | ||
|
|
||
| At the end of your analysis, you MUST provide an ANALYSIS_RESULT block in this exact format: | ||
|
|
||
| ANALYSIS_RESULT: | ||
| - root_cause_summary: <one-line summary of the actual root cause> | ||
| - affected_components: <comma-separated list of affected operators/components> | ||
| - key_error_patterns: <comma-separated key error strings for matching> | ||
| - known_symptoms: <comma-separated symptom summaries, or "none"> | ||
| - test_name: {test_name} | ||
| - confidence_level: <1-5, where 5 is highest confidence in the root cause> | ||
| - release: {release} | ||
| ``` |
There was a problem hiding this comment.
Add a language identifier for the prompt template fence (MD040).
The fenced block beginning at Line 92 should declare a language.
Suggested fix
-```
+```text
Analyze the test failure "{test_name}" in this Prow job: {prow_url}
@@
-- release: {release}
-```
+- release: {release}
+```📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ``` | |
| Analyze the test failure "{test_name}" in this Prow job: {prow_url} | |
| Use the ci:prow-job-analyze-test-failure skill to perform a thorough root cause analysis. | |
| IMPORTANT: | |
| - Trace to the actual root cause, not just symptoms | |
| - Download log bundles, examine pod logs, cite specific error messages | |
| - Never stop at symptoms like "0 nodes ready", "operator degraded", or "crash-looping" | |
| - Look for the underlying reason (network issues, CNI config loss, registry failures, etc.) | |
| At the end of your analysis, you MUST provide an ANALYSIS_RESULT block in this exact format: | |
| ANALYSIS_RESULT: | |
| - root_cause_summary: <one-line summary of the actual root cause> | |
| - affected_components: <comma-separated list of affected operators/components> | |
| - key_error_patterns: <comma-separated key error strings for matching> | |
| - known_symptoms: <comma-separated symptom summaries, or "none"> | |
| - test_name: {test_name} | |
| - confidence_level: <1-5, where 5 is highest confidence in the root cause> | |
| - release: {release} | |
| ``` |
🧰 Tools
🪛 markdownlint-cli2 (0.22.0)
[warning] 92-92: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@plugins/ci/skills/analyze-cascade-similarity/SKILL.md` around lines 92 - 113,
The fenced prompt block in SKILL.md (the Analyze the test failure ...
ANALYSIS_RESULT block including placeholders like {test_name}, {prow_url}, and
{release}) is missing a language identifier; update the opening triple-backtick
to include a language token (e.g., ```text) so the block becomes "```text" and
keep the rest of the block unchanged, ensuring the closing "```" remains
present.
| # Extract origin release result | ||
| origin_release = list(results.keys())[0] | ||
| origin_result = results[origin_release] |
There was a problem hiding this comment.
Select origin release explicitly, not by map iteration order.
Using the “first key” as origin can point comparisons at the wrong baseline when result assembly order differs from release chronology.
Suggested fix
-def compare_root_causes(results: dict) -> dict:
+def compare_root_causes(results: dict, origin_release: str) -> dict:
@@
- origin_release = list(results.keys())[0]
- origin_result = results[origin_release]
+ if origin_release not in results:
+ return {
+ "same_root_cause": False,
+ "similarity_score": 0.0,
+ "reason": "Origin release analysis missing"
+ }
+ origin_result = results[origin_release]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@plugins/ci/skills/analyze-cascade-similarity/SKILL.md` around lines 209 -
211, Replace the brittle selection of origin_release = list(results.keys())[0]
with an explicit determination of the origin: either accept an explicit origin
identifier/parameter or compute the earliest release from results by sorting
keys using a proper chronological ordering (e.g., semantic-version or timestamp
from each results[release] entry) and then assign origin_release to that
computed value; update origin_result = results[origin_release] accordingly so
origin selection no longer depends on map iteration order.
| if conf1 < 3 or conf2 < 3: | ||
| return { | ||
| "same_root_cause": False, | ||
| "similarity_score": 0.0, | ||
| "reason": "Low confidence in one or both analyses" | ||
| } |
There was a problem hiding this comment.
Do not auto-classify low-confidence comparisons as false positives.
Line 243 currently forces low-confidence cases into same_root_cause: False, which conflicts with Lines 421-424 (“Don’t automatically classify as false positive”).
Suggested fix for consistent classification semantics
if conf1 < 3 or conf2 < 3:
return {
"same_root_cause": False,
"similarity_score": 0.0,
+ "needs_manual_review": True,
"reason": "Low confidence in one or both analyses"
}Also applies to: 421-424
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@plugins/ci/skills/analyze-cascade-similarity/SKILL.md` around lines 243 -
248, The current early-return for low confidence incorrectly sets
"same_root_cause": False; change this branch to avoid auto-classifying as
false-positive by returning an indeterminate value (e.g., "same_root_cause":
None or omitting the key) while preserving "similarity_score" and a descriptive
"reason" like "Low confidence in one or both analyses" so semantics match the
later non-classification behavior (see the handling around lines 421-424);
update the low-confidence branch that produces the dict with keys
same_root_cause, similarity_score, and reason to reflect this indeterminate
result instead of False.
| 1. **Network Access**: Must be able to reach Sippy API endpoints and GCS (for Prow artifacts) | ||
| 2. **Installed plugins**: | ||
| - `ci` plugin (for fetch-releases, fetch-regression-details, prow-job-analyze-test-failure) | ||
| - `teams` plugin (for list-regressions) | ||
|
|
There was a problem hiding this comment.
Add missing prerequisite for fetch-test-runs.
Line 80 requires fetch-test-runs, but it is not listed in prerequisites, which can cause setup failures for users following this doc end-to-end.
Suggested doc fix
2. **Installed plugins**:
- - `ci` plugin (for fetch-releases, fetch-regression-details, prow-job-analyze-test-failure)
+ - `ci` plugin (for fetch-releases, fetch-regression-details, fetch-test-runs, prow-job-analyze-test-failure)
- `teams` plugin (for list-regressions)Also applies to: 79-81
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@plugins/ci/skills/detect-backport-regressions/SKILL.md` around lines 41 - 45,
The prerequisites list omits the `fetch-test-runs` skill required by the
workflow; update the "Installed plugins" bullet under the prerequisites to
include `fetch-test-runs` alongside `ci` and `teams` so lines that call
fetch-test-runs (e.g., references around "Line 80" / the fetch-test-runs
requirement) are satisfied; specifically add "`- `fetch-test-runs` (for
fetch-test-runs)`" to the installed plugins block so users following the doc
end-to-end won't miss this dependency.
| ``` | ||
| ANALYSIS_RESULT: | ||
| - root_cause_summary: <one-line summary> | ||
| - affected_components: <comma-separated list of affected operators/components> | ||
| - key_error_patterns: <comma-separated key error strings for matching> | ||
| - known_symptoms: <comma-separated symptom summaries, or "none"> | ||
| - test_name: <the name of the failing test> | ||
| - confidence_level: <1-5, where 5 is highest confidence in the root cause> | ||
| ``` |
There was a problem hiding this comment.
Specify languages for fenced code blocks (MD040).
The code fences at Lines 92, 153, and 161 are missing language identifiers.
Suggested markdownlint-compliant fix
- ```
+ ```text
ANALYSIS_RESULT:
- root_cause_summary: <one-line summary>
- affected_components: <comma-separated list of affected operators/components>
- key_error_patterns: <comma-separated key error strings for matching>
- known_symptoms: <comma-separated symptom summaries, or "none">
- test_name: <the name of the failing test>
- confidence_level: <1-5, where 5 is highest confidence in the root cause>
```-```
+```text
ERROR: Could not auto-detect current development release
HINT: Use --current-release to manually specify (e.g., --current-release 4.22)
```diff
-```
+```text
✅ No cascading regressions detected - system is healthy!
</details>
Also applies to: 153-156, 161-163
<details>
<summary>🧰 Tools</summary>
<details>
<summary>🪛 markdownlint-cli2 (0.22.0)</summary>
[warning] 92-92: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
</details>
</details>
<details>
<summary>🤖 Prompt for AI Agents</summary>
Verify each finding against the current code and only fix it if needed.
In @plugins/ci/skills/detect-backport-regressions/SKILL.md around lines 92 -
100, The three fenced code blocks containing the ANALYSIS_RESULT block, the
"ERROR: Could not auto-detect current development release" message, and the "✅
No cascading regressions detected - system is healthy!" message should include a
language identifier (e.g., text) after the opening triple backticks; update the
fences around the ANALYSIS_RESULT block, the auto-detect error block, and the
healthy status block in SKILL.md to use ```text (or another appropriate
language) so markdownlint rule MD040 is satisfied.
</details>
<!-- fingerprinting:phantom:poseidon:hawk:d857237d-3e28-469d-8b04-71c3b6b69b09 -->
<!-- This is an auto-generated comment by CodeRabbit -->
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What this PR does / why we need it:
Detect regressions that have cascaded from newer releases to older releases due to problematic backports
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Chores