MLFlow eval

Author: EoghanOConnor

skills/rca-annotator/SKILL.md

@@ -1,12 +1,17 @@
 ---
 name: rca-annotator
+<<<<<<< HEAD
 description: LLM-as-judge skill that independently evaluates root-cause-analysis outputs to produce ground-truth annotations with multi-pass consistency checks, difficulty calibration, and complete evidence traceability.
+=======
+description: Structured annotation tool that walks users through reviewing and labeling root-cause-analysis outputs, with evidence traceability, difficulty calibration, and alternative diagnosis capture.
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 allowed-tools:
   - Read
   - Write
   - Bash
 ---
 
+<<<<<<< HEAD
 # RCA Annotator (LLM-as-Judge)
 
 An LLM-as-judge skill that independently re-analyzes raw RCA evidence (step1, step3, step4 outputs) to produce ground-truth annotations. The judge forms its own conclusion from the same source evidence the agent had — without reading the agent's diagnosis — producing an unbiased reference label for scoring agent accuracy.
@@ -20,6 +25,21 @@ An LLM-as-judge skill that independently re-analyzes raw RCA evidence (step1, st
 **Use when**: steps 1-4 outputs exist and you need labeled ground-truth data, difficulty benchmarking, or alternative diagnosis identification.
 
 **Do NOT use** to perform initial RCA (use `root-cause-analysis`) or directly score agent output (compare annotation externally).
+=======
+# RCA Annotator
+
+A structured annotation tool that presents the `root-cause-analysis` agent's diagnosis to the user and guides them through labeling it — capturing whether the diagnosis is correct, evidence quality, difficulty, and alternative hypotheses.
+
+| | `root-cause-analysis` (Agent) | `rca-annotator` (Annotation tool) |
+|---|---|---|
+| **Purpose** | Diagnose failures for users | Capture human-labeled ground-truth data |
+| **Reads** | Logs, Splunk, GitHub (live) | Step 1/3/4/5 output files (offline) |
+| **Output** | Human-readable diagnosis | Structured `annotation.json` |
+
+**Use when**: a `root-cause-analysis` run is complete and you want to annotate its output as correct, incorrect, or partially correct — for evaluation, benchmarking, or dataset building.
+
+**Do NOT use** to perform initial RCA (use `root-cause-analysis`).
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 
 ## Prerequisites
 
@@ -28,6 +48,10 @@ Verify root-cause-analysis has been completed.
 - `JUMPBOX_URI` (optional) — SSH connection string (e.g. `"user@host -p 2222"`). If unset, uses local `.analysis/` only.
 - SSH keys configured in `~/.ssh/config` if using jumpbox; `ssh` and `rsync` installed.
 - **Required files** in `.analysis/<job_id>/`:
+<<<<<<< HEAD
+=======
+  - `step5_summary.json` — Agent's final diagnosis (primary input)
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
   - `step1_job_context.json` — Job metadata, failed tasks
   - `step3_correlation.json` — Timeline with AAP + Splunk events
   - `step4_github_fetch_history.json` — Configuration and code context
@@ -37,11 +61,18 @@ If missing, run `root-cause-analysis` skill first.
 ## Workflow
 
 0. Download from jumpbox (if `JUMPBOX_URI` set) or verify local files
+<<<<<<< HEAD
 1. Read step1, step3, step4 (never the agent's diagnosis)
 2. Independently determine root cause with evidence scoring and traceability
 3. *(Optional)* Multi-pass consistency check
 4. Write `annotation_draft.json`
 5. Upload to jumpbox (if `JUMPBOX_URI` set)
+=======
+1. Read `step5_summary.json` — present the agent's diagnosis to the user
+2. Walk through annotation questions interactively — the user labels each section
+3. Write `annotation.json` with the user's labels
+4. Upload to jumpbox (if `JUMPBOX_URI` set)
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 
 ---
 
@@ -56,6 +87,7 @@ Downloads from jumpbox `/usr/local/mlflow/<job_id>/` to local `.analysis/<job_id
 
 ---
 
+<<<<<<< HEAD
 ## Step 1: Read Outputs
 
 Read in order: **step1** (job metadata, failed tasks, errors) → **step3** (correlated timeline incl. Splunk) → **step4** (config hierarchy and code). Skip `step2_splunk_logs.json` — step3 already includes correlated events.
@@ -133,11 +165,112 @@ For high-stakes annotations or medium/low initial confidence. Run 2-3 independen
 - All differ → confidence: low, flag for human review
 
 Skip if initial confidence is high and difficulty is easy.
+=======
+## Step 1: Read Agent Diagnosis
+
+Read `step5_summary.json` and present the agent's diagnosis clearly to the user:
+
+- Root cause category and summary
+- Confidence level
+- Key evidence cited
+- Difficulty score (if present)
+- Recommendations
+- Alternative diagnoses (if any)
+
+This is the starting point for annotation. The user is reviewing the agent's work.
+
+---
+
+## Step 2: Interactive Annotation
+
+Walk through each question below with the user. Present the relevant section from `step5_summary.json` before asking each question. Wait for the user's response before continuing.
+
+### 1. Root Cause Category
+
+Present the agent's category and summary. Ask:
+
+> **Is the root cause category correct?** *(e.g. `configuration`, `infrastructure`, `credential` — or should it be something else?)*
+
+Valid categories: `configuration` | `infrastructure` | `application_bug` | `dependency` | `network` | `resource` | `cloud_api` | `credential` | `secrets` | `unknown`
+
+### 2. Summary Accuracy
+
+Present the agent's summary sentence. Ask:
+
+> **Is the summary accurate and specific?** *(Does it clearly describe what failed and why?)*
+
+### 3. Evidence
+
+Present the evidence items the agent cited. Ask:
+
+> **Is any evidence missing or wrong?** *(Any key log lines, config values, or Splunk events that were overlooked or incorrectly cited?)*
+
+If the user wants to cross-check, read step1/step3/step4 and compare against what the agent cited. This is reference material for validation — not a re-analysis.
+
+**Evidence traceability format** (for any new or corrected evidence items the user provides):
+
+```json
+{
+  "source": "step1 | step3 | step4",
+  "source_file": ".analysis/<job_id>/step1_job_context.json",
+  "json_path": "failed_tasks[0].duration",
+  "exact_value": 917.565567,
+  "exact_quote": "optional — literal text for code/config",
+  "line_number": 5,
+  "github_path": "owner/repo:path/to/file.yml:line",
+  "message": "The relevant log line or config snippet.",
+  "confidence": "high | medium | low",
+  "is_root_cause": true
+}
+```
+
+### 4. Difficulty Rating
+
+Present the agent's difficulty score (or estimate one from the evidence). Present the calibration rubric to help the user score:
+
+| Criterion | Points |
+|---|---|
+| Requires cross-source correlation (AAP + Splunk + GitHub) | +3 |
+| Requires understanding code behavior | +2 |
+| Error message is generic or misleading | +2 |
+| Requires variable precedence/override knowledge | +1 |
+| Requires domain knowledge (K8s, Ansible, cloud APIs) | +1 |
+| Multiple plausible alternatives exist | +1 |
+| Timing dependencies are critical | +1 |
+
+Mapping: 0–3 = easy, 4–6 = medium, 7–10 = hard.
+
+Ask:
+
+> **Is the difficulty rating appropriate?** *(Score of X / 10 — too easy, too hard, or about right? Use the rubric above if helpful.)*
+
+### 5. Alternative Diagnoses
+
+Present any alternative diagnoses the agent identified. Ask:
+
+> **Any alternative diagnoses to add or correct?** *(Other plausible-but-incorrect hypotheses worth capturing?)*
+
+Alternative diagnosis format:
+
+```json
+{
+  "category": "infrastructure",
+  "summary": "A plausible but wrong diagnosis.",
+  "why_wrong": "Why the evidence does not support this.",
+  "plausibility": "high | medium | low",
+  "supporting_evidence": ["long timeout", "destroy action"],
+  "contradicting_evidence": ["test environment", "auth retry pattern"]
+}
+```
+
+`plausibility`: `high` = shares many characteristics | `medium` = some evidence | `low` = superficial similarity
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 
 ---
 
 ## Step 3: Write Annotation
 
+<<<<<<< HEAD
 Verify before writing:
 - All required files read (step1, step3, step4)
 - Root cause category matches evidence
@@ -148,6 +281,17 @@ Verify before writing:
 - Alternative diagnoses have plausibility levels
 
 Write `annotation_draft.json` to `.analysis/<job_id>/`.
+=======
+After all questions are answered, verify before writing:
+
+- Root cause category confirmed or corrected
+- Exactly one evidence item has `is_root_cause: true`
+- All evidence has traceability (source_file, json_path, exact_value/quote)
+- Difficulty score calculated with justification
+- Alternative diagnoses have plausibility levels
+
+Write `annotation.json` to `.analysis/<job_id>/`.
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 
 ---
 
@@ -158,29 +302,50 @@ cd skills/rca-annotator
 python scripts/cli.py upload --job-id <job_id>
 ```
 
+<<<<<<< HEAD
 Uploads `.analysis/<job_id>/annotation_draft.json` to jumpbox if `JUMPBOX_URI` set. Local copy always preserved. If `JUMPBOX_URI` unset, file remains local only.
+=======
+Uploads `.analysis/<job_id>/annotation.json` to jumpbox if `JUMPBOX_URI` set. Local copy always preserved. If `JUMPBOX_URI` unset, file remains local only.
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 
 ---
 
 ## Output Format
 
+<<<<<<< HEAD
 Save to `.analysis/<job_id>/annotation_draft.json`:
+=======
+Save to `.analysis/<job_id>/annotation.json`:
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 
 ```json
 {
   "job_id": "1234567",
+<<<<<<< HEAD
   "annotator": "claude_judge",
   "annotated_at": "2026-03-19T12:00:00Z",
   "difficulty": "easy | medium | hard",
   "difficulty_score": 5,
   "difficulty_justification": "Requires correlating task code (+2) with missing configs...",
+=======
+  "annotated_at": "2026-03-19T12:05:00Z",
+
+  "category_correct": true,
+  "category_comment": "Confirmed — matches the auth retry pattern.",
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 
   "root_cause": {
     "category": "configuration | infrastructure | application_bug | dependency | network | resource | cloud_api | credential | secrets | unknown",
     "summary": "One sentence describing what failed and why.",
     "confidence": "high | medium | low"
   },
 
+<<<<<<< HEAD
+=======
+  "summary_accurate": true,
+  "summary_comment": "Clear and specific.",
+
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
   "evidence": [
     {
       "source": "step1 | step3 | step4",
@@ -196,6 +361,17 @@ Save to `.analysis/<job_id>/annotation_draft.json`:
     }
   ],
 
+<<<<<<< HEAD
+=======
+  "evidence_feedback": "Missing the kubeconfig 404 from step4 github_fetches.",
+
+  "difficulty": "easy | medium | hard",
+  "difficulty_score": 5,
+  "difficulty_justification": "Requires correlating task code (+2) with missing configs and interpreting generic MODULE FAILURE (+2). Total: 5.",
+  "difficulty_appropriate": false,
+  "difficulty_comment": "Should be hard (8/10) — requires deep variable precedence knowledge.",
+
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
   "recommendations": [
     {
       "priority": "high | medium | low",
@@ -217,6 +393,7 @@ Save to `.analysis/<job_id>/annotation_draft.json`:
       "supporting_evidence": ["long timeout", "destroy action"],
       "contradicting_evidence": ["test environment", "auth retry pattern"]
     }
+<<<<<<< HEAD
   ],
 
   "consistency_check": {
@@ -229,5 +406,8 @@ Save to `.analysis/<job_id>/annotation_draft.json`:
       "confidence_mode": "high"
     }
   }
+=======
+  ]
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 }
 ```

skills/rca-annotator/schemas/schema.json

@@ -1,17 +1,30 @@
 {
   "job_id": "1234567",
+<<<<<<< HEAD
   "annotator": "claude_judge",
   "annotated_at": "2026-03-08T12:00:00Z",
   "difficulty": "easy | medium | hard",
   "difficulty_score": 5,
   "difficulty_justification": "Requires correlating task code (+2) with missing configs (+0) and interpreting generic MODULE FAILURE (+2) and understanding default(omit) behavior (+1). Total: 5 points.",
+=======
+  "annotated_at": "2026-03-19T12:05:00Z",
+
+  "category_correct": true,
+  "category_comment": "Confirmed — credential issue matches the auth retry pattern.",
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 
   "root_cause": {
     "category": "configuration | infrastructure | application_bug | dependency | network | resource | cloud_api | credential | secrets | unknown",
     "summary": "One sentence describing what failed and why.",
     "confidence": "high | medium | low"
   },
 
+<<<<<<< HEAD
+=======
+  "summary_accurate": true,
+  "summary_comment": "Clear and specific enough.",
+
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
   "evidence": [
     {
       "source": "step1 | step3 | step4 | aap_job | splunk_ocp | agnosticv_config | agnosticd_code",
@@ -27,6 +40,17 @@
     }
   ],
 
+<<<<<<< HEAD
+=======
+  "evidence_feedback": "Missing the kubeconfig 404 from step4 github_fetches.",
+
+  "difficulty": "easy | medium | hard",
+  "difficulty_score": 5,
+  "difficulty_justification": "Requires correlating task code (+2) with missing configs (+0) and interpreting generic MODULE FAILURE (+2) and understanding default(omit) behavior (+1). Total: 5 points.",
+  "difficulty_appropriate": false,
+  "difficulty_comment": "Should be hard (8/10), not medium — requires deep variable precedence knowledge.",
+
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
   "recommendations": [
     {
       "priority": "high | medium | low",
@@ -54,6 +78,7 @@
         "Another contradicting factor"
       ]
     }
+<<<<<<< HEAD
   ],
 
   "consistency_check": {
@@ -66,4 +91,7 @@
       "confidence_mode": "high"
     }
   }
+=======
+  ]
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 }

skills/rca-annotator/scripts/cli.py

@@ -3,7 +3,11 @@
 RCA-Annotator CLI - Command-line interface for jumpbox synchronization.
 
 This script provides commands to download analysis files from jumpbox
+<<<<<<< HEAD
 and upload annotation_draft.json back to jumpbox.
+=======
+and upload annotation.json back to jumpbox.
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
 
 Usage:
     python scripts/cli.py download --job-id <job_id>
@@ -51,7 +55,11 @@ def main():
     # Upload command
     upload_parser = subparsers.add_parser(
         "upload",
+<<<<<<< HEAD
         help="Upload annotation_draft.json to jumpbox",
+=======
+        help="Upload annotation.json to jumpbox",
+>>>>>>> 56b6ac4 (Feat/rca annotator (#14))
     )
     upload_parser.add_argument(
         "--job-id",