A Claude Code skill for investigating production incidents. Describe what's broken in plain text, and it searches your observability stack in parallel to find the root cause.
You describe the problem. It fans out across your tools and comes back with hypotheses.
Phase 1 runs three agents at once -- one hits Glean for historical context and service ownership, one checks incident.io for anything other responders have already found, and one sweeps Sentry for error spikes and stack traces. The findings get merged into early hypotheses, and you pick which threads to pull.
Phase 2 goes deeper on whatever you point it at. Coralogix for logs, traces, and metrics. GitHub for recent merges and deploys. Atlassian for architecture docs, Jira tickets, and past post-mortems. Each agent tests a specific hypothesis rather than searching broadly, so there's less to filter through.
After Phase 2 it ranks everything by evidence strength and timing, then asks if you want another round. Keep going until you have an answer or run out of leads.
Nothing writes. The entire skill is read-only across every tool.
flowchart TD
intake["Parse incident description\nExtract: services, errors,\ntime window, regions, affected users"]
preflight["Pre-flight: check all\nMCP tools are connected"]
ready{All tools available?}
warn["Report missing tools\nAsk user how to proceed"]
p1["Phase 1: Broad Sweep\n(3 parallel subagents)"]
glean["Glean\nHistory + ownership"]
incidentio["Incident.io\nActive incidents + alerts"]
sentry["Sentry\nErrors + stack traces"]
synth["Synthesize Phase 1 findings\nForm early hypotheses\nPresent to user"]
steer["User steers:\nwhich threads to pull"]
p2["Phase 2: Deep Investigation\n(user-directed subagents)"]
coralogix["Coralogix\nLogs + traces + metrics"]
github["GitHub\nCode changes + deploys"]
atlassian["Atlassian\nBusiness context + docs"]
manual["Manual check prompts\n(if triggers match)"]
rank["Rank hypotheses\nPresent findings"]
iterate{User: dig deeper?}
done([Done])
intake --> preflight
preflight --> ready
ready -->|yes| p1
ready -->|no| warn
warn -->|"user: proceed anyway"| p1
p1 --> glean
p1 --> incidentio
p1 --> sentry
glean --> synth
incidentio --> synth
sentry --> synth
synth --> steer
steer --> p2
p2 --> coralogix
p2 --> github
p2 --> atlassian
coralogix --> manual
github --> manual
atlassian --> manual
manual --> rank
rank --> iterate
iterate -->|yes| steer
iterate -->|no| done
style intake fill:#b3d9ff,color:#000
style preflight fill:#b3d9ff,color:#000
style ready fill:#b3d9ff,color:#000
style warn fill:#ffb3b3,color:#000
style p1 fill:#b3ffcc,color:#000
style glean fill:#b3ffcc,color:#000
style incidentio fill:#b3ffcc,color:#000
style sentry fill:#b3ffcc,color:#000
style synth fill:#b3ffcc,color:#000
style steer fill:#b3ffcc,color:#000
style p2 fill:#ffd9b3,color:#000
style coralogix fill:#ffd9b3,color:#000
style github fill:#ffd9b3,color:#000
style atlassian fill:#ffd9b3,color:#000
style manual fill:#ffd9b3,color:#000
style rank fill:#ffd9b3,color:#000
style iterate fill:#ffd9b3,color:#000
style done fill:#e0e0e0,color:#000
The skill needs five MCP integrations connected before it can run. They're all available through Claude.ai's built-in integrations.
To connect one: go to Claude.ai → Settings → Integrations, find the service, and complete the OAuth flow. Once connected, it shows up in Claude Code automatically when you're signed in with the same account.
| Integration | Used for | Notes |
|---|---|---|
| Glean | Service ownership, historical incidents, runbooks | Requires your org's Glean instance to be connected |
| Incident.io | Active incidents, alert history, responder notes | Needs read access to your incident.io workspace |
| Sentry | Error spikes, stack traces, release correlation | Connects to your Sentry org |
| Coralogix | Logs, traces, and metrics | Needs a Coralogix API key scoped to your environments |
| Atlassian | Confluence docs, Jira tickets, post-mortems | Connects to your Atlassian Cloud instance |
Before starting, the skill checks which tools are connected and reports anything missing. You can run with a partial set -- the investigation just won't cover those sources.
The code change investigation uses gh to query PRs and deploys. If you don't have it:
# macOS
brew install gh
# Linux
# See https://github.com/cli/cli/blob/trunk/docs/install_linux.md
gh auth loginClone the repo directly into your Claude Code skills folder:
git clone https://github.com/LorcanChinnock/incident-rca ~/.claude/skills/incident-rcaClaude Code picks it up automatically. You should see incident-rca in your available skills.
To update later:
cd ~/.claude/skills/incident-rca && git pullDescribe the incident in plain text:
/incident-rca Checkout is returning 500 errors for about 30% of users in EU region.
Started around 14:00 UTC today. Payments team is seeing it in their dashboards.
You can also just describe symptoms in conversation -- Claude will recognise the pattern and offer to use the skill.
What happens next:
- Pre-flight check confirms your tools are connected
- Three parallel sweeps run (Glean, incident.io, Sentry)
- You get early hypotheses and choose which to investigate
- Targeted deep dives run against the sources you pick
- Ranked root cause hypotheses come back with supporting evidence
- If code changes reference feature flags, you get prompted to check LaunchDarkly
Some systems can't be queried through MCP. When the investigation points at one of them, it tells you what to check and what to look for.
Right now the only manual check is LaunchDarkly, triggered when code changes reference feature flags. Adding more is just a matter of editing references/manual-checks.md.
incident-rca/
├── SKILL.md # Orchestrator
├── prompts/
│ ├── glean-sweep-prompt.md # Phase 1: orientation + history
│ ├── incident-io-sweep-prompt.md # Phase 1: active incidents + alerts
│ ├── sentry-sweep-prompt.md # Phase 1: errors + stack traces
│ ├── coralogix-investigation-prompt.md # Phase 2: logs + traces + metrics
│ ├── code-change-investigation-prompt.md # Phase 2: git history + deploys
│ └── atlassian-context-prompt.md # Phase 2: business context + docs
└── references/
├── manual-checks.md # Manual check triggers
└── tool-read-only-constraints.md # Per-MCP allowed/forbidden tools
To add a new data source, create a prompt template in prompts/, add its constraints to references/tool-read-only-constraints.md, and wire it into SKILL.md under the right phase.
To add a manual check, add a section to references/manual-checks.md with trigger signals and a prompt template.
To change how hypotheses get ranked, edit the ranking criteria in SKILL.md.
Everything is read-only. Every subagent prompt has an inline allowlist of permitted tools and an explicit list of forbidden operations. The master list is in references/tool-read-only-constraints.md. The skill won't create Jira tickets, update incidents, comment on PRs, or push code.