INSPECT.md

Interactive TUI

perf-sentinel ships an interactive terminal UI for exploring findings, span trees and cross-trace correlations. It exposes three views as a single drill-down: Analyze (the summary dashboard), Inspect (the multi-panel browser) and Explain (one trace's full-screen span tree). Whichever entry point you use, you move between the three views without leaving the TUI. A fourth view, Disclose, stands apart: it previews a public-disclosure report rather than exploring findings, so it has no drill-down into the other three (see REPORTING.md).

Entry points:

• perf-sentinel analyze --tui [--input <events.json>]: opens on the Analyze view.
• perf-sentinel demo --tui: same view on the embedded demo dataset, no input file needed.
• perf-sentinel inspect --input <events.json>: opens on the Inspect view, reads a raw events file or a pre-computed Report JSON.
• perf-sentinel explain --tui --trace-id <id> --input <events.json>: opens on the Explain view, focused on that trace.
• perf-sentinel query --daemon <URL> inspect [--sort <KEY>]: live mode, opens on Inspect, reads findings and traces from a running daemon over HTTP. --sort takes the same keys as analyze --sort.
• perf-sentinel disclose --tui --input <archive>... --org-config <PATH>: opens the Disclose preview.

In live mode (0.5.24+), the TUI also lets the operator acknowledge and revoke findings interactively from the terminal.

This TUI is the developer's trace and finding browser. For deployment monitoring there is a separate live operator TUI, perf-sentinel query monitor (since 0.8.8). It has six tabs, cycled with Tab: Advisor (the daemon's settings-advisor hints), Energy (the effective energy/carbon mix per service and per region), Trends (live braille charts of the energy/carbon per window and of the runtime gauges as a share of their configured caps), Scrapers (live health of the energy backends from /api/energy), Config (the read-only daemon parameters with their defaults and a one-line explanation each) and Incidents (the restarts and memory events the alerting posted, each with the findings frozen from its window, since 0.20.0). The tabs auto-refresh from the daemon every --refresh seconds (default 5). When the daemon becomes unreachable, the last good snapshot stays on screen with a stale indicator. Read-only: no acknowledgments. No API key is needed except for the Incidents tab, or for every tab on a daemon with [daemon] require_read_key = true. Pass --api-key-file (or PERF_SENTINEL_DAEMON_API_KEY), the read-only [daemon] read_api_key suffices, and the monitor sends it on every poll.

Views and drill-down

The three views form one drill-down. Enter descends, Esc ascends:

Analyze  --Enter-->  Inspect  --Enter-->  Explain
         <---Esc---           <---Esc---

• Analyze: the scrollable summary (GreenOps waste, top offenders, quality gate), the same content as analyze stdout. Enter descends to Inspect.
• Inspect: the multi-panel browser described below. Enter drills through the panels and, from Detail, opens Explain. Esc walks back toward Analyze.
• Explain: the selected trace's annotated span tree, full screen and scrollable. Esc returns to the Inspect Detail panel.

A tab bar at the top highlights the active view. Span trees draw from raw spans (inspect --input <events>.json or query inspect) or from the masked trees a Report carries (a daemon snapshot, or tempo/jaeger-query --format json). A trace whose spans the input does not hold shows a hint instead.

Layout

The screen splits into a 2-row layout:

┌─ Traces ──┬─ Findings ────────────────┬─ Correlations ────┐
│ trace-1   │ [1] N+1 SQL CRITICAL      │ svc-a -> svc-b    │
│ trace-2   │ [2] Redundant SQL WARNING │ ...               │
│ ...       │ [3] Slow HTTP INFO        │                   │
├───────────┴───────────────────────────┴───────────────────┤
│ Detail (full-width, span tree + finding metadata)         │
└───────────────────────────────────────────────────────────┘

The active panel border highlights cyan, the rest stays gray.

Keybindings

Navigation works with arrow keys throughout. In the Inspect view the vim keys h / j / k / l also apply, and j / k scroll the Analyze and Explain views.

Key	Action
q	Quit
↑ / k	Move selection up, or scroll (Analyze, Explain)
↓ / j	Move selection down, or scroll (Analyze, Explain)
→ / Tab / l	Cycle to next panel (Inspect)
← / BackTab / h	Cycle to previous panel (Inspect)
Enter	Drill down one step (see below)
Esc	Walk back up one step
m	Toggle mouse mode to drag-resize panel borders
r	Reset panel sizes to their defaults (Inspect)
s	Cycle the sort: impact, severity, id (Inspect)
f	Cycle the severity filter (Inspect)
a	Acknowledge the selected finding (live mode)
u	Revoke the existing ack (live mode)

s orders both lists at once, the traces and the findings inside the selected trace, on the same keys and directions as analyze --sort and the dashboard. --sort on analyze --tui and on query inspect opens directly in that order.

f cycles the Findings panel through every severity, then critical, warning and info on their own. The panel title and the tab bar both name the active filter, and a trace whose findings are all filtered out keeps its row with an empty panel: the trace list answers what ran, not what matches.

m toggles mouse capture (opt-in, so native terminal selection and copy-paste stay available while it is off). With it on, drag the border between two panels to redistribute their space. Hovering a border highlights it with a handle glyph, since a terminal application cannot change the OS mouse pointer. r resets the layout to its defaults. Panel sizes are per-session and not persisted.

Enter drills down: from Analyze to Inspect, then through the Inspect panels (Traces, Findings, Detail), then from Detail to Explain. From the Correlations panel it jumps to the Detail of the correlation's sample trace. Esc reverses each step, ascending from the top Inspect panels back to Analyze.

a and u are no-op in batch mode (inspect --input): acknowledgment requires a running daemon to persist.

Acknowledgment workflow (live mode)

When launched via query inspect, the TUI fetches findings with ?include_acked=true so already-acknowledged findings appear in the list with an italic gray [acked by <user>] indicator at the end of the line.

a: create an acknowledgment

Pressing a on a selected finding opens a modal centered on the screen with three input fields:

Field	Constraint	Default
Reason	1 to 256 chars, single line	empty (required)
Expires	empty, 24h, 7d, ISO8601, etc	empty (no expiration)
By	1 to 128 chars	$USER env var

Plus two buttons (Submit / Cancel).

Modal navigation:

Key	Action
Tab	Move focus to the next field / button
BackTab	Move focus backwards
Enter (text)	Advance to the next field
Enter (Submit)	Submit the form
Enter (Cancel)	Close the modal without submitting
Esc	Cancel the modal
Backspace	Delete the last char of the focused buffer

On submit, the TUI posts to /api/findings/<sig>/ack and closes the modal on 201. On error (4xx/5xx), the modal stays open with the error message at the bottom (red text).

u: revoke an acknowledgment

Pressing u on an acknowledged finding opens a confirmation modal. Submit / Enter issues a DELETE /api/findings/<sig>/ack. Cancel / Esc closes without revoking.

Expires format

Mirrors the CLI ack helper (since 0.5.22):

• Empty: no expiration, ack persists until manually revoked
• 24h, 7d, 30m: relative duration parsed by humantime
• 2026-05-11T00:00:00Z: ISO8601 absolute datetime

Invalid input shows expires: <error> in the modal footer without sending the request.

Authentication

The TUI mirrors the CLI ack helper's auth resolution:

1. PERF_SENTINEL_DAEMON_API_KEY env var (priority 1)
2. --api-key-file <path> flag on query inspect (priority 2)

# env var
export PERF_SENTINEL_DAEMON_API_KEY=$(cat ~/.config/perf-sentinel/key)
perf-sentinel query --daemon http://localhost:4318 inspect

# file
perf-sentinel query --daemon http://localhost:4318 inspect \
  --api-key-file ~/.config/perf-sentinel/key

Both are equivalent. Trailing newlines are trimmed, and on Unix the file is opened with O_NOFOLLOW (symlinks are refused).

No interactive password prompt in the TUI. Raw mode and the alternate screen are incompatible with rpassword TTY input. If the daemon answers 401 without an env or file key, the modal shows "API key required: set PERF_SENTINEL_DAEMON_API_KEY or pass --api-key-file when launching query inspect." Quit, set the key, relaunch.

When the daemon has no [daemon.ack] api_key configured (default for loopback deployments), no key is needed and the modal submits directly.

Caveats

Synchronous HTTP freezes the UI

run_loop is synchronous and the daemon ack write is performed via tokio::runtime::Handle::current().block_on(...) from inside the loop. The UI freezes for the duration of the request, typically 100-300 ms on localhost, longer over the network. Acceptable for a scope-minimal release. An async event loop refactor may follow if user feedback signals friction.

Findings list snapshot

The findings list is fetched once at boot. a/u refresh only the ack state via a second GET /api/findings?include_acked=true. The list of findings itself does not change in-session. To pick up newly ingested traces, quit and relaunch.

TOML acks visible, not modifiable

Findings acked in .perf-sentinel-acknowledgments.toml (CI ack) appear with the [acked by <user>] indicator and the source field set to toml. The TUI cannot promote a daemon ack to TOML or edit the TOML file. For permanent acks, edit the file via PR review per ACK-WORKFLOW.md.

The live operator monitor

perf-sentinel query monitor (since 0.8.8) is the operator-side counterpart to the developer's Inspect browser above. It runs against a live daemon, polls it on a fixed cadence (--refresh seconds, default 5) and is read-only. Tab cycles the six tabs and BackTab cycles back, j/k or the arrow keys scroll, q or Esc quits. On the Trends tab, m toggles mouse mode to drag-resize the chart borders and r resets them, the same affordance as the Inspect browser. The data each tab surfaces (config hints, source provenance, per-region intensities) is categorical and high-cardinality, so the bounded-label rule keeps it off Prometheus /metrics.

• Advisor renders the daemon's settings-advisor hints (warning_details), color-coded by kind. A well-dimensioned daemon reports no hints. The capture below is an undersized one whose trace window is pinned near its cap.

  

• Energy shows the effective energy/carbon mix straight from the live green_summary: per service (effective source, measured share, energy, region) and per region (grid intensity, cold embedded vs hot real-time source). The window line names the energy source, for example source scaphandre_rapl on 2 of 5 services · rest modeled from I/O counts. A service none of whose operations was measured reads I/O counts as its source.

  

• Trends plots the poll history as live braille charts: energy and carbon per scoring window on top, and below them the Headroom chart, each runtime gauge (active_traces, analysis_queue_depth, stored_findings) as a percentage of its configured cap with the settings advisor's 90% threshold drawn in. When the effective grid intensity is static, the two top curves track each other. They diverge when the intensity moves, either from the live Electricity Maps real-time feed or from a shifting regional mix (the capture below stages the latter). One point lands per refresh tick, up to 240 points (20 minutes at the default 5 s), and the history lives in the monitor only: restarting it starts a fresh window. The capacity fields need a 0.8.8 daemon. Against an older one the Headroom panel degrades to a hint.

  

• Scrapers reads /api/energy for live backend health. Here cloud_energy is configured but its endpoint is unreachable, so its freshness age climbs while the unconfigured backends stay dashed.

  

• Config reads /api/config for the daemon's effective [daemon] settings, read-only. Each parameter shows its current value, the compiled-in default (computed locally), and a one-line explanation of what it does. Values that differ from the default are flagged modified. Secrets are summarized server-side (TLS as configured/not, the ack and read API keys as set/unset, the incident store as enabled/disabled) and never shown in clear. Needs a 0.8.8+ daemon, older ones show a hint.

  

• Incidents reads GET /api/incidents for the restarts and memory events your alerting posted to the daemon (since 0.20.0), newest first, each with the findings frozen from the window before it. Each incident shows one bold row, the alert's detail below it, then one indented row per finding. The bold row holds the start as local time, the service as ns/service when the alert carried a namespace, the kind, firing or ended, the finding count and a capture marker: complete, partial when the ring had already evicted part of the window so the list is short of what fired, or empty ring. A finding row shows its type, severity, endpoint, window tally and whether it fired before or after the restart, that marker leading the row so a narrow terminal never clips it. This is the one tab that needs an API key unless the daemon runs with [daemon] require_read_key = true, which needs it on every tab: pass --api-key-file <PATH> (or set PERF_SENTINEL_DAEMON_API_KEY), and the read-only [daemon] read_api_key suffices. Without a key the tab names the 401 and how to pass one. A daemon running with [daemon.incidents] enabled = false reads as a 503 hint, a daemon older than 0.20.0 as a 404 hint. None of the three flips [STALE], which tracks the report poll alone, and a transient failure keeps the last list on screen. The monitor polls the 20 newest incidents. perf-sentinel query incidents --offset pages further and --namespace narrows the list to one namespace.

  

When the daemon becomes unreachable, the last good snapshot stays on screen with a [STALE] indicator instead of going blank.

Terminal theme

Both TUIs (this monitor and the Inspect browser) follow the terminal's own theme: they use the 16 named ANSI colors and never impose a background, so a light-themed terminal renders them on a light background and a dark one on dark. Secondary text (titles, hints, column headers) is drawn with the terminal's dim attribute rather than a fixed gray, so it stays legible on either background. There is no --theme flag: configure the colors in your terminal emulator.

See also

• ACK-WORKFLOW.md for the relationship between TOML CI acks and daemon JSONL acks, and the full decision table.
• CLI.md for the perf-sentinel ack subcommand reference (CLI-side equivalent of a/u).
• HTML-REPORT.md for the browser-side ack flow via --daemon-url.
• QUERY-API.md for the /api/energy endpoint the Scrapers tab reads, and the daemon's read-only HTTP surface.
• CONFIGURATION.md for the [daemon.ack] server-side config reference.