diff --git a/README.md b/README.md index 8261140..660ffed 100644 --- a/README.md +++ b/README.md @@ -142,18 +142,21 @@ Cooldown behavior: - [`demos/rule-evaluation-and-dedup-demo/README.md`](demos/rule-evaluation-and-dedup-demo/README.md) explains the third demo and links its committed before/after dedup artifacts - [`demos/config-change-investigation-demo/README.md`](demos/config-change-investigation-demo/README.md) explains the config-change investigation demo and its committed artifacts +- [`docs/reviewer-pack.md`](docs/reviewer-pack.md) is the top-level no-guessing reviewer pack - [`docs/reviewer-path.md`](docs/reviewer-path.md) maps common review questions to the right demo and artifacts +- [`docs/architecture.md`](docs/architecture.md) diagrams the local file-based detection workflow - [`docs/sample-output.md`](docs/sample-output.md) summarizes the committed sample artifacts -- [`docs/roadmap.md`](docs/roadmap.md) sketches the next demo directions +- [`docs/roadmap.md`](docs/roadmap.md) sketches the v0.7 / v1.0 consolidation direction - [`data/processed/summary.json`](data/processed/summary.json) captures the default run in machine-readable form - [`data/processed/richer_sample/summary.json`](data/processed/richer_sample/summary.json) captures the richer scenario pack - [`tests/`](tests/) keeps regression coverage close to the CLI behavior and windowing logic -## Next Demo Directions +## v0.7 / v1.0 Direction -- add a focused auth/login anomaly triage walkthrough on top of the existing window features -- add a compact config-change drift follow-up scenario using the current deterministic evidence model -- keep sample-output docs and public repo presentation aligned with the checked-in demo state +- stabilize the four-demo matrix and avoid broad platform expansion +- freeze reviewer-visible artifact names unless a rename is intentionally coordinated across docs, tests, and sample outputs +- use [`docs/reviewer-pack.md`](docs/reviewer-pack.md) and [`docs/architecture.md`](docs/architecture.md) as the consolidation entrypoints +- add at most one more demo before v1-style consolidation ## Scope diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..b125aec --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,43 @@ +# Architecture + +`telemetry-lab` is a local, file-based detection workflow lab. It is organized around committed sample inputs, deterministic demo pipelines, and reviewer-facing artifacts. + +```mermaid +flowchart TD + Inputs["Committed sample inputs
JSONL, CSV, YAML configs"] + CLI["Local CLI entrypoints
run, run-ai-demo, run-rule-dedup-demo, run-config-change-demo"] + Window["telemetry-window-demo
normalize -> windows -> features -> alerts"] + AI["ai-assisted-detection-demo
rules -> cases -> JSON-only drafting"] + Dedup["rule-evaluation-and-dedup-demo
raw hits -> cooldown -> suppression reasons"] + Config["config-change-investigation-demo
config changes -> bounded evidence correlation"] + Artifacts["Reviewer artifacts
CSV, JSON, JSONL, Markdown, PNG"] + Review["Reviewer inspection
README, reviewer path, reviewer pack, tests"] + + Inputs --> CLI + CLI --> Window + CLI --> AI + CLI --> Dedup + CLI --> Config + Window --> Artifacts + AI --> Artifacts + Dedup --> Artifacts + Config --> Artifacts + Artifacts --> Review +``` + +## Design Rules + +- Detection decisions stay deterministic and inspectable. +- The AI-assisted demo is limited to bounded JSON-only case drafting. +- Artifacts are file-based and suitable for local regeneration or GitHub review. +- Artifact names are reviewer-visible contracts during the v0.7 / v1.0 consolidation path. +- The repository does not provide production monitoring, real-time ingestion, dashboards, alert routing, case management, autonomous response, or final incident verdicts. + +## Demo Boundaries + +| Demo | Boundary | +| --- | --- | +| `telemetry-window-demo` | Converts raw events into window features and alerts; does not become a live stream processor. | +| `ai-assisted-detection-demo` | Drafts constrained summaries from deterministic cases; does not decide incident outcomes or call tools. | +| `rule-evaluation-and-dedup-demo` | Explains cooldown and suppression behavior; does not route alerts. | +| `config-change-investigation-demo` | Correlates risky changes with bounded local evidence; does not monitor live infrastructure. | diff --git a/docs/reviewer-brief.md b/docs/reviewer-brief.md index 42d3b9d..fc7c97e 100644 --- a/docs/reviewer-brief.md +++ b/docs/reviewer-brief.md @@ -71,4 +71,4 @@ The other demos emit reviewer-facing artifacts such as `dedup_report.md`, `inves ## Next milestone -Add another compact investigation path that strengthens the bridge from telemetry features to analyst-facing monitoring and detection workflows. +Consolidate the current reviewer path for v0.7 / v1.0: keep the demo matrix stable, freeze reviewer-visible artifact names, maintain one top-level reviewer pack, and keep the architecture diagram aligned with actual CLI and artifact behavior. diff --git a/docs/reviewer-pack.md b/docs/reviewer-pack.md new file mode 100644 index 0000000..6f76404 --- /dev/null +++ b/docs/reviewer-pack.md @@ -0,0 +1,68 @@ +# Reviewer Pack + +This is the top-level reviewer pack for `telemetry-lab`. + +Use it as a short, no-guessing path through the repository. The project is a local, file-based detection workflow lab: it is not a SIEM, not a dashboard, and not a production monitoring platform. + +## What to Review + +Start with the stable demo matrix in [`docs/reviewer-path.md`](reviewer-path.md). It maps common review questions to the demo and artifacts that answer them: + +| Review question | Demo | Primary evidence | +| --- | --- | --- | +| How are raw events converted to alert features? | `telemetry-window-demo` | `data/processed/features.csv`, `data/processed/alerts.csv`, `data/processed/summary.json` | +| How is AI constrained? | `ai-assisted-detection-demo` | `demos/ai-assisted-detection-demo/artifacts/case_summaries.json`, `demos/ai-assisted-detection-demo/artifacts/audit_traces.jsonl`, guardrails in `demos/ai-assisted-detection-demo/README.md` | +| How are duplicate alerts reduced? | `rule-evaluation-and-dedup-demo` | `demos/rule-evaluation-and-dedup-demo/artifacts/rule_hits_before_dedup.json`, `demos/rule-evaluation-and-dedup-demo/artifacts/rule_hits_after_dedup.json`, `demos/rule-evaluation-and-dedup-demo/artifacts/dedup_explanations.json` | +| How are risky config changes investigated? | `config-change-investigation-demo` | `demos/config-change-investigation-demo/artifacts/investigation_hits.json`, `demos/config-change-investigation-demo/artifacts/investigation_report.md` | + +## Architecture + +See [`docs/architecture.md`](architecture.md) for the repository-level workflow diagram. + +The important shape is: + +1. committed sample inputs and configs +2. deterministic local pipelines +3. bounded AI drafting only where explicitly marked +4. committed or regenerated artifacts +5. reviewer inspection through docs, tests, and reports + +## Artifact Naming Contract + +The current artifact names are reviewer-facing contracts for the v0.7 / v1.0 consolidation path. + +- Keep the demo matrix stable unless a demo is intentionally retired. +- Prefer additive artifacts over renaming existing reviewer-visible outputs. +- If an artifact must be renamed, update the README, reviewer path, this reviewer pack, demo README, tests, and committed sample outputs in the same change. +- Do not introduce platform-style names that imply alert routing, case management, real-time ingestion, or autonomous response. + +## Fast Verification + +From the repository root: + +```bash +python -m pip install -e ".[dev]" +python -m telemetry_window_demo.cli run --config configs/default.yaml +python -m telemetry_window_demo.cli run-ai-demo +python -m telemetry_window_demo.cli run-rule-dedup-demo +python -m telemetry_window_demo.cli run-config-change-demo +pytest +``` + +Use the same Python interpreter for install, tests, and demo commands. + +## Review Anchors + +- [`docs/reviewer-brief.md`](reviewer-brief.md): short problem / value summary +- [`docs/reviewer-path.md`](reviewer-path.md): demo choice by review question +- [`docs/architecture.md`](architecture.md): local file-based workflow diagram +- [`docs/sample-output.md`](sample-output.md): committed output counts and sample artifacts +- [`tests/test_reviewer_docs.py`](../tests/test_reviewer_docs.py): regression checks for reviewer-facing docs + +## Boundaries + +- No production monitoring claims +- No real-time ingestion or streaming state +- No dashboard, alert routing, or case-management service +- No autonomous response actions +- No final incident verdicts from the AI-assisted demo diff --git a/docs/roadmap.md b/docs/roadmap.md index ea9be9b..01ab5c7 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,25 +1,40 @@ -# Roadmap - -This repository is intentionally small, so the next steps should be new demos that make the existing telemetry pipeline easier to understand rather than a broad platform build-out. - +# Roadmap + +`telemetry-lab` is moving from demo expansion toward v0.7 / v1.0 consolidation. + +The repo now has four reviewer-verifiable demos and a clear [`docs/reviewer-path.md`](reviewer-path.md). The priority is to keep the demo matrix stable, preserve artifact names, and make review faster without implying a SIEM, dashboard, or production monitoring platform. + Recently added: - [rule-evaluation-and-dedup-demo](../demos/rule-evaluation-and-dedup-demo/README.md) now shows raw rule hits, retained alerts, and suppression reasons side by side. - [config-change-investigation-demo](../demos/config-change-investigation-demo/README.md) now shows risky configuration changes, bounded evidence attachment, and deterministic investigation summaries. -- Core telemetry-window validation now gives clearer failures for malformed inputs, invalid run configs, rule parameters, plot CSV values, and output window bounds. - -## 1. Auth/Login Anomaly Triage Demo - -Goal: -Add a demo that walks from bursty login failures into follow-on signals such as source spread, eventual success, or repeated target concentration. - -Why it helps the portfolio: -This strengthens the repo's analyst-facing story. It shows how simple window features and rule output can support a concrete triage narrative instead of stopping at generic alert generation. - -## 2. Config-Change Drift Follow-Up Demo - -Goal: -Add a compact follow-up scenario centered on repeated config drift, rollback attempts, and evidence that the remediation path actually reduced nearby denials or noisy follow-on signals. - -Why it helps the portfolio: -This would build on the current config-change investigation demo without changing the repo's local, file-based character. It would show not just the initial risky change, but also how deterministic evidence can support a short remediation narrative. +- [`docs/reviewer-path.md`](reviewer-path.md) maps common review questions to the right demo and artifacts. +- [`docs/reviewer-pack.md`](reviewer-pack.md) collects the top-level reviewer flow and artifact naming contract. +- [`docs/architecture.md`](architecture.md) describes the local file-based workflow shape. + +## v0.7 / v1.0 Consolidation + +1. Stabilize the demo matrix. +2. Freeze reviewer-visible artifact names unless a rename is intentional and documented across README, reviewer docs, demo docs, tests, and sample outputs. +3. Keep one top-level reviewer pack as the primary no-guessing entrypoint. +4. Keep the architecture diagram aligned with actual CLI and artifact behavior. +5. Prefer regression tests and documentation accuracy over adding new workflow surface area. + +## Optional Final Demo + +At most one more demo should be added before v1-style consolidation. + +Good candidates: + +- auth/login anomaly triage from bursty login failures into follow-on signals +- config-change drift follow-up showing rollback attempts and reduced nearby denials + +Only add one if it clearly strengthens the detection workflow portfolio without turning the repo into a platform. + +## Non-Directions + +- No production monitoring claims +- No realtime ingestion or streaming state +- No dashboard or service deployment +- No alert routing or case management +- No autonomous response diff --git a/tests/test_reviewer_docs.py b/tests/test_reviewer_docs.py index 11abfa6..f79c9f9 100644 --- a/tests/test_reviewer_docs.py +++ b/tests/test_reviewer_docs.py @@ -74,4 +74,33 @@ def test_readme_links_reviewer_path_and_uses_lab_framing() -> None: assert "A local, file-based detection workflow lab" in readme assert "not a SIEM, dashboard, or monitoring platform" in readme + assert "[`docs/reviewer-pack.md`](docs/reviewer-pack.md)" in readme assert "[`docs/reviewer-path.md`](docs/reviewer-path.md)" in readme + assert "[`docs/architecture.md`](docs/architecture.md)" in readme + + +def test_top_level_reviewer_pack_covers_matrix_and_artifact_contract() -> None: + reviewer_pack = _read_repo_file("docs/reviewer-pack.md") + + assert "top-level reviewer pack" in reviewer_pack + assert "Artifact Naming Contract" in reviewer_pack + assert "[`docs/reviewer-path.md`](reviewer-path.md)" in reviewer_pack + assert "[`docs/architecture.md`](architecture.md)" in reviewer_pack + + for question, demo_name, artifact_paths in REVIEWER_DEMO_MATRIX: + assert question in reviewer_pack + assert f"`{demo_name}`" in reviewer_pack + for artifact_path in artifact_paths: + assert f"`{artifact_path}`" in reviewer_pack + + +def test_architecture_doc_keeps_local_file_based_boundaries() -> None: + architecture = _read_repo_file("docs/architecture.md") + + assert "flowchart TD" in architecture + assert "local, file-based detection workflow lab" in architecture + assert "Artifact names are reviewer-visible contracts" in architecture + assert "does not provide production monitoring" in architecture + + for _, demo_name, _ in REVIEWER_DEMO_MATRIX: + assert f"`{demo_name}`" in architecture