Authored by Aadarsh Kadam. This document is the institutional memory of the DetectionForge build. Each entry is a structured mini-postmortem: what we set out to do, what problem surfaced, how we diagnosed it, what decision was made and why, what got committed, and what was learned. Readers with access to the git history can verify every claim — commit SHAs are cited throughout.
Audience: a technical interviewer, a future collaborator, or the author six months from now.
What we set out to do. Stand up a working repository skeleton with the five-stage pipeline (lint → test → convert → score → build) wired as CLI stubs, dependency manifest, Docker target, and directory structure. The goal was a pip install -e .[dev] that succeeds and a forge lint that exits 0, with no pipeline logic yet filled in.
What happened. The scaffold landed cleanly in commit eccde8f (2026-05-05). All five stages were stub files (4-line pass bodies), pyproject.toml built against hatchling, Click, Pydantic, pySigma, and pyyaml. Directory structure matched PDR §2: rules/, data/attack/, data/benign/, docs/adr/, scripts/, site/, tests/. The first rule — T1059.001 — was authored as a placeholder with synthetic positive fixtures to exercise the data shape.
What was learned. The synthetic fixture decision was made quickly and provisionally. Within the same phase it was reversed: T1547.001 (registry run key) had a synthetic fixture assembled from Atomic Red Team docs, contradicting the core project principle of testing against real capture data. Commit 8e23d85 (2026-05-06) swapped it for T1053.005 (scheduled task persistence) backed by OTRF SDWIN-200921175806 — the first real dataset in the project. The lesson was recorded immediately: synthetic fixtures are a measurement liability, not a shortcut. They produce green tests that prove the test, not the detection.
What was committed. eccde8f — initial scaffold. 8e23d85 — T1547.001 → T1053.005 swap, first real OTRF dataset.
What we set out to do. Implement forge test: a command that evaluates each rule's detection logic against its positive and negative fixtures and reports precision, recall, and F1. The natural first question was whether pySigma — already a dependency for forge convert — could serve double-duty as an in-process rule evaluator.
What problem surfaced. An exhaustive search of the pySigma API produced nothing. There is no match() method, no evaluate(event_dict) path, and no documented bridge from a parsed SigmaRule object to a Python boolean over a real log event. pySigma's architecture is a one-way compiler: parse rule → transform via processing pipeline → emit query string. Step 3 is mandatory. The match methods that do exist live in sigma.processing.conditions and operate on SigmaString internal types used during rule transformation, not on real JSON events.
How we diagnosed it. A 20-minute spike ran both T1059.001 and T1003.001 through SigmaCollection.from_yaml() → SplunkBackend().convert(). Compilation worked cleanly (2–4ms per rule). But tracing inward from SigmaRule, SigmaDetection, SigmaDetectionItem, ConditionAND, ConditionOR found no evaluation surface. Every submodule was explored.
What decision was made and why. ADR-002 formalized the split: pySigma for forge convert (query-string generation), a hand-rolled evaluator for forge test (Python booleans over event dicts). The hand-rolled path was lighter than either alternative — spinning up a live Splunk process or writing a shim that re-parses pySigma's AST. Critically, ADR-002 contains a discipline rule: the evaluator supports exactly the modifiers present in the current rule set (|endswith, |contains, |contains|all, |startswith). Adding a modifier requires a real rule that needs it and a test for that modifier in the same commit. The evaluator is deliberately not a reimplementation of the Sigma spec.
What was committed. 71311e1 (2026-05-06) — forge/test_harness.py (360 lines), four modifiers, recursive and/not condition parser, reports/results.json output, three PDR §12 test cases, and ADR-002 (85 lines). forge test exits 0 on all-pass, 2 on any regression. The spike result is recorded verbatim in the ADR so any future reader understands why the split exists.
What we set out to do. Replace the 3-event synthetic baseline with real background events drawn from OTRF captures. The design choice was forced early: single shared workstation_baseline.json (Option A) versus per-rule baselines. Option A was chosen because it measures cross-technique false positives honestly — a discovery rule that fires on a credential-access background event is a real FP, not a hidden one.
What happened. Commit 7f8a97b (2026-05-06) replaced the synthetic baseline with 50 EID 1 background events extracted from SDWIN-200921175806 (the empire_schtasks ZIP). The events came from background lab activity: svchost, .NET workers, Windows Defender. scripts/extract_benign.py was added for reproducible extraction, and data/captures/_decisions.md was created as the verification log — the evidentiary record for every dataset opened, including future rejections.
What was learned. The background extraction introduced the first cross-technique contamination problem, which only became visible much later in Phase 3. The SDWIN-200921175806 ZIP contained exactly the kind of benign background the project needed — but other OTRF ZIPs would later be found to have attack events in their "background" data. The Option A decision made those contamination events visible as measured FPs rather than invisible behind per-rule filtering. That was the correct choice: measured FPs drive rule improvement; hidden ones don't.
What was committed. 7f8a97b — 50-event real baseline, extraction script, decisions log initialized.
What we set out to do. Author five new rules using real OTRF positive fixtures. Each rule required opening a ZIP, confirming EID 1 events existed, extracting the attack signal, and authoring detection logic against it. Dataset selection and rejection were logged in _decisions.md as the evidence chain.
What happened. Two of the five planned techniques had no usable OTRF EID 1 data. T1218.011 (rundll32, non-comsvcs variant) had no OTRF dataset with EID 1 coverage. T1562.001 (Defender disable) likewise. Both were swapped within the pre-approved limit: T1218.010 (regsvr32 squiblydoo, SDWIN-200721232741) and T1033 (system owner via whoami, SDWIN-200904032946) were confirmed from already-downloaded ZIPs. The five rules that landed: T1033, T1055.001, T1087.001, T1218.010, T1548.002. The benign baseline grew from 50 to 74 events (background from five new ZIPs).
T1059.001's precision dropped to 0.800 in this commit. The cause diagnosed at the time: the T1218.010 ZIP's background contained a powershell.exe -enc event (a regsvr32 follow-on stager) that the T1059.001 rule correctly fires on. This was not a rule defect — the detection logic is sound — but the event's presence in the benign pool was an Option A artifact. The FP was documented in the rule's falsepositives field as a known corpus-labeling artifact to be resolved in Phase 3.
Footnote (superseded in Phase 3): This Phase 2 explanation was partially revised in Phase 3. See Entry 3.3 — most events of this shape were later identified as duplicate attack events leaking via the dedup-key bug (Computer field included in dedup key allowed the same attack event from different OTRF lab VMs to remain in the baseline). The genuinely cross-technique events were promoted to T1059.001 positive fixtures during the Category 2 expansion rather than retained as FPs. The precision became 1.000 after the structural filter and Category 2 expansion. The Phase 2 framing was not wrong about the event being a real T1059.001 indicator — it was incomplete about why the event was in the baseline at all.
What was committed. 41c2c8e (2026-05-07) — five new rules, updated fixtures for T1003.001 and T1059.001, benign baseline at 74 events, and full decisions log entries for all Day 2 datasets.
What we set out to do. Each build phase was supposed to end with an explicit gate-clearance request before the next phase began. Phase 2 Day 3 and Day 4 were built and committed without the project lead clearing the preceding gate.
What happened. Commit 5a4a135 (forge score) was delivered as Phase 2 Day 4 without an explicit gate conversation after Day 3's precision re-measurement deliverable. The Day 3 work (T1059.001 FP documentation, benign ZIP _meta.yml files, commit ab29a9a) was technically sound, but the gate was bypassed. The project lead caught the violation during a session review and required a full evidenced accounting.
How we diagnosed it. Looking at the commit log, ab29a9a and 5a4a135 are back-to-back with no gate conversation between them. The pattern is identifiable: two consecutive chunk commits with no recorded approval message.
What decision was made and why. ADR-003 was written as a self-referential contract: every chunk completion summary must end with the phrase "requesting gate clearance" before the next chunk begins. The phrase is a hard-to-miss checkpoint that survives context summarization. The ADR records the real incident — the gate was bypassed, the project lead caught it — rather than being written speculatively. Retroactive log entry 7e28cb3 documented the gate compression in _decisions.md.
What was committed. 4fbe6a5 — ADR-003. 7e28cb3 — retroactive gate violation log.
What was learned. Build momentum in long sessions is a real process risk. An explicit phrase in the completion summary is a better safeguard than a general instruction. The ADR's consequence section notes that gate violations are identifiable from the git log — which is exactly how this one was caught.
What we set out to do. Scale the benign baseline from 74 to 3,000+ events by batch-extracting background EID 1 events from all OTRF atomic Windows ZIPs. scripts/batch_extract_benign.py would enumerate all 114 ZIPs via the GitHub Contents API and process them in one run.
What problem surfaced. The first full run produced contaminated output. Windows image paths (e.g., C:\Windows\System32\calc.exe) use backslashes. Python's Path(windows_path).name on Linux returns the entire string as the filename — backslash is not a path separator on Linux. The attacker-tool exclusion filter (if image_basename in ATTACKER_TOOLS) was silently passing everything because image_basename was C:\Windows\System32\calc.exe rather than calc.exe. Seven directories that had been extracted before the bug was found had to be re-extracted.
How we diagnosed it. The bug was caught mid-run when the extracted event lists for certain ZIPs contained calc.exe and notepad.exe events that should have been excluded. Inspecting the exclusion logic revealed the Path.name behavior. Fix: image.replace("\\", "/").split("/")[-1].lower() for Windows basename extraction on Linux.
What was committed. 1048ab9 (2026-05-08) — 105 ZIPs successfully extracted, 1,565 new background events, batch extraction script, and the path-bug fix documented in the commit message. The run log is at data/benign/_batch_run_log.json.
What was learned. Windows path handling on Linux requires explicit / splitting. Path.name is not safe for cross-platform string paths. The provenance search for T1059.001's two unconfirmed events (UtcTime 2020-10-18, Empire DNS+HTTP stager) exhausted all 114 atomic ZIPs with no match — surfacing the need for compound dataset inspection in Day 2.
What we set out to do. Phase 3 Day 1 began with forge test showing a regression: one rule was failing lint and the test harness. Diagnosis needed before corpus expansion could proceed.
What problem surfaced. T1110_003_password_spraying.yml had been scaffolded during a Phase 2 Day 5 forge new demo and never removed from rules/windows/credential_access/. It had lived there for the entirety of Phase 3 Day 1, failing lint on every run. The failure was suppressed in session summaries because summaries only reported passing rules. The catch-all EventID: 1 detection logic with an empty fixture produced precision=0 and an exit 2.
Why it wasn't caught sooner. forge new scaffolds a rule with TODO detection logic and an empty positive fixture. Both immediately fail lint. But there was no mechanism to quarantine unfinished rules — they were either in the active rule path or deleted. Since deletion would lose the work, the file stayed in rules/ and silently broke the build.
What decision was made and why. ADR-004 introduced rules/_drafts/ as a quarantine directory. Lint and test skip any path whose parts include _drafts. Both tools report the draft count at end of run: "Skipped N draft rule(s) in rules/_drafts/." forge new now writes to _drafts/ by default and prints the graduation path. A later amendment (Phase 3 Day 4) added _drafts/UNSOURCED/ for rules gated on harness or corpus infrastructure changes rather than authoring time — the naming distinction makes the nature of the blockage immediately visible to a reviewer browsing the repo.
What was committed. 96b8751 (2026-05-08) — two filter lines (lint.py, test_harness.py), one path change and one print in new.py, ADR-004, and T1110_003_password_spraying.yml moved to rules/_drafts/ as the first exercise of the convention.
What we set out to do. Merge the 1,508-event post-dedup baseline (from 119 ZIPs: 114 atomic + 8 compound LSASS_campaign + 1 apt29_evals_day1_manual) into workstation_baseline.json and run forge test to measure the precision impact.
What problem surfaced. forge test returned catastrophic precision numbers. T1087.001 dropped from 1.000 to 0.167 — a rule that had been clean was now hitting 15 FPs out of 16 fires. Other rules showed similar drops. The initial instinct was to look for cross-technique contamination in the new baseline events, and an Option A categorization scheme (Category 1: noise, Category 2: same-technique different-instance, Category 3: rule defect) was proposed to triage the FPs. The user rejected all proposed options because the categorization itself was wrong.
The correct diagnosis. Most of the "FPs" were not benign events matching the rule — they were duplicate attack events. The deduplication had been run on (Image, CommandLine, ParentImage, Computer). The Computer field differs across OTRF lab VMs: the same attack command run on WORKSTATION6.theshire.local and MORDORDC.theshire.local produces two events with identical (Image, CommandLine, ParentImage) but different Computer values. After dedup, both events remained in the pool. When they appeared in the baseline, they matched the rule's detection logic because they were attack events — just with a different Computer field than the ones already in the positive fixture.
The fix: ADR-005 structural attack-event filter. The baseline was re-filtered on hash(Image, CommandLine, ParentImage) — Computer and timestamps excluded. Any event whose hash matched a positive fixture event was removed from the baseline, regardless of Computer or timestamp differences. This is the labeling axiom: an event that is identical to a known attack event in every field that determines detection behavior cannot simultaneously serve as a false positive for that rule.
Two passes were run: Pass 1 before fixture expansion (9 events removed), Pass 2 after Category 2 fixture expansion (18 additional events removed). The 25 Category 2 FPs that remained after Pass 1 were legitimate same-technique different-instance events — confirmed attack artifacts not yet in the positive fixture. These were promoted to fixtures with full provenance documentation. The 2 Category 3 FPs (T1087.001 catching T1136.001 net user /add) were retained deliberately, documented, and the fix was deferred to Phase 3 Day 4 with a promised before/after measurement.
What was committed. 3bd07a8 (2026-05-09) — structural filter script (scripts/apply_structural_filter.py), ADR-005, fixture expansion (+18 TPs across 4 rules), baseline at 1,481 events, and docs/measurements/phase3-day2-precision-delta.md with the full delta table. Final measurements: mean precision 0.975 / recall 1.000 / F1 0.986 across 8 rules; T1087.001 at 0.800 by design.
Footnote (superseded in Phase 4): The "1,481 events" figure was a raw concatenation count that included 419 cross-dataset duplicate events. The correct unique-event count for this corpus under strict (Image, CommandLine, ParentImage, Computer) deduplication was approximately 1,051. The precision measurements remain valid — precision arithmetic is not affected by duplicate-counting in the benign pool — but the headline corpus size was overstated. See Entry 4.3 for the full accounting and correction.
What was learned. Deduplication key selection is a measurement design decision, not a data-cleaning step. Including Computer in the dedup key is correct for event uniqueness but wrong for the structural filter, because the same attack behavior looks different across lab VMs only in fields that don't affect detection. The two operations need to be separated: dedup on full identity (including Computer) to avoid inflating event counts, then filter on detection-relevant identity (excluding Computer) to remove measurement artifacts.
What we set out to do. Implement forge convert: compile each active Sigma rule to three SIEM query languages (Splunk SPL, Elastic EQL, Microsoft Sentinel KQL) using pySigma backends. PRD §8 required a ≥95% success rate across the full rule-backend matrix.
What happened. The pySigma compilation API, which had been spiked in ADR-002, composed cleanly: SigmaCollection.from_yaml() → backend.convert() in two calls. Three backends were wired: pySigma-backend-splunk 2.1.0, pySigma-backend-elasticsearch 2.0.2, pySigma-backend-kusto 1.0.1. The per-rule failure protocol was: silent-continue on any single rule-backend failure, compute success rate across the full matrix, exit non-zero if rate fell below the forge.toml threshold.
Phase 3 Day 3 result: 24/24 conversions succeeded — 100.0% across 8 rules × 3 backends. No translation losses. All Sigma modifiers in the current rule set had direct equivalents in all three target languages. The conversion matrix and full query output are at docs/measurements/phase3-day3-conversion-matrix.md.
What was committed. 0f152a8 (2026-05-10) — forge/convert.py (296 lines), CLI integration, measurement doc, annotated example for T1003.001.
What was learned. The Sigma modifier set constraint from ADR-002 paid off here. Keeping the rule set to four modifiers meant the conversion matrix was clean with no edge cases. A rule set that used |re (regex), |cidr, or |windash would have required additional backend mapping work and risked partial-failure scenarios.
What we set out to do. In Phase 3 Day 4, close the documented T1087.001 rule defect that had been retained in the Phase 3 Day 2 measurements. The defect had a precision of 0.800 with 2 known false positives, and a before/after measurement was promised at declaration time.
What the defect was. T1087.001's detection.selection included CommandLine|contains: [user, localgroup, group]. The user keyword was over-broad: it matched not only net user (account discovery) but also net user /add (Create Account — T1136.001). Two events in the 1,481-event corpus were T1136.001 artifacts, not T1087.001 discovery: net.exe user /add backdoor paw0rd1 (parent: WmiPrvSE.exe) and net1.exe user /add backdoor paw0rd1 (parent: net.exe). Both were Empire lateral movement events from the same OTRF dataset that later populated the T1136.001 positive fixture.
The fix. A filter_account_creation block using CommandLine|contains: ' /add ' (with surrounding spaces for flag-boundary matching) excluded the false positives. The filter_system block (excluding services.exe/svchost.exe parents) was removed simultaneously — it caught 0 events in the 1,481-event corpus and was dead code. Empirical removal with documented provenance is better than retaining untestable filters.
Before/after measurement. Precision 0.800 → 1.000. Recall held at 1.000. This is the iterative-improvement loop the project's measurement infrastructure was built to demonstrate: the harness detects a defect at corpus expansion time, the defect is logged honestly with a measured number, the fix produces a measurable improvement, and the entire loop is reproducible from git history. docs/measurements/phase3-day4-t1087-fix.md records the full chain of evidence.
What was committed. c9d93d2 — 4-line YAML change plus measurement doc.
What we set out to do. Phase 3 Day 4 included attempting to author T1110.003 (password spraying) and T1547.001 (registry run key persistence) — two techniques that had been deferred or left in draft state.
T1110.003 investigation. OTRF data exists: SDWIN-201022042947 (purplesharp_ad_playbook_I) has 7 EID 4625 events with a textbook spray signature (7 distinct TargetUserNames, 12-millisecond window, all SubStatus 0xc000006a). But two structural blockers ruled out an honest evaluation: (1) the hand-rolled evaluator per ADR-002 supports only four per-event modifiers — honest spray detection requires count() by + timeframe aggregation, which the harness does not implement; (2) the 1,481-event baseline contains zero EID 4625 events, so a rule selecting on EventID 4625 would report precision = 1.0 by corpus composition, not detection quality.
T1547.001 investigation. Registry run key persistence fires on Sysmon EID 12/13 (registry events), not EID 1. The entire baseline and all existing fixtures are EID 1 only. Same structural mismatch as T1110.003: a rule selecting on registry events against an EID 1 baseline has no opportunity to false-positive, so precision = 1.0 is guaranteed — and meaningless.
The lesson formalized. Both deferrals produced the same generalization, which was written into _decisions.md and the README: "A rule's precision number is only meaningful when the benign baseline contains events of the same logsource and event type the rule selects on. Cross-logsource testing is structural mismatch, not measurement." This principle constrains all future Phase 4 work: corpus expansion must be paired with the rules it enables.
What was committed. 7d42022 — T1110.003 deferral note with full Phase 4 prerequisites. a4d563c — Phase 3 Day 4 closure including T1547.001 parallel deferral. Both rules moved to rules/_drafts/UNSOURCED/.
What we set out to do. Close the density gap from 8 to 13 rules. Candidate techniques were proposed from OTRF data with direct ZIP inspection — not from training-data recall. Each rule: real OTRF fixture, precision ≥ expected threshold, all three SIEM conversions passing, ADR-005 structural filter updated after each rule's fixtures landed.
What happened. Five rules landed cleanly:
- T1136.001 Create Account via
net user /addand WMIC remote exec (2 OTRF datasets) - T1003.002 SAM hive copy via
esentutl /vss(1 OTRF dataset — single-event fixture, documented) - T1218.005 mshta.exe remote payload execution (2 OTRF datasets after a mid-authoring FP was resolved: a
vbscript:variant in cmd_mshta_vbscript_execute_psh was correctly promoted to a Category 2 fixture) - T1105 Ingress Tool Transfer via BITSAdmin (1 OTRF dataset — single-event fixture, documented; dual ATT&CK mapping noted: T1105 Ingress Transfer vs T1197 BITS Jobs)
- T1021.006 WinRM remote command via wsmprovhost (3 OTRF datasets: Empire, Covenant, PurpleSharp — framework diversity documented explicitly)
Tactic coverage grew from 6 to 8 (added command-and-control, lateral-movement). All 13 rules at precision 1.000 / recall 1.000 / F1 1.000. SIEM conversion: 39/39 (100%) on the expanded matrix.
One notable T1218.005 precision event: the first test run returned precision 0.500 — one FP. Inspection found the vbscript: command-line variant in cmd_mshta_vbscript_execute_psh, a different mshta execution path that the rule correctly fires on. It was promoted to a Category 2 fixture rather than excluded, bringing precision to 1.000. This is the Category 2 protocol working as designed: when the rule fires on a real attack event not yet in the fixture, you add it to the fixture, not relax the rule.
What was committed. c7fe487, 8df5983, 20050bc, 284012d, aaf5402 — one commit per rule. a4d563c — Phase 3 Day 4 closure documentation in _decisions.md.
What was learned. The single-event fixture caveats (T1003.002, T1105, T1218.010, T1548.002) are honest limitations, not failures. Binary recall against a single event is structurally 1.000 but does not mean all real-world variants are covered. Documenting this explicitly in the README and in the _meta.yml files provides an honest picture: recall coverage reflects framework and variant diversity in the OTRF corpus, not all possible attacker implementations.
Footnote (superseded in Phase 4): The Phase 3 Day 4 close measurements were taken against a 1,481-event baseline that contained 419 cross-dataset duplicate events. The correct unique-event count under strict dedup was ~1,051. The mean precision 1.000 / recall 1.000 / F1 1.000 figures remain valid — precision arithmetic is not affected by duplicate-counting in the benign pool. The Phase 4 corpus rebuild corrects the baseline to 1,173 unique events (expanded with two new datasets). See Entry 4.3.
What happened. During the Phase 3 Day 2 compound dataset extraction run, the session was interrupted. The apt29_evals_day2_manual.zip download stalled — a 41MB capture that would not complete within the session's time budget. The corpus was closed at 1,481 events without it. The seven LSASS_campaign compound ZIPs and apt29_evals_day1_manual were successfully committed.
Recovery. The batch extraction script had been written with idempotent directory handling — re-running it would add new data rather than duplicating existing data. The crash left no inconsistent state in the committed corpus because the baseline was rebuilt from per-directory events.json files, not from a partially-written merged file. The decision to defer apt29_evals_day2_manual was logged in _decisions.md with a Phase 4 retry action item and explicit reasoning: Monday deadline made retrying this ZIP schedule-inadvisable given marginal corpus value.
What was learned. The per-directory storage pattern (data/benign/<dataset_slug>/events.json) plus the merge-on-build approach was the correct choice for crash resilience. A single monolithic baseline file updated in-place would have been unrecoverable. The architecture decision from Phase 2 Day 1 (Option 1: separate attack/ and benign/ directories) paid a concrete dividend here.
Relationship to the T1110.003 latent regression. The crash recovery was also when the T1110.003 latent build breakage was discovered. Running forge test from a clean state after recovery surfaced the empty-fixture precision=0 failure that led directly to ADR-004. The crash was the forcing function for the architectural fix.
Three snapshots that capture the full arc of measurement discipline: Phase 2 close (small corpus, first real baseline), Phase 3 Day 2 close (corpus expanded 20×, precision-delta crisis resolved), Phase 3 Day 4 close (five new rules, all metrics at ceiling).
| Metric | Phase 2 close (74-event corpus, 8 rules) | Phase 3 Day 2 close (1,481-event corpus†, 8 rules) | Phase 3 Day 4 close (1,481-event corpus†, 13 rules) |
|---|---|---|---|
| Rules | 8 | 8 | 13 |
| Mean precision | 0.975* | 0.975 | 1.000 |
| Mean recall | 1.000 | 1.000 | 1.000 |
| Mean F1 | 0.986 | 0.986 | 1.000 |
| Tactics covered | 6 | 6 | 8 |
| Corpus (benign events) | 74 | 1,481† | 1,481† |
| SIEM conversion | — | — | 39/39 (100%) |
| Deferred rules | 0 | 0 | 2 (T1110.003, T1547.001) |
* Phase 2 close mean precision was 0.975 (seven rules at 1.000, T1059.001 at 0.800). The Phase 3 Day 2 re-measurement of the same 8 rules against the expanded corpus resolved T1059.001's 0.800 but surfaced T1087.001's defect, leaving the mean unchanged at 0.975. These are separate measurement events against different corpora — the middle column is the precision-delta crisis artifact, not a continuation of the Phase 2 number.
† The 1,481-event figure reported at Phase 3 Day 2 and Day 4 close was a raw concatenation count including 419 cross-dataset duplicate events. The correct unique-event count under strict deduplication was ~1,051. The precision figures are not affected by this correction — see Entry 4.3 for full accounting.
This document was backfilled through Phase 3 Day 4 in commit 5b0825d. All subsequent phase entries are appended in real time.
The original target. PRD §8 specified ≥60 rules as the v1 completeness criterion. That number was calibrated against existing detection-as-code repositories before any rules existed: Sigma HQ maintains hundreds of community rules; Elastic's detection-rules repo has hundreds more. Against that landscape, 60 rules seemed like a credible portfolio threshold.
What Phase 3 revealed. Each rule in this project carries a fixed cost that the PRD estimate did not account for: dataset selection and ZIP inspection against OTRF's raw archive, positive fixture extraction, attacker-tool exclusion list verification, ADR-005 structural filter regeneration, three SIEM backend conversions via pySigma, a corpus decisions log entry, and a journey doc entry documenting what was found and what was decided. In Phase 3 Day 4, five rules were authored end-to-end in approximately 7.5 hours — 90 minutes per rule. Producing 47 additional rules from the current base of 13 requires approximately 70 hours of focused authoring time. That is a second project, not a polish sprint.
The cost alone would not justify a scope change. The second finding did. Past approximately 20 rules across 8+ tactics, additional rule count produces diminishing returns on the project's load-bearing claim. The claim that carries interview weight is: "I built a measurement infrastructure that detects precision regressions as the benign corpus grows, triages false positives by root cause, enforces logsource coherence, and documents scope decisions based on evidence from the data." That claim is fully demonstrated at 20 rules. A reviewer who reads the journey doc at 20 rules and at 60 rules asks the same question: "does this person understand detection engineering at production depth?" The answer does not change.
The decision. v1 launches with ~20 rules across 9–10 tactics. The PRD §8 target of ≥60 rules is reframed as a v2+ aspiration in the README alongside an explicit reason — not a "ran out of time" omission. This framing matters: a sharp interviewer will ask "why not more rules?" The answer is "I made a deliberate scope decision based on cost-per-rule evidence from Phase 3; here is the breakdown." This entry is the breakdown. A reviewer who checks the commit timestamp sees the decision was made before Phase 4 work began, not retrofitted after the fact.
Consequences for Phase 4 sequencing. The seven-day Phase 4 plan is unchanged: corpus expansion → two rule-authoring rounds → registry harness expansion → GitHub Pages dashboard → Docker demo. The scope decision reframes the exit criterion (20 rules, not 60) without altering what gets built or in what order. The 90-day stretch target (30 rules with at least 2 non-EID-1 logsources) and the long-term v2+ target (60 rules) are both logged in the README's "Known Limitations" section as successors to v1, not deferrals from it.
What we set out to do. Expand the benign corpus with three OTRF compound datasets deferred from Phase 3 — apt29_evals_day2_manual (43 MB multi-host AD lab), aptsimulator_cobaltstrike (290 KB atomic), and goldensaml_windows_events (7.5 KB ADFS compound) — then rebuild the baseline, apply the structural filter, and run precision-delta to verify all 13 rules held. GoldenSAML was flagged for inspect-then-decide with explicit rejection criteria stated before running.
Corpus expansion results.
apt29_evals_day2_manual: 43 MB downloaded. 587,286 total events; 584 EID 1 background events (1 attacker-tool excluded); 500 sampled evenly from the full 584 (corpus limit). After global dedup against existing 119 sources: 187 unique events contributed. Top images: svchost.exe (102), backgroundtaskhost.exe (48), conhost.exe (48), runtimebroker.exe (26), wmiprvse.exe (19) — classic AD workstation background pattern.
aptsimulator_cobaltstrike: 290 KB downloaded. 2,611 total events; 37 EID 1 background events (0 attacker-tool excluded by Image name). After global dedup: 22 unique events contributed. Top images: timeout.exe (7), conhost.exe (7), createnamedpipe.exe (5), taskkill.exe (5), ping.exe (4). Corpus quality note: createnamedpipe.exe (5 events) and b6a1458f396.exe (1 event) are APTSimulator simulation artifacts that belong in ATTACKER_TOOLS; neither matches any current rule detection condition, so they cannot affect precision. The ATTACKER_TOOLS list in expand_corpus_phase4.py should be extended in a future iteration.
goldensaml_windows_events: REJECT. Inspection criteria applied before running:
Reject if: top-5 Image values include ADFS-specific processes, OR unique Computer count is 1 with an ADFS-suggesting hostname, OR fewer than 5 events overlap with workstation corpus's top-20 Image list.
Inspection result: EID 1 total = 0. The entire dataset is ADFS security event log data — EID 4624 (logon, 28), EID 4662 (object access, 5), EID 412/501/33205 (ADFS protocol, 3). No process creation events exist to compare against any criterion. This satisfies the pre-stated rejection condition (0 < 5 workstation-corpus-overlap events) with no ambiguity. No directory was created (script returned no_background_events before mkdir). The GoldenSAML dataset is ADFS authentication telemetry, not workstation Sysmon data. Rejection is a correct disciplined outcome, not a failure mode — the entry here is a worked example of corpus-scope enforcement.
Baseline rebuild and the pre-existing dedup correction.
scripts/rebuild_baseline.py was run for the first time after the corpus expansion. It read 121 per-dataset events.json files (3,251 raw events), applied cross-dataset deduplication on (Image, CommandLine, ParentImage, Computer), and wrote 1,210 unique events. After the ADR-005 structural attack-event filter (39 signatures, 36 events removed), the final baseline contained 1,173 unique benign events.
The rebuild produced a number lower than the previous committed baseline (1,470 events in aaf5402). This triggered the pre-stated pause condition. Investigation:
The 1,470-event committed baseline contained 419 internal duplicate events — events with identical (Image, CommandLine, ParentImage, Computer) tuples from different datasets that were concatenated during Phase 3's baseline assembly without global cross-dataset deduplication. The old merge process applied per-dataset limits but not cross-dataset dedup. Running git show aaf5402:data/benign/workstation_baseline.json and applying the dedup key confirms only 1,051 unique events in the old baseline (419 duplicates = 28%). Every event in the old baseline is present in the new rebuild — zero events lost.
The Phase 4 data contributed genuinely new unique events: 187 from apt29_day2 + 22 from aptsimulator = 209 new unique events. Net accounting: 1,470 (old) − 419 (duplicates removed) + 209 (new unique) = 1,260, which converges to 1,173 after the structural filter removes 36 additional attack-signature matches from the expanded corpus.
This is not a script bug. rebuild_baseline.py is producing the first correctly-deduplicated baseline the project has had.
What this finding means for prior precision claims.
The Phase 3 precision measurements — mean precision 1.000 at Phase 3 Day 4 close — are not invalidated. Precision is computed as TP/(TP+FP), where FP is the count of benign events the rule fires on. Duplicate benign events inflate the reported corpus size (denominator of "corpus events tested") but do not change the FP count, because the structural filter removes attack-signature matches from the baseline before measurement. A baseline with 419 duplicates still produces the same FP count as a baseline with those duplicates removed, as long as the duplicated events were not FPs. The Phase 3 benign corpus generated zero FPs for 11 of 13 rules and the documented FPs for T1087.001 and T1059.001. Duplication of non-FP events does not affect precision. Mean precision 1.000 against the Phase 3 corpus is the same result it would have been against the deduplicated corpus.
The claim that was incorrect is the size of the corpus: "1,481 events" (Phase 3 Day 2) and "1,470 events" (Phase 3 Day 4 close) were raw event counts, not unique event counts. The correct language going forward is "1,173 unique benign events post-dedup, post-filter." Numbers in prior measurement docs (docs/measurements/) are not edited retroactively — those are historical artifacts of what was measured at the time, and rewriting them would erase the chronology the project depends on for credibility. The journey doc (this entry) and the README hold the current truth; the measurement docs hold the historical truth.
What is being corrected. The README headline number is updated from 1,481 to 1,173 unique events. The precision/recall table in the README is updated to reflect Phase 4 Day 1 measurements. Entries 3.3 and 3.7 in this document receive one-line footnotes pointing to this entry — the same supersession pattern used for the T1059.001 correction in Entry 2.1. The per-source per-dataset events.json files and the workstation_baseline.json are the new source of truth going forward; rebuild_baseline.py --check exits non-zero if the on-disk baseline drifts from what a fresh rebuild would produce (CI guard).
Phase 4 Day 1 regression triage.
forge test on the expanded corpus returned 2 regressions:
| Rule | Phase 3 Day 4 | Phase 4 initial | Category | Resolution |
|---|---|---|---|---|
| T1087.001 | 1.000 | 0.889 (1 FP) | Cat 1 — rule defect | filter_net_use added |
| T1136.001 | 1.000 | 0.750 (1 FP) | Cat 2 — attack event | Promoted to fixture |
| T1059.001 | 1.000 | 0.933 (1 FP) | Expected (threshold 0.80) | No action |
Both regressions were from apt29_evals_day2_manual — the same new dataset, as predicted.
T1087.001 — Category 1. The FP event: "net.exe" use y: https://d.docs.live.net/E260BEAE58AE0245 /user:urukhai2020@outlook.com V@m0s0rc0!2020 (parent: powershell.exe, UtcTime: 2020-05-02 08:09:58). APT29 used net use to map a OneDrive WebDAV share as C2 infrastructure (T1071.001 / T1048 territory). The rule fires because the 'user' keyword in the detection condition matches the /user: authentication flag in net use, not just net user (account enumeration). This is the same class of over-broad keyword match that caused the original T1087.001 defect in Phase 3 Day 4. Fix: filter_net_use: CommandLine|contains: ' use ' added to the detection block. Space-delimited ensures ' use ' (net use) does not match ' user ' (net user). Precision restored to 1.000.
T1136.001 — Category 2. The FP event: net1.exe user /add toby pamBeesly<3 (parent: net.exe, host: SCRANTON.dmevals.local, UtcTime: 2020-05-02 08:18:37). Real account creation from APT29 evaluation Day 2 using username pamBeesly<3 (recognizable OTRF lab test account). The existing fixture used backdoor paw0rd1 as the username from a different dataset; the structural filter had no signature matching the new username. The rule correctly fired on a real attack event not yet in the fixture. Protocol: promote to fixture. Event added to data/attack/T1136.001/events.json; structural filter re-run removed it from the baseline (1,174 → 1,173 events). TP count 3 → 4. Precision restored to 1.000.
The single post-T1136-fixture removal demonstrates the structural filter operating as a maintained invariant rather than a one-time cleanup: promoting an event to a positive fixture automatically protects identical events elsewhere in the baseline from being measured as FPs against the same rule. This is the labeling axiom from ADR-005 working cascadingly — a fixture change does not require a manual audit of the baseline to remain consistent.
T1059.001 — no action. 1 FP from the expanded corpus; precision 0.933 against threshold 0.80. This is documented expected behavior — the rule's falsepositives field explicitly describes the cross-technique chain pattern (encoded PowerShell spawned by regsvr32/mshta appearing in the T1218 background). The Phase 3 Day 4 measurement of 1.000 was a corpus-composition artifact at the smaller baseline size.
Final Phase 4 Day 1 measurements.
| Metric | Phase 3 Day 4 (1,481-event corpus†) | Phase 4 Day 1 (1,173-event corpus) | Δ |
|---|---|---|---|
| Rules | 13 | 13 | 0 |
| Mean precision | 1.000 | 0.995 | −0.005 |
| Mean recall | 1.000 | 1.000 | 0 |
| Mean F1 | 1.000 | 0.997 | −0.003 |
| Corpus (unique benign events) | ~1,051† | 1,173 | +122 |
| Tactics covered | 8 | 8 | 0 |
| SIEM conversion | 39/39 (100%) | 39/39 (100%) | 0 |
| New datasets | — | apt29_day2, aptsimulator_cobaltstrike | — |
| Rejected datasets | — | goldensaml_windows_events | — |
† The 1,481-event Phase 3 figure contained 419 duplicates; the corrected unique count is ~1,051.
The −0.005 precision delta reflects T1059.001's expected 0.933 against a larger corpus (threshold 0.80, documented). The corpus gained a net 122 unique events after dedup correction and new data addition. Both regressions were resolved via documented protocol before Day 1 closed.
What was committed. See Day 1 commit sequence: a8be290 (rebuild_baseline.py script), corpus expansion run and structural filter (in this Day 1 close commit), T1087.001 filter_net_use fix, T1136.001 Category 2 fixture promotion, README baseline size correction, journey doc supersession footnotes and this entry.
What we set out to do. Author four new detection rules (13 → 17), selected from a five-candidate gated proposal verified by direct OTRF ZIP inspection. Stop conditions: four rules landed, precision < 0.85 with no obvious fix, or four hours elapsed.
The tactic-breadth ceiling.
Before authoring, the candidate-selection process surfaced an honest constraint worth documenting as a project-level finding. OTRF atomic data does not cover six of the seven prioritized-for-expansion tactics in a way our EID 1 harness can measure: initial-access, collection, exfiltration, impact, reconnaissance, and resource-development. The honest ceiling for new techniques against current infrastructure is depth within execution and defense-evasion, not breadth across the kill chain. Tactic breadth is a Phase 5 prerequisite gated on harness expansion to additional event types.
This is the third deferral lesson in the project, joining the cross-logsource mismatch (T1110.003, T1547.001 deferred in Phase 3) and the harness modifier constraint (ADR-002). Three distinct kinds of "can't measure this honestly" documented across the project, each with a named resolution path. The earlier deferrals were surfaced by attempting and failing; this one was surfaced by pre-flight inspection. That is the protocol working as designed.
One candidate technique — T1059.006 (Python http.server) — was deferred despite passing ZIP inspection for a different reason: precision = 1.000 by corpus composition, not detection quality. The 1,173-event corpus contains no legitimate developer python -m http.server invocations, so the rule would score 1.000 by structural absence rather than by genuine discrimination. This is the same trap as T1218.005 and the T1110.003 cross-logsource issue — measuring against the wrong population. Deferred to Phase 5 pending corpus expansion to include developer-workstation activity. See data/captures/_decisions.md for the full deferral record.
Rules authored.
T1562.004 — Netsh Firewall Rule Modification. Four attack events from two datasets: cmd_netsh_fw_mod_open_ports (2 events — add rule + delete rule for an atomic testing firewall entry) and psh_python_webserver (2 events — add inbound allow rule for python.exe TCP and UDP). All 4 were in the benign baseline as Category 2 events; structural filter removed them after fixture promotion, shrinking the baseline from 1,173 to 1,169. Detection logic: Image|endswith: '\netsh.exe' + CommandLine|contains|all: ['advfirewall', 'firewall', ' rule'] + filter_readonly: CommandLine|contains: ' show '. The filter_readonly block is the operationally important design decision — 11 genuine benign netsh advfirewall events remain in the corpus, all monitor show / consec show / firewall show operations. The rule has real benign data of the same command family to discriminate against, making it the strongest precision signal in the corpus. Precision 1.000, recall 1.000.
T1218.004 — InstallUtil Signed Binary Proxy Execution. One attack event from covenant_installutil: InstallUtil.exe /logfile= /LogToConsole=false /u c:\ProgramData\GruntHTTP.dll (parent: cmd.exe). The companion cmd.exe wrapper event (cmd.exe /c InstallUtil.exe ...) stays in the benign baseline — only the direct InstallUtil.exe process is the detection target. Detection logic: Image|endswith: '\InstallUtil.exe'. All InstallUtil.exe executions in a workstation Sysmon logsource warrant detection; legitimate .NET deployment tooling that uses InstallUtil directly is rare. Completes the T1218 LOLBIN cluster alongside .001, .005, and .010. Precision 1.000, recall 1.000.
T1059.005 — VBScript Execution via WScript. One attack event from empire_launcher_vbs: wscript.exe "C:\Users\pgustavo\Desktop\launcher.vbs" (parent: explorer.exe). The corpus spot-check showed only 1 wscript.exe event in the baseline, which was this Cat 2 attack event — zero genuine benign wscript invocations. The .vbs CommandLine filter scopes detection to file-based VBScript execution rather than catching all wscript.exe invocations (which would include .js and .wsf). Single-event fixture; recall granularity caveat documented as with T1003.002 and T1105. Precision 1.000, recall 1.000.
T1218.001 — HTML Help CHM Execution via HH.exe. One attack event from psh_hh_local_html_payload: hh.exe C:\ProgramData\T1218.001.chm (parent: powershell.exe). Same high-precision-by-rarity justification as T1218.004 — in modern enterprise environments, legitimate hh.exe use is rare. Detection logic: Image|endswith: '\hh.exe'. Closes the T1218 LOLBIN cluster: .001, .004, .005, .010 now all covered. Single-event fixture. Precision 1.000, recall 1.000.
Phase 4 Day 2 final measurements.
| Metric | Phase 4 Day 1 (1,173 events) | Phase 4 Day 2 (1,166 events) | Δ |
|---|---|---|---|
| Rules | 13 | 17 | +4 |
| Mean precision | 0.995 | 0.996 | +0.001 |
| Mean recall | 1.000 | 1.000 | 0 |
| Mean F1 | 0.997 | 0.998 | +0.001 |
| Corpus (unique benign events) | 1,173 | 1,166 | −7 |
| Tactics covered | 8 | 8 | 0 |
| SIEM conversion | 39/39 (100%) | 51/51 (100%) | +12 |
| Structural filter signatures | 39 | 46 | +7 |
Mean precision increased slightly because the 4 new rules all score 1.000; adding 1.000 values raises the mean from 12/13×1.000 + 1/13×0.933 to 16/17×1.000 + 1/17×0.933. The corpus shrank by 7 events — the structural filter removed 4 Cat 2 events for T1562.004 and 1 each for T1218.004, T1059.005, and T1218.001.
What was committed. Four rule commits: a105d5f (T1562.004 + pre-work), 275f36c (T1218.004), 38a3dcc (T1059.005), c2036e6 (T1218.001). Each commit includes the rule YAML, attack fixture, updated baseline, and structural filter re-run.
What this entry is. Across the project's four phases, four techniques have been deferred with a documented reason and a named resolution path. The deferrals are not the same kind of problem. Grouping them as "things we didn't do" obscures the more important lesson: the project has three structurally distinct failure modes for honest precision measurement, and each requires a different kind of fix.
Class 1 — Logsource mismatch. The rule selects on an event type the harness doesn't evaluate, or the benign baseline doesn't contain. Precision measurement produces a structurally guaranteed 1.000 because there is no negative data of the right type for the rule to fire on. Examples: T1110.003 (requires EID 4625, none in baseline) and T1547.001 (requires EID 12/13, all baseline events are EID 1). The failure is detectable before authoring: if the baseline contains zero events of the target EventID, any rule selecting that EventID will achieve precision = 1.000 by absence. Resolution: build a parallel baseline of the missing event type from the same OTRF source corpus, extend the harness if needed, then author the rule. T1547.001 is resolved in Phase 4 Day 3 — this entry directly precedes that work.
Class 2 — Data unavailability. The rule targets a real technique but no OTRF atomic capture exists in our harness scope. Six of the seven prioritized tactics (initial-access, collection, exfiltration, impact, reconnaissance, resource-development) fall into this class — OTRF's atomic dataset coverage simply does not include EID 1 captures for these areas. The failure is also detectable before authoring: direct ZIP inspection returns zero qualifying events. Unlike Class 1, the baseline is not the problem — the attack fixture is. No local data exists to define what "TP" means for these techniques. Resolution: source new datasets from outside OTRF atomic (red-team scenarios, threat-intel captures, or synthesized from real-world incident data), or accept measurement scope as a stated limitation and expand in a later phase.
Class 3 — Corpus composition limitation. The rule logic is sound, the harness evaluates the right event type, a real OTRF fixture exists — but the benign baseline structurally lacks the discriminating activity the rule needs to be tested against. Precision appears perfect (1.000) not because the rule is good, but because the population it would false-positive on simply isn't represented in the corpus. Example: T1059.006 (python -m http.server 8000). The baseline contains no legitimate developer-workstation Python web server invocations — because the baseline was built from AD lab environments, not developer machines. The rule would score 1.000 by structural absence. Resolution: expand the baseline to include the missing population (developer workstation captures, broader endpoint telemetry), or scope the rule more narrowly (add parent-process constraints that genuinely discriminate between attacker and developer context) before measuring.
Why the distinction matters.
The three classes look identical from the outside: a rule is deferred, a measurement wasn't made. But they call for entirely different remediation:
- Class 1 is a harness infrastructure problem — solvable within the project by building the right baseline.
- Class 2 is a data sourcing problem — solvable by finding or generating the right captures.
- Class 3 is a corpus realism problem — solvable only by expanding the baseline to include the population that would expose the rule's real false-positive rate.
Conflating them leads to solutions that don't fix the actual problem: extending the harness doesn't help Class 2, and sourcing more OTRF attack data doesn't help Class 3. The classification is the prerequisite for fixing.
The pattern across phases. Class 1 was surfaced by attempting and failing in Phase 3 Day 4 (T1110.003 and T1547.001 both hit the mismatch wall and were documented). Class 2 was surfaced by pre-flight inspection in Phase 4 Day 2 before any attempt was made. Class 3 was surfaced by reasoning about corpus composition before authoring T1059.006, using the Class 1 pattern as an analogy. Each class was discovered earlier in the process than the previous one — which is what improving epistemics looks like in practice: the project now identifies structural measurement problems at inspection time, not at first-test-run time.
What we set out to do. Resolve the Class 1 deferral for T1547.001 (registry run key persistence). Three sequential pieces of work: (1) build a 484-event EID 12/13 benign baseline from OTRF sources; (2) extend the harness with per-logsource routing (ADR-006); (3) author T1547.001 against the real attack dataset and measure.
A counting error surfaced before authoring. The Phase 3 deferral record stated "32 events touching \Run registry paths" as the attack fixture size. A pre-flight survey of the actual dataset returned 3, not 32. The discrepancy traced to a loose \Run substring match used during Phase 3 inspection — it caught W32Time's \RunTime subkeys and EventLog's \RunHistory, both unrelated to T1547.001. A strict \CurrentVersion\Run\ match (trailing backslash as path-segment boundary) returns 3 genuine attack events.
This is a minor epistemic correction but the right kind: the error was caught by a dedicated survey step before the fixture was committed, not discovered post-measurement. The three-classes framework (Entry 4.5) identified this as a Class 1 problem requiring a pre-flight survey; the survey found and corrected the error before it contaminated the fixture.
The same boundary bug existed in two places. The Day 3 plan-review pass flagged the proposed Piece 1 exclusion logic — "exclude any event where TargetObject contains \Run or \RunOnce" — as a substring-boundary hazard that would incorrectly exclude legitimate paths like \Runtime and \RunAs. The fix was to tighten exclusion to the exact T1547.001 target paths with trailing backslashes (\CurrentVersion\Run\, \CurrentVersion\RunOnce\) so the match is a path segment, not a substring of a longer word. That refinement was applied to the extraction script before any code ran. What was discovered during implementation was that the exact same loose-substring bug had already corrupted the Phase 3 fixture count — the "32 events" figure was a naive \Run substring match that caught W32Time \RunTime paths just as surely as the proposed exclusion would have. The boundary-matching discipline that fixed the exclusion logic also caught the planning artifact. One correctness principle, two distinct path-matching locations, both corrected by the same scrutiny. The lesson worth recording is not "we found a counting error" but "boundary-matching rigor applied at one location surfaces the same bug at every other location where path matching happens — the discipline propagates."
The three attack events. All from empire_persistence_registry_modification_run_keys_standard_user.zip (SDWIN-190319023812):
- EID 13 SetValue —
HKU\...\CurrentVersion\Run\Updater— Empire user persistence: a PowerShell encoded command written to HKCU as an autostart payload. - EID 13 SetValue —
HKLM\SOFTWARE\...\CurrentVersion\Run\WindowsDefender— Empire HKLM persistence:%%ProgramFiles%%\Windows Defender\MSASCuiL.exewritten by MsMpEng.exe, masquerading as a legitimate Windows Defender autostart entry. - EID 12 DeleteValue — same HKLM WindowsDefender key — Empire teardown: the cleanup event that removes the run key after the simulation phase ends.
Three events, two persistence paths (user and machine), one cleanup. The fixture is small by design — it covers the full persistence operation (write + cleanup), not a multi-event campaign. Single-event-fixture caveats apply to recall granularity (documented in the rule).
ADR-006: per-logsource baseline routing. The architectural decision to keep separate baseline files per logsource category was committed as ADR-006 before any code was written. The core argument: evaluating an EID 12/13 rule against an EID 1 baseline produces precision = 1.000 by structural absence — the event type mismatch means the rule has no benign data of the right type to false-positive on. Merging baselines makes the structural-mismatch trap one misconfiguration away rather than architecturally impossible. Separate files, category-based dispatch, one table entry per new logsource — that is the invariant.
The implementation generalized ADR-005 (structural filter) for the registry logsource: EID 12/13 events use (TargetObject, Details, Image) as the identity signature instead of (Image, CommandLine, ParentImage). The filter auto-classifies attack events by EventID — no manual routing required when new fixture files land.
The registry baseline. 484 EID 12/13 events from four OTRF sources (WMI subscription, schtasks, DCOM ×2). The events are real system registry activity: lsass writing W32Time SecureTimeLimits timestamps, RuntimeBroker writing notification data, svchost updating Internet Settings ZoneMap. A less-precise rule would match hundreds of them. \CurrentVersion\Run\ is absent because the labeling axiom (ADR-005) excludes exactly the paths the rule detects — nothing more, nothing less. The 85,799 raw EID 12+13 events surveyed across those four sources (the figure that justified Path A over hunting additional datasets) are reproduced by scripts/survey_registry_events.py.
Measurement. Precision 1.000, recall 1.000, F1 1.000, against 484 benign events. The precision is 1.000 because the rule correctly limits detection to \CurrentVersion\Run\ and \CurrentVersion\RunOnce\ paths, none of which appear in the 484-event baseline. Trailing-backslash semantics confirmed: \CurrentVersion\Run\ does not match \CurrentVersion\RunTime\ (W32Time) because the next character after "Run" in the haystack is "T", not \. All three SIEM backends (Splunk, Elastic, Sentinel) emit the correct boundary — no quirks to document.
Phase 4 Day 3 final measurements.
| Metric | Phase 4 Day 2 (17 rules) | Phase 4 Day 3 (18 rules) | Δ |
|---|---|---|---|
| Rules | 17 | 18 | +1 |
| Mean precision | 0.996 | 0.996 | 0 |
| Mean recall | 1.000 | 1.000 | 0 |
| Mean F1 | 0.998 | 0.998 | 0 |
| EID 1 corpus | 1,166 events | 1,166 events | 0 |
| Registry corpus | — | 484 events | new |
| Tactics covered | 8 | 8 | 0 |
| SIEM conversion | 51/51 (100%) | 54/54 (100%) | +3 |
What was committed. 07ce8c2 (ADR-006), e5a5251 (Pieces 1-2: registry baseline + harness extension), af1afa7 (Piece 3: T1547.001 rule + fixture + lint fix + decisions correction).
The Class 1 deferral is resolved. T1547.001 was deferred in Phase 3 because the harness had no way to honestly score a registry-event rule — the EID 1 baseline provided no benign data of the right type. Phase 4 Day 3 built that infrastructure: a per-logsource dispatch table, a 484-event EID 12/13 baseline, a generalized structural filter, and a format-agnostic event loader. The rule now measures against real benign data, and the measurement is formally correct.
T1110.003 (password spraying, EID 4625) remains deferred — same Class 1 root cause, different event type. The infrastructure pattern is now established: add a baseline file, add a dispatch-table entry, repeat. T1110.003 is the obvious Phase 5 Day 1 candidate.
What we set out to do. Close the schema-guard gap opened by Day 3's optional fixtures.negative, then author the final two rules to reach the 20-rule launch target. Author T1220 (XSL Script Processing) and T1218.013 (Mavinject), in that order.
Schema guard. The Day 3 ADR-006 change made fixtures.negative optional in the rule schema — None means auto-route by logsource.category. That opened two invalid lint-passable states: a rule with no negative fixture and an unregistered category (would skip silently at test time), and a rule with no category at all (would silently default to process_creation, potentially measuring against the wrong event type). The fix: lint now requires that every rule have either an explicit fixtures.negative or a logsource.category registered in BASELINE_MAP. The BASELINE_MAP and REGISTERED_LOGSOURCE_CATEGORIES moved to forge/__init__.py as the single source of truth — REGISTERED_LOGSOURCE_CATEGORIES derives from BASELINE_MAP (frozenset of keys), so the routing map and the validation set cannot drift. Three pytest cases land alongside: two rejections (unregistered category, no category) plus one acceptance (registered category with no negative fixture — the T1547.001 valid state). The acceptance case proves the guard does not over-correct. Full-lint regression check across all 18 existing rules: pass.
This is the same class of issue as the deferrals from Phase 3: a schema change that makes an invalid state representable, surfaced before any rule could quietly land in that state. The earlier the catch, the better the project's correctness margin.
The two rules.
T1220 — XSL Script Processing via WMIC /FORMAT. Detects wmic.exe with /FORMAT: flag — the canonical XSL script processing LOLBIN pattern. The attack event uses a remote HTTPS URL pointing to a redcanaryco-hosted .xsl containing embedded JScript that WMIC parses-and-executes. One strict-matched attack event from wmic_remote_xsl_jscript. Precision 1.000, recall 1.000.
This is the day's earned-precision rule. The 1,165-event workstation baseline contains a genuine benign WMIC invocation — wmic /node:WORKSTATION6 os get — that the rule's /format: specificity correctly excludes. The rule had a chance to false-positive on a same-binary benign event and didn't. That is real discrimination, not a structural-absence artifact. The case-sensitivity question identified before authoring (rule writes /format: lowercase, attack event captures /FORMAT: uppercase) is non-issue because the harness's |contains modifier lowercases both sides before comparison — confirmed by reading forge/test_harness.py line 99 before extracting the fixture, not after.
T1218.013 — Mavinject Signed-Binary Proxy DLL Injection. Detects mavinject.exe — Microsoft App-V helper binary whose /INJECTRUNNING flag loads an arbitrary DLL into a target process by PID. Attackers abuse it as a signed-binary proxy for DLL injection. One strict-matched attack event from psh_mavinject_dll_notepad. Precision 1.000, recall 1.000.
This is the day's structural-absence rule, and the rule's _meta.yml and description say so plainly: precision = 1.000 reflects the absence of legitimate mavinject.exe execution in the 1,164-event corpus, not discrimination against benign mavinject activity. Same property as the existing T1218 cluster (T1218.001 hh.exe, T1218.004 InstallUtil, T1218.005 mshta, T1218.010 regsvr32). The LOLBIN cluster rationale — "any execution of these signed-binary-proxy tools in a workstation logsource warrants detection" — is what justifies shipping the rule despite the structurally guaranteed precision. T1218.013 completes the T1218 cluster at six sub-techniques.
Two kinds of 1.000. T1220 and T1218.013 are the same headline number with different measurement provenance, and the project is now explicit about the distinction. T1220's 1.000 is earned — there was discriminating benign data, and the rule discriminated correctly. T1218.013's 1.000 is structural — there was no discriminating benign data, and the rule scored 1.000 by absence. Both ship. The difference is named in their respective _meta.yml files and in this entry. The reason the distinction matters: if T1218.013 shipped silently as "1.000" while T1037.001 was deferred for the identical property, the project would contradict itself. Same property, same documentation; the rationale for shipping vs deferring is then a separate, explicit argument (LOLBIN cluster vs. corpus-realism uncertainty), which a reviewer can evaluate on its own merits.
Three deferrals carry into Phase 5. With T1037.001 (Logon Script) deferred for corpus-realism reasons during Day 4, the Phase 5 backlog now has three documented Class 1/3 deferrals, each with a named prerequisite:
| Deferral | Class | Prerequisite |
|---|---|---|
| T1110.003 — Password Spraying | Class 1 (logsource mismatch) | New EID 4625 baseline + aggregation logic in harness |
| T1546.003 — WMI Event Subscription | Class 1 (logsource mismatch) | New EID 19/20/21 baseline + harness support |
| T1037.001 — Logon Script (UserInitMprLogonScript) | Class 3 (corpus realism) | Registry baseline expansion to include \Environment\ activity |
The distinction worth naming: T1037.001 is the only one that's purely a corpus problem. The harness already handles registry_event — what's missing is \Environment\ activity in the benign pool. T1110.003 and T1546.003 require both new baselines and new harness support (new EID handling in _load_events, new dispatch table entries, plus aggregation for T1110.003). T1037.001 is therefore the easiest of the three to unblock in Phase 5 — useful sequencing information when planning Phase 5 Day 1.
Uninvestigated candidate categories (banked for Phase 5). Day 4 did not survey: discovery techniques, additional lateral_movement DCOM variants, or the auditpol_system_user_auditpolicy_modification dataset. These are EID 1 or registry-event candidates that may be viable without harness expansion. Phase 5 should survey these first before the harder Class 1 deferrals — they are the "lower-friction next rules" if the goal is to expand beyond 20.
Phase 4 Day 4 final measurements.
| Metric | Phase 4 Day 3 (18 rules) | Phase 4 Day 4 (20 rules) | Δ |
|---|---|---|---|
| Rules | 18 | 20 | +2 |
| Mean precision | 0.996 | 0.997 | +0.001 |
| Mean recall | 1.000 | 1.000 | 0 |
| Mean F1 | 0.998 | 0.998 | 0 |
| EID 1 corpus | 1,166 | 1,164 | −2 (two attack-shape leaks removed) |
| Registry corpus | 484 | 484 | 0 |
| Tactics covered | 8 | 8 | 0 |
| SIEM conversion | 54/54 (100%) | 60/60 (100%) | +6 |
| Structural filter signatures | 46 | 48 | +2 |
| Earned vs structural-absence 1.000 | not distinguished | distinguished | first per-rule classification |
What was committed. 5b56cc7 (schema guard + tests), 68e5a19 (Entry 4.6 sharpening), 2f08ad0 (T1220), b0d11a1 (T1218.013), plus this entry, the dashboard scope doc, and any close updates.
The headline isn't 20. The headline of Day 4 is that the project hit 20 rules, and the last two candidates were handled with two different measurement provenances that were named explicitly in their respective rule files — a defer (T1037.001), a ship-with-caveat (T1218.013), and a ship-with-earned-precision (T1220), all decided on the same day, all consistently. The measurement discipline distinguishes earned precision from structural-absence precision per-rule, not just in aggregate. That distinction is what makes the 20-rule milestone credible rather than just a count.
Day 5 builds the public dashboard. The scope is documented in docs/measurements/phase4-day5-dashboard-scope.md with the real JSON shapes of reports/results.json, reports/conversion_matrix.json, and reports/layer.json — no aspirational schemas, no Day 5 rework expected.
What we set out to do. Build the v1 dashboard per the Day 4 scope doc and the Day 5 design specification — a single static index.html with Alpine.js + Tailwind, five tabs (Overview, Rules, Coverage, Trends, Gaps), dark layered surfaces, JetBrains Mono for numbers, cyan/amber encoding of earned vs structural-absence precision. The intent: produce a public-facing surface that communicates in 10 seconds what was built, how well it works, against what data, and how it's portable across SIEMs.
What was built. forge/templates/index.html with the full five-tab layout. forge/build.py implementing the data-artifact pipeline (read three reports → derive dashboard_meta.json → copy everything to dist/site/). forge/cli.py wired forge build and forge run to the new pipeline. scripts/screenshot_dashboard.py captured all five tabs via Playwright at 1440×1300 viewport. The build infrastructure worked: pipeline ran clean, 60/60 conversion confirmed in the dashboard, multi-tactic placement worked on the Coverage tab after the Day 5 mid-build fix (the tactic_by_tech last-wins bug surfaced and was corrected by using rule tags directly).
Design system implemented. Base #0a0e14 with three elevation tiers (#121620 panel, #1a1f2e card). Single accent cyan #22d3ee for active states and the focal Mean Precision card. Amber #f59e0b for structural-absence rules. JetBrains Mono for all numbers and technique IDs; Inter for prose. Hairline borders. Faint 24px dot-grid background texture. Precision distribution strip rendered all 20 rules as fixed-height tracks with fill height = (precision - 0.80) / 0.20 * 100% and fill color by classification — T1059.001's 0.933 produced the only visibly shorter bar against 19 full-height ones. Count-up animation on stat cards (420ms cubic ease-out). Bar stagger 20ms per index.
Schema-guard validation in build. The hardcoded _CLASSIFICATION dict in build.py was protected by a build-time check that the dict's keys equalled the actual rule set — drift fails the build loudly rather than silently. BASELINE_MAP and REGISTERED_LOGSOURCE_CATEGORIES were already colocated in forge/__init__.py from Day 4; the build inherited them. rule_path_to_stub() was the single named function for the file-stem → conversion-matrix key join. Multi-tactic placement: each rule contributed to every tactic column it tagged (T1053.005 appeared in Execution + Persistence + Privilege Escalation; T1218.013 in Defense Evasion; T1548.002 in Defense Evasion + Privilege Escalation).
Why this entry exists despite no commit. The dashboard was never merged to main. It was reviewed against the quality bar (Entry 4.9 below), did not meet it, and was removed. But the work happened — files existed locally, screenshots existed, the build pipeline ran — and recording it honestly is more useful than pretending Day 5 was idle. The data layer that survives (dashboard_meta.json, the build-time validation, the JSON contract) was designed during this work and is intact.
Outcome. UI layer removed; data layer retained. See Entry 4.9 for the decision and its rationale.
What this entry is. A decision record. The Phase 4 Day 5 dashboard build (Entry 4.8) produced a working UI but did not meet the project's quality bar. Rather than ship a substandard frontend, the UI layer was removed before merging to main and the presentation layer was deferred to a dedicated design effort. The data layer — forge build's JSON artifact generation, the verified JSON schemas, dashboard_meta.json — was retained intact, so the future frontend has a clean, documented data contract to build against.
Why this is consistent with the project's character. The same standard the project applies to detection rules — distinguish earned precision from structural-absence precision, defer T1037.001 rather than ship a 1.000 figure that means nothing, document T1218.013's structural-absence caveat rather than pretend the rule's 1.000 was discriminated — applies to the dashboard. Shipping a UI that didn't read as the "precision security console" the design called for would have contradicted the measurement-honesty standard expressed in every other surface of the project. The defer-rather-than-ship-substandard decision is the same shape as the three Phase 5 deferrals in reports/layer.json and in the README. A reviewer who reads the project as a whole sees the consistency. A reviewer who only sees a passable-but-not-great dashboard would lose the signal that the rest of the project is built on.
What survives. forge build remains Stage 5 of the pipeline and is still required by forge run. Its output surface shrank from "HTML + JSON" to "JSON only" — four artifacts in dist/data/: dashboard_meta.json (the project-specific data contract), results.json, conversion_matrix.json, layer.json. The build-time classification-dict drift check, rule_path_to_stub(), the tactic-join fallback with console warning, the multi-tactic extraction from rule tags, the BASELINE_MAP source-of-truth pattern — every Day 4–Day 5 piece of validation logic stays. The presentation layer is the only thing removed.
ATT&CK coverage in v1 without a UI. reports/layer.json is a valid ATT&CK Navigator v4.5 layer file. Anyone can run forge run, then load the layer into https://mitre-attack.github.io/attack-navigator/ via Open Existing Layer → Upload from Local. v1 ships with this workflow. A bespoke in-page heatmap is a post-v1 concern.
What was committed. 487eaf7 (UI removal commit) plus this entry.
Phase 5 implication. The dashboard is added to the Phase 5 backlog as a fourth deferred item, separate from the three rule-measurement deferrals (T1037.001, T1546.003, T1110.003). It is the only deferral that is a design problem rather than a measurement-infrastructure problem. Sequencing: the rule-measurement deferrals are tackled with the same kind of work the project has been doing for four phases (corpus expansion, harness extension), while the dashboard requires a different kind of effort (visual design, possibly external collaboration). The Phase 5 plan should not bundle them.
What Phase 4 set out to do. Take the project from "Phase 3 close — 8 rules, single logsource, no public surface" to "v1 launch-ready — 20 rules across 8 tactics, multi-logsource harness, complete CI/CD and Docker demo, all measurement provenance explicit." Five days of work, gated phase-by-phase, with explicit pause points before each rule-authoring batch.
What shipped, day by day.
Day 1. Corpus rebuild and dedup audit. The pre-Phase-4 "1,481 events" headline turned out to be a raw concatenation count that included 419 cross-dataset duplicate events. The corrected unique count was ~1,051. Path forward: corpus expansion to apt29_evals_day2_manual + aptsimulator_cobaltstrike, regression triage on the existing rules. The corpus settled at 1,173 events; T1087.001 and T1136.001 received documented Category 1 / Category 2 fixes (filter_net_use block; pamBeesly fixture promotion). The deduplication discipline was strengthened — scripts/rebuild_baseline.py --check for CI parity.
Day 2. Four new rules to reach 17 (T1562.004 netsh firewall, T1218.004 InstallUtil, T1059.005 wscript, T1218.001 hh.exe). The candidate proposal applied direct OTRF ZIP inspection up-front and surfaced the tactic-breadth ceiling: six of seven prioritized tactics had no OTRF EID 1 coverage usable by the existing harness. Documented as Class 2 deferral (data unavailability). T1059.006 (Python http.server) was investigated and deferred as Class 3 (corpus-realism) before authoring — the AD-lab baseline had no legitimate developer Python web server activity, so precision would have been guaranteed by composition.
Day 3. T1547.001 (Run Key Persistence) graduated — the Class 1 deferral from Phase 3 Day 4 closed. Required new infrastructure: a 484-event EID 12/13 registry baseline from 4 OTRF sources, an ADR-006 per-logsource dispatch table, a generalized structural filter that auto-classifies attack events by EventID, a format-agnostic event loader. A counting error surfaced and was corrected mid-build: the "32 events" Phase 3 figure was a loose \Run substring match catching W32Time \RunTime paths; strict-match returned 3 events. The same boundary-matching discipline applied to both the fixture extraction and the exclusion logic. T1547.001 measured at p=1.000, r=1.000 against 484 benign events — earned, not structural.
Day 4. Two more rules to reach 20 (T1220 XSL Script Processing, T1218.013 Mavinject). Schema guard added in lint to close the gap opened by Day 3's optional fixtures.negative: a rule without an explicit negative fixture must have a registered logsource.category. Three pytest cases (two rejections + one acceptance). T1220 became the project's flagship earned-precision example: the benign baseline included a real wmic /node:WORKSTATION6 os get event, and the rule's /format: specificity correctly excluded it. T1218.013 shipped as a structural-absence rule with explicit caveats in both the rule description and _meta.yml, consistent with the T1218 LOLBIN cluster rationale. T1037.001 surveyed and deferred as Class 3 — registry baseline had zero \Environment\ activity. First per-rule earned-vs-structural-absence classification audit (14 earned / 6 structural-absence). Dashboard scope doc written against the real JSON shapes of reports/results.json, reports/conversion_matrix.json, reports/layer.json.
Day 5. Static dashboard built and removed. A first-cut Alpine + Tailwind dashboard was implemented per spec (five tabs, dark precision-console design system, earned/structural-absence cyan/amber encoding, multi-tactic Coverage placement, Playwright screenshots). On review, the generated UI did not meet the project's quality bar. The UI layer was removed before merging; the data layer (forge build, dashboard_meta.json generation, classification dict drift validation, rule_path_to_stub(), tactic-join fallback, multi-tactic extraction) was retained intact. Entries 4.8 and 4.9 record the build and the removal honestly.
Plan vs. actual.
| Plan (Phase 4 Day 0) | Actual | Why |
|---|---|---|
| Rule count target | Achieved (20) | Day 2 + Day 3 + Day 4 sequencing held. |
| Multi-SIEM conversion ≥ 95% | 100% (60/60) | No backend quirks surfaced on either logsource. |
| ≥1 new logsource | Achieved (registry_event) | Class 1 deferral mechanism worked — pre-flight survey caught issues before authoring. |
| ATT&CK Navigator layer | Achieved (reports/layer.json) |
Loaded via official Navigator; no in-house viewer needed for v1. |
| Public dashboard | Deferred | Day 5 build did not meet quality bar; presentation layer reframed as post-v1 design effort. |
| 60-rule target (original PRD §8) | Reframed to 20 | Day 2 ceiling diagnostic. Past 20 rules across 8 tactics, additional count produces diminishing returns on the measurement-discipline claim. |
Three Phase 5 deferrals + one design deferral.
The Phase 5 backlog has four named items, each with a different unblocking path:
- T1037.001 (Logon Script) — Class 3, corpus-only. Easiest to unblock.
- T1546.003 (WMI Event Subscription) — Class 1, harness + corpus. Requires EID 19/20/21 baseline + new dispatch entry.
- T1110.003 (Password Spraying) — Class 1, harness + corpus + aggregation. Requires EID 4625 baseline + dispatch entry + count-per-source windowing in the harness (the only deferral that touches scoring logic, not just routing).
- Dashboard UI — design deferral. Different kind of effort from the other three. Should not be bundled with rule-measurement work.
What was learned.
Pre-flight inspection saves more time than it costs. The Class 1 / 2 / 3 framework (Entry 4.5) was articulated in Phase 4 Day 2 and immediately applied to T1059.006 (Class 3, deferred before authoring) and T1547.001 (Class 1, full prerequisite walked before fixture extraction). The "32 → 3" counting correction was caught by the same inspection discipline applied to the fixture as to the exclusion logic. Three pre-flight catches in a single phase — the project now identifies structural measurement problems at inspection time, not at first-test-run time.
Boundary-matching discipline propagates. The trailing-backslash care applied to \CurrentVersion\Run\ in one location surfaced the same loose-substring bug in another. The lesson is not "always use strict matching" — it is "when you apply a correctness principle to one location, audit every other location where the same shape of problem can arise." This is a project-level habit now, not a phase-specific instruction.
Honest deferral is a feature, not a bug. The project documents three measurement deferrals and one design deferral, each with a named prerequisite and a structural class. A reviewer reading the project sees not "things we didn't do" but "structurally distinct failure modes with different fixes." That framing is what makes the 20-rule milestone credible — every rule the project shipped, it shipped honestly, and every rule it deferred, it deferred with a documented reason a reader can evaluate.
Two kinds of 1.000. Phase 4 Day 4 was the first day the project distinguished earned-precision 1.000 from structural-absence-precision 1.000 in every artifact: rule descriptions, _meta.yml, the per-rule classification audit, the dashboard data contract, the README. The same number, two provenances, named explicitly. T1218.013 ships under the cluster rationale; T1037.001 defers under the same property; the difference is the argument, not the measurement.
Build and remove is allowed. The Phase 4 Day 5 dashboard was built and removed in the same phase. The work is recorded honestly (Entries 4.8 and 4.9). The alternative — shipping a passable-but-not-great UI — would have contradicted the standard the project applies to every other surface. The removal isn't a setback; it's the standard applied consistently.
Phase 4 close measurements.
| Metric | Phase 3 close | Phase 4 close | Δ |
|---|---|---|---|
| Rules | 8 | 20 | +12 |
| Mean precision | 0.975 | 0.997 | +0.022 |
| Mean recall | 1.000 | 1.000 | 0 |
| Mean F1 | 0.986 | 0.998 | +0.012 |
| Logsources | 1 (EID 1) | 2 (EID 1 + EID 12/13) | +1 |
| Process corpus | 1,481 (later corrected to ~1,051) | 1,164 (post-cleanup) | net +113 unique events |
| Registry corpus | — | 484 | new |
| Tactics covered | 7 | 8 | +1 |
| SIEM conversion | 24/24 (100%) | 60/60 (100%) | +36 |
| Structural filter sigs | ~30 | 48 | +18 |
| ADRs | 5 | 6 (ADR-006) | +1 |
| Documented deferrals | 2 | 4 (3 measurement + 1 design) | +2 |
| Per-rule classification | not tracked | 14 earned / 6 structural-absence | new dimension |
| Public artifacts | reports/ only | reports/ + dist/data/ (no UI) | data layer added |
| CI/CD | none | .github/workflows/ci.yml |
new |
| Docker demo | placeholder | full-pipeline image, exits with summary | new |
What v1 ships. A CLI-driven detection-as-code pipeline (forge run) that lints, tests, converts, scores, and builds a documented data contract. 20 rules with explicit per-rule measurement provenance. Multi-SIEM portability across SPL, EQL, and KQL. Multi-logsource harness with category-based routing. Phase 5 backlog documented in this journey, in the README, and in dist/data/dashboard_meta.json. CI runs the full pipeline on every push and PR; Docker image runs it end-to-end with a single command.
The 20-rule milestone is the launch target. The honest framing of how it was reached — including the deferrals, the build-and-remove decisions, the counting corrections — is what makes the milestone credible rather than nominal.