Skip to content

Anchor prepared patterns on the rarest literal run#8

Open
killerra wants to merge 2 commits into
scanner-nitsfrom
anchor-frequency
Open

Anchor prepared patterns on the rarest literal run#8
killerra wants to merge 2 commits into
scanner-nitsfrom
anchor-frequency

Conversation

@killerra

@killerra killerra commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Anchor-selection follow-up, stacked on #7. The static heuristic prefers the longest literal window, ranked by a fixed byte-rarity table — it cannot know how common a run actually is in the binary being scanned. ? ? 48 8b 0d ? ? ? ? 15 7c anchors on 48 8b 0d (639 occurrences in the PE fixture) even though 15 7c occurs only 32 times, so the executor verifies ~20× more candidates than necessary.

Changes

bench(scanner): cover the prepared-pattern scan path

  • Adds matches_prepared benches next to the existing matches_code ones, plus a prepare_pattern group timing preparation itself. Committed first so the criterion baseline covers the new bench points on unchanged behaviour.

perf(scanner): anchor prepared patterns on the rarest literal run

  • Scanner::prepare_pattern / prepare_pattern_str now extract one candidate window per maximal literal run of the pattern and count each candidate's actual occurrences in the view's code with memchr::memmem (early-exit once a candidate exceeds the current best). The prepared pattern anchors on the run with the fewest occurrences; ties prefer the longer window, then the static score.
  • Exact counts instead of a frequency model: a byte-histogram approach was measured first and picked the same anchor as the static heuristic — an independence model can't see that 48 8b 0d is a common instruction encoding rather than three coincidental bytes.
  • The anchor stays a pure necessary-condition filter (execution verifies every candidate), so results are identical; only scan speed changes. The unprepared paths (matches_code, finds_code) and the view-less PreparedPattern::from_atoms are untouched.
  • Counting only runs when the pattern has at least two literal runs; single-run patterns skip it entirely.

Benchmarks (scan_anchor, criterion A/B against the bench-only commit)

PE64 fixture:

bench before after Δ
matches_prepared/leading_wildcards 1.010 ms 54.9 µs −94.5% (~18×)
matches_prepared/late_rare_run 571 µs 537 µs −6.2%
matches_prepared/weak_lead_two_runs 50.4 µs 47.9 µs −4.0%
matches_prepared/strong_lead_control 1.451 ms 1.433 ms −1.9%
matches_code/* (all four) flat
prepare_pattern (≥2 literal runs) ~200 ns 95–526 µs one-off counting cost
prepare_pattern (single run) ~160 ns ~240 ns gated, no counting

ELF64 fixture: all scan benches flat (±3%). On that fixture no candidate run is materially rarer than the static pick, so counting confirms the static choice — the refinement pays off exactly where the binary makes a difference and never costs anything at scan time.

Verification

  • New unit tests: crafted-view re-anchoring on the rare run, single-run gate keeps the static anchor, prepared-vs-unprepared match equivalence on a view where the two paths anchor differently, and a PE-fixture pin (? ? 48 8b 0d ? ? ? ? 15 7c → anchors on 15 7c).
  • cargo test --workspace --all-targets --all-features + cargo test --workspace --doc green.
  • cargo clippy --workspace --all-targets --all-features -- -D warnings -D clippy::all clean; cargo fmt --all -- --check clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant