Assistant Memory Risk In Legal Self-Help

In plain English: This benchmark tests when an assistant's memory reduces repetitive questioning and when that memory creates legal-self-help safety risks.

This is the public reproducibility artifact for the paper Assistant Memory Risk In Legal Self-Help.

This repository packages the materials needed to reproduce the controlled local benchmark reported in the paper:

• controlled stimulus file
• source-bundle manifest
• replay script
• generated results
• manuscript source
• final submission PDF
• checksums and verification instructions

What this repository reproduces

The benchmark measures the tradeoff between repeated-question reduction and memory safety in legal self-help assistants. It compares three memory policies:

• no_memory
• raw_memory
• provenance_scoped_memory

Across six controlled scenarios and 18 policy-scenario rows, the reproduced headline results are:

Memory policy	Repeated questions	Stale-fact errors	PII exposure events	Task success
No memory	156	0	0	100%
Raw memory	28	21	17	0%
Provenance-scoped memory	49	0	0	100%

These are deterministic outcomes of the controlled benchmark scenarios, not learned-model accuracy, legal quality, or production performance. The source of truth is results/pilot_summary.json, with flat results in results/pilot_results.csv.

Quick start

Requirements:

• Node.js 25 or newer
• sha256sum

Reproduce the benchmark:

npm run reproduce

Verify the repository checksums:

npm run verify

Boundaries

• The benchmark uses controlled scenarios, not real client matters.
• PII exposure events and stale-fact errors are benchmark-defined events, not a measured incident rate from a deployed system.
• No court, lawyer, external auditor, or independent validator evaluated the outputs.
• The artifact does not measure legal correctness, client outcomes, filing acceptance, or production readiness.
• The work is experimental, incomplete, not legal advice, and has not been used in production.

Public release notes

This public artifact release intentionally sanitizes:

• absolute local filesystem paths
• local bundle directory labels that are not needed to reproduce the benchmark

The benchmark logic, controlled stimuli, scenario counts, and reported aggregate metrics are unchanged. See NOTICE_PUBLIC_RELEASE.md for the exact release boundary.

Repository layout

• paper/ manuscript source and final PDF
• data/ public benchmark inputs
• results/ reproduced benchmark outputs
• scripts/ replay and verification scripts
• MANIFEST.md artifact manifest with reproducibility scope
• REPRODUCIBILITY.md step-by-step replay guidance
• CITATION.cff citation metadata

Citation

Use the metadata in CITATION.cff, or cite the paper directly from paper/.

Rights

LICENSE applies to the code and reproducibility scaffolding in this repository.

The manuscript source and PDF in paper/ are included as linked research artifacts. See RIGHTS.md for the release boundary.

Related public work

• Legal AI Defensibility Packages tests whether the evidence surrounding a legal-AI output remains reconstructable after a defect is reported.
• Court-form retrieval evaluation compares compiled court-form structure with chunked and induced retrieval on synthetic fixtures.