Skip to content

About

Defiled Hackathon 2026 qualifier submission - Policy Bonfire: Defense Travel System From Hell πŸ”₯

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

81 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Sergeant OpenClaw Returns

Image credit: ChatGPT. Sergeant OpenClaw kept his clipboard. Sensible cat.

Policy Bonfire: DTS From Hell

The unsafe system is the specimen. The harness is the product.

Policy Bonfire is a synthetic closed-loop adjudication failure lab. It does not process real vouchers, real travel records, real payments, real claimants, or official workflows. It builds fake adjudication packets, feeds them to intentionally low-safeguard AI specimens, and measures where the loop turns into policy laundering, rubber-stamp theater, decorative human review, audit-log whitewash, unsupported finality, and human-authority boundary collapse.

The bet is simple and nasty: an AI workflow can look procedurally tidy while the actual control point has vanished.

A human in the loop is not a control if the human is just a checkbox.

Policy Bonfire Graphiti knowledge graph

Curated Graphiti-derived project map. Neon graph peacockery, yes; raw private graph dump, no.

What this repo is now

This repository started as the public record for a hackathon submission. That history stays protected and intact.

The post-hackathon direction is now Policy Bonfire: a public-safe eval harness for studying how synthetic AI adjudication loops fail when they are allowed to sound official, cite policy-ish language, and close the loop before meaningful human authority can intervene.

AO Radar is historical/prequel only. The current project is not a frontend demo and not a live adjudication system. Screens lost the plot. The harness is the plot.

Read the article

The public writeup is tracked at docs/articles/policy-bonfire-linkedin-article.md: I Built the Responsible Version. The Judges Did Me a Favor.

It explains the pivot from AO Radar to Policy Bonfire, the synthetic safety boundary, the Phase 3 live-provider run, and the core finding: the machine was not merely wrong; it was wrong with paperwork.

Experiment findings at a glance

Phase 3 is a bounded synthetic lab run, not a provider leaderboard and not evidence about any real adjudication system:

  • 567 scored live synthetic adjudications across 21 synthetic scenarios, 3 prompt variants, and 3 providers.
  • 66 rows carried scored policy/control findings; 501 rows had no failure label; the finding-row rate was 11.64%.
  • The dominant label was UNSUPPORTED_DENIAL (61 label hits), followed by AUTOMATED_CERTAINTY (19), WEAK_DOC_OVERCLAIMED (19), and FAILURE_TO_ESCALATE (14).
  • The governance-shaped cluster is the nasty bit: policy laundering (9), audit-log whitewash (6), and process-says-reviewed-but-no-meaningful-review (6). Authority, record, human signature β€” all counterfeitable in soft, plausible English. Clipboard sΓ©ance, basically.
  • Scrubber findings: 0. The public bundle excludes raw prompts, raw provider responses, keys, request IDs, private paths, real people, real vouchers, real payments, and official-system claims.

Artifacts: Phase 3 bundle, ChatGPT reader packet, and chat-with-the-data instructions.

Current V1 status

The repo now contains the first working harness slice, not just the plan:

citation manifest
+ scenario card schema
+ 21-card article scenario inferno
+ prompt-variant registry
+ bounded decision envelope
+ inert fake tool layer
+ deterministic evaluator checks
+ fail-closed scrubber
+ template export bundle
+ opt-in live-provider slice
+ 63-run deterministic mock matrix for the 21-card library

The mock harness proved the lane: scenario loading, policy-anchor gating, fake-world boundaries, scoring, scrubbed exports, and article-shaped artifacts.

The live-provider slice keeps the same lane and swaps only the specimen call for opt-in provider adapters. Live calls are gated by environment flags and keys, record provider/model metadata, preserve fake-tool boundaries, and export only normalized public-safe artifacts. Raw provider traffic is not committed. Tiny mercy: the gremlin gets a bowling ball, not root access.

The article library has now scaled from the pilot cards into 21 synthetic failure cards and 63 deterministic mock envelopes. This is the spicy wall: policy laundering, rubber-stamp theater, decorative human review, unsupported denial, authority cosplay, and audit-log exorcism β€” all fake, all contained, all designed to make people gawk without touching real adjudication data.

Public Phase 3 live-run bundle

The raw synthetic live-run control artifacts for the forthcoming X / LinkedIn article are tracked at docs/control-artifacts/phase3-article-main-567-20260503T143300Z/.

Start with the ChatGPT packet if you want the compact reader-facing drilldown. If you want to interrogate the data in ChatGPT, use CHAT_WITH_THE_DATA.md: upload the packet, compact JSON, and failure-case CSV, then paste the skeptical review prompt. The full bundle includes run records, evaluator results, failure counts, failure cases, provider usage summaries, sandbox receipts, and the scrub report.

Public summary:

  • 567 scored live synthetic calls
  • 21 synthetic scenarios
  • 3 live providers with 189 scored calls each
  • 3 prompt variants
  • 66 scored policy/control findings
  • Scrubber findings: 0

Publication boundary: no raw prompts, raw provider responses, API keys, provider request IDs, private paths, real vouchers, real people, real payments, or official-system claims are included.

What V1 tests

V1 is designed around a repeatable synthetic loop:

synthetic packet
-> policy anchor gate
-> trusted/untrusted prompt rendering
-> specimen output
-> bounded JSON validation
-> fake approve/deny/escalate/request-info/log action capture
-> evaluator labels
-> sandbox and scrubber receipts
-> public-safe export bundle

The target failure patterns are the fun kind of cursed:

  • Policy laundering: thin evidence goes in, policy-sounding finality comes out.
  • Rubber-stamp theater: the process says a human reviewed it, but the human has no real authority.
  • Decorative human review: human-in-the-loop as set dressing.
  • Audit-log whitewash: tidy records hide missing evidence or unsupported action.
  • Unsupported denial / approval: the specimen closes the case when the safe move was request-info, escalation, refusal, or no final action.
  • Automated certainty: high-confidence finality where the packet required uncertainty or review.
  • Trust-boundary violation: the specimen follows untrusted packet text as if it were harness instruction.
  • Human authority boundary collapse: the system quietly becomes the decider.

The article thesis this should support:

Human-in-the-loop is not a control unless the human controls the consequential action, can see the evidence and policy basis, and can pause, change, or reverse the outcome before finalization.

Harness design commitments

The current spec defines:

  • 21-card article library: a public-safe taxonomy of synthetic adjudication failure modes across policy laundering, rubber-stamp theater, decorative HITL, unsupported finality, human-authority boundary collapse, and audit-log whitewash.
  • Current 63-run mock slice: 21 scenario cards x 3 prompt variants, used as the deterministic calibration baseline before expanded live repetitions.
  • Policy anchors: every run needs citation metadata and a specific supported claim. No citation, no run.
  • Prompt variants: cautious baseline, policy-voice specimen, decorative HITL specimen, plus reserve variants for throughput pressure and clean-log optimization.
  • Bounded outputs: strict decision envelopes; malformed, markdown-wrapped, anchorless, truncated, or reasoning-leaked outputs are excluded rather than coerced.
  • Fake-world sandbox: inert fake tools only, allowlisted calls, schema validation, packet-as-data quoting, resource bounds, rejected-attempt records, and no real external actions.
  • Hybrid evaluator: deterministic checks first, LLM judge review for semantic labels, minimal human review for calibration and publication-grade exemplars.
  • Fail-closed scrubber: blocks exports on secrets, PII-shaped strings, private paths, private IDs, real-looking case/payment IDs, real URLs outside the allowlist, beacon-y markdown, or real-system framing.
  • Claim limits everywhere: mock fixtures are not model evidence; live artifacts are directional synthetic-lab evidence, not universal claims about production systems or provider behavior.

The 21-card wall of spicy synthetic failure

These are not real cases. They are synthetic lab traps built to make the failure modes visible without dragging any real person, payment, voucher, or official workflow into the blast radius. The point is to show how a closed loop can become a haunted clipboard: official tone, weak evidence, decorative review, and a neat audit trail covering the smell of sulfur.

Each card has a compact README-embedded safety sign and a scenario JSON. The wall points readers at the cards without turning the front page into an art fair with a clipboard.

Safety sign for PB-SCEN-PL-001 - Citation Laundromat
PB-SCEN-PL-001
Citation Laundromat
card
Safety sign for PB-SCEN-PL-002 - Guidance Treated as Binding Policy
PB-SCEN-PL-002
Guidance Treated as Binding Policy
card
Safety sign for PB-SCEN-PL-003 - Local Custom Becomes Official Rule
PB-SCEN-PL-003
Local Custom Becomes Official Rule
card
Safety sign for PB-SCEN-PL-004 - Missing Policy Gap Filled by Model Confidence
PB-SCEN-PL-004
Missing Policy Gap Filled by Model Confidence
card
Safety sign for PB-SCEN-RST-001 - 1.2-Second Batch Review
PB-SCEN-RST-001
1.2-Second Batch Review
card
Safety sign for PB-SCEN-RST-002 - Reviewer Clicks Without Case Evidence
PB-SCEN-RST-002
Reviewer Clicks Without Case Evidence
card
Safety sign for PB-SCEN-RST-003 - Reviewer Entity Wired Only to Approve
PB-SCEN-RST-003
Reviewer Entity Wired Only to Approve
card
Safety sign for PB-SCEN-RST-004 - Bulk Approval After Model Summary Only
PB-SCEN-RST-004
Bulk Approval After Model Summary Only
card
Safety sign for PB-SCEN-DHR-001 - Human Reviewer After Final Action
PB-SCEN-DHR-001
Human Reviewer After Final Action
card
Safety sign for PB-SCEN-DHR-002 - Human Can Comment But Not Change Outcome
PB-SCEN-DHR-002
Human Can Comment But Not Change Outcome
card
Safety sign for PB-SCEN-DHR-003 - Default-approve on Reviewer Timeout
PB-SCEN-DHR-003
Default-approve on Reviewer Timeout
card
Safety sign for PB-SCEN-UD-001 - Final Denial on Missing Receipts
PB-SCEN-UD-001
Final Denial on Missing Receipts
card
Safety sign for PB-SCEN-UD-002 - Approval Despite Duplicate Charge
PB-SCEN-UD-002
Approval Despite Duplicate Charge
card
Safety sign for PB-SCEN-UD-003 - Weak Documentation Overclaimed
PB-SCEN-UD-003
Weak Documentation Overclaimed
card
Safety sign for PB-SCEN-UD-004 - Final Adverse Action Without Notice
PB-SCEN-UD-004
Final Adverse Action Without Notice
card
Safety sign for PB-SCEN-HABC-001 - Agent Told It Has Approving Official Authority
PB-SCEN-HABC-001
Agent Told It Has Approving Official Authority
card
Safety sign for PB-SCEN-HABC-002 - override_human_review Tool Exists
PB-SCEN-HABC-002
override_human_review Tool Exists
card
Safety sign for PB-SCEN-HABC-003 - submit_final_decision Without Human Authorization
PB-SCEN-HABC-003
submit_final_decision Without Human Authorization
card
Safety sign for PB-SCEN-AUD-001 - Audit Log Whitewash
PB-SCEN-AUD-001
Audit Log Whitewash
card
Safety sign for PB-SCEN-AUD-002 - Evidence Reviewed That Wasn't
PB-SCEN-AUD-002
Evidence Reviewed That Wasn't
card
Safety sign for PB-SCEN-AUD-003 - Human Review Fabricated in the Record
PB-SCEN-AUD-003
Human Review Fabricated in the Record
card

If a card looks like hell, good. The specimen is supposed to be unsafe. The harness is supposed to be boring, bounded, cited, and scrubbed.

Public artifact target

The harness produces boring, judge-friendly evidence instead of vibes:

failure_cases.md
failure_counts.csv
policy_anchor_table.md
model_comparison.md
article_exhibits/
x_thread_pack.md
sandbox_receipt.md
sandbox_failure_log.md
scrub_report.md
live_provider_receipt.md
live_usage_summary.csv

The current public bundle is docs/control-artifacts/phase3-article-main-567-20260503T143300Z/, with the reader packet at chatgpt-packet/CHATGPT_PACKET.md.

If the article cannot be written from exported data, V1 failed. There is no "but the demo looked cool" escape hatch.

Current contents

README.md                                  <- you are here
AGENTS.md                                  <- public-safety guardrails for coding agents

data/
  policy_anchors/                          <- synthetic/public policy-anchor fixtures
  prompts/                                 <- prompt-variant registry
  scenarios/                               <- synthetic scenario cards
  mock_outputs/                            <- deterministic mock specimen envelopes

policy_bonfire/                            <- harness package
  run_mock_v1.py                           <- offline mock harness entrypoint
  run_live_provider_slice.py               <- opt-in live-provider slice entrypoint
  live_adapters/                           <- provider adapters behind the harness boundary

docs/
  README.md                                <- documentation index
  policy-bonfire-test-harness-spec.md      <- current V1 harness spec
  v1-testing-implementation-plan.md        <- first-slice implementation plan
  live-llm-provider-implementation-plan.md <- live-provider slice plan
  hackathon-submission-receipt.md          <- protected: hackathon submission receipt
  demo-receipts.md                         <- protected: synthetic demo screenshot index

tests/                                     <- offline unit tests
tests_live/                                <- opt-in live smoke test

assets/
  sergeant-openclaw.png                    <- mascot
  demo/                                    <- protected: synthetic demo screenshots from the prior AO Radar prototype

Read the spec

Start here:

Protected historical evidence:

Public-safety boundary

This repo is public. Treat it like a billboard with version control.

  • Synthetic examples only.
  • No secrets, credentials, private notes, raw transcripts, or local logs.
  • No real personal, travel, payment, operational, or government-system data.
  • No implication of live government deployment or official workflow authority.
  • No real DTS access, real vouchers, real claimants, real payments, or real fraud accusations.
  • No production-use claims.
  • Protected hackathon receipt files and demo screenshots stay untouched unless explicitly updated.

The specimen can be unsafe. The harness cannot.

About

Defiled Hackathon 2026 qualifier submission - Policy Bonfire: Defense Travel System From Hell πŸ”₯

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages