Skip to content

feat: claude-cross-prompt agent (E4 prompt-sensitivity ablation)#38

Merged
benedikt-schesch merged 1 commit into
mainfrom
feat/claude-cross-prompt-baseline
Jun 9, 2026
Merged

feat: claude-cross-prompt agent (E4 prompt-sensitivity ablation)#38
benedikt-schesch merged 1 commit into
mainfrom
feat/claude-cross-prompt-baseline

Conversation

@benedikt-schesch

Copy link
Copy Markdown
Collaborator

Summary

E4 — claude-cross-prompt: directly answers Reviewer 2's verbatim question: "What happens when translate-library.md is used to translate an executable?"

Approach: SWAP project-type prompts. Libraries (cases ending in _lib) get translate-executable.md; executables get translate-library.md. Same agent, same harness, just the "wrong" prompt for each project type.

Scope: only meaningful on Test-Corpus batteries with lib/exec dispatch — n=214 cases (149 libs + 65 execs across B01_organic, B01_synthetic, B02_organic, B02_synthetic, P00_perlin). P01_sphincs uses translate-shared.md (no lib/exec dispatch); CRUST datasets use a single prompt regardless of project type. Ablation does not apply there.

Result — clean asymmetric finding

Direction Score Engineered Drop
Libs got exec prompt 4/149 (2.7%) ~145/149 (97%) -95pp catastrophic
Execs got lib prompt 59/65 (91%) ~63/65 (97%) -4pp tied

Per-battery breakdown

Battery Libs (cross) Execs (cross)
B01_organic 1/38 0/1
B01_synthetic 1/42 43/44
B02_organic 0/42 2/3
B02_synthetic 2/27 13/16
P00_perlin n/a 1/1

Headline finding

The project-type dispatch is not structurally necessary — only the FFI guidance is.

  • Library prompt on executables → essentially harmless (91% pass). The cdylib/#[no_mangle]/extern "C" instructions don't break anything when the case is actually an executable; the agent still produces a working binary.
  • Executable prompt on libraries → catastrophic (2.7%). Without FFI guidance, the agent writes idiomatic Rust that compiles cleanly but produces a .so with no exported C symbols. Runtime FFI tests fail to dlopen+dlsym.

Paper-level implication

We can publish a major prompt-simplification recommendation:

  • Drop translate-executable.md entirely
  • Use translate-library.md for both libs and execs
  • Two-prompt dispatch (library / shared) replaces three-prompt (exec / library / shared) with no measured loss

Harness changes

  • New Agent variant: ClaudeCrossPrompt (cli.rs)
  • Verify phase skipped (consistent with other ablation agents)
  • dispatch_translate routes is_lib=truetranslate-executable.md and is_lib=falsetranslate-library.md (the swap)
  • dispatch_translate_shared falls through to standard translate-shared.md

Harness bug fixes (discovered during E4 development)

  • translate_one_independent now skips post_process_independent for ClaudeCrossPrompt. The lib-vs-bin override was forcing every Cargo.toml to mismatch the agent's actual files (lib cases got [lib] even when the agent wrote main.rs), causing 100% build failure that hid the actual experimental result.
  • [workspace] section is still added to each Cargo.toml so cargo doesn't try to absorb each case into a parent workspace ("current package believes it's in a workspace when it's not" error).

Cost

  • $93.66 over 208 sessions (4 fresh batteries + B01_synthetic v3 from earlier patched run)
  • Two earlier corrupted runs ($24.33 + $25.37) were overwritten before commit — the bugs were caught and fixed before any committed data was wrong

Test plan

  • All 5 in-scope batteries completed cleanly across 214 cases
  • cargo build --release clean on tools/
  • CLI --help lists the new agent
  • Existing agents untouched
  • Submodule pointer reachable from results/main

Directly answers Reviewer 2's verbatim question: "What happens when
translate-library.md is used to translate an executable?"

E4 — claude-cross-prompt: SWAP project-type prompts. Libraries (cases
ending in _lib) are translated using translate-executable.md; executables
are translated using translate-library.md. Same agent_plain configuration,
same harness flow, just the "wrong" prompt for each project type.

Scope: only meaningful on Test-Corpus batteries with lib/exec dispatch
(B01_organic, B01_synthetic, B02_organic, B02_synthetic, P00_perlin) —
n=214 cases (149 libs + 65 execs). P01_sphincs uses translate-shared.md
(no lib/exec dispatch) and CRUST datasets use a single prompt regardless
of project type, so the ablation does not apply there.

Final scores:
  Direction              | Cross-prompt | Engineered baseline | Drop
  Libs got exec prompt   | 4/149  (2.7%)| ~145/149            | -95pp ★
  Execs got lib prompt   | 59/65  (91%) | ~63/65              | -4pp tied

Per-battery:
  B01_organic:    libs 1/38  execs 0/1
  B01_synthetic:  libs 1/42  execs 43/44
  B02_organic:    libs 0/42  execs 2/3
  B02_synthetic:  libs 2/27  execs 13/16
  P00_perlin:     execs 1/1

Headline finding: the project-type dispatch is not structurally necessary —
only the FFI guidance is. The library prompt is essentially harmless on
executables (91% pass), but the executable prompt is catastrophic on
libraries (2.7%). Without #[no_mangle] / extern "C" / cdylib instructions,
the agent writes idiomatic Rust that compiles cleanly but produces a .so
with no exported C symbols. Runtime FFI tests fail to dlopen+dlsym.

This is a publishable prompt-simplification recommendation: drop
translate-executable.md entirely; use translate-library.md for both libs
and execs. Two-prompt dispatch (library/shared) replaces three-prompt
(exec/library/shared) with no measured loss.

Harness changes:
- New Agent variant: ClaudeCrossPrompt (cli.rs)
- Verify phase skipped (E4 prompt-sensitivity ablation)
- Battery resolves to "claude-cross-prompt" results dir
- dispatch_translate routes is_lib=true → translate-executable.md and
  is_lib=false → translate-library.md (the swap)
- dispatch_translate_shared falls back to standard translate-shared.md
  (cross-prompt has no analog for shared-source batteries; out of scope)

Harness bug fixes (also in this PR, discovered during E4 development):
- translate_one_independent skips post_process_independent for
  ClaudeCrossPrompt. The lib-vs-bin override was forcing every Cargo.toml
  to mismatch the agent's actual files (lib cases got [lib] when agent
  wrote main.rs, etc.), causing 100% build failure that hid the actual
  experimental result.
- [workspace] section is still added to each Cargo.toml so cargo doesn't
  try to absorb each case into a parent workspace ("current package
  believes it's in a workspace when it's not" error).

Cost: $93.66 over 208 sessions (4 fresh batteries + B01_synthetic v3 from
earlier patched run). Two earlier corrupted runs ($24.33 + $25.37) were
overwritten before commit.
@benedikt-schesch benedikt-schesch merged commit 86040de into main Jun 9, 2026
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants