feat: claude-cross-prompt agent (E4 prompt-sensitivity ablation)#38
Merged
Merged
Conversation
Directly answers Reviewer 2's verbatim question: "What happens when
translate-library.md is used to translate an executable?"
E4 — claude-cross-prompt: SWAP project-type prompts. Libraries (cases
ending in _lib) are translated using translate-executable.md; executables
are translated using translate-library.md. Same agent_plain configuration,
same harness flow, just the "wrong" prompt for each project type.
Scope: only meaningful on Test-Corpus batteries with lib/exec dispatch
(B01_organic, B01_synthetic, B02_organic, B02_synthetic, P00_perlin) —
n=214 cases (149 libs + 65 execs). P01_sphincs uses translate-shared.md
(no lib/exec dispatch) and CRUST datasets use a single prompt regardless
of project type, so the ablation does not apply there.
Final scores:
Direction | Cross-prompt | Engineered baseline | Drop
Libs got exec prompt | 4/149 (2.7%)| ~145/149 | -95pp ★
Execs got lib prompt | 59/65 (91%) | ~63/65 | -4pp tied
Per-battery:
B01_organic: libs 1/38 execs 0/1
B01_synthetic: libs 1/42 execs 43/44
B02_organic: libs 0/42 execs 2/3
B02_synthetic: libs 2/27 execs 13/16
P00_perlin: execs 1/1
Headline finding: the project-type dispatch is not structurally necessary —
only the FFI guidance is. The library prompt is essentially harmless on
executables (91% pass), but the executable prompt is catastrophic on
libraries (2.7%). Without #[no_mangle] / extern "C" / cdylib instructions,
the agent writes idiomatic Rust that compiles cleanly but produces a .so
with no exported C symbols. Runtime FFI tests fail to dlopen+dlsym.
This is a publishable prompt-simplification recommendation: drop
translate-executable.md entirely; use translate-library.md for both libs
and execs. Two-prompt dispatch (library/shared) replaces three-prompt
(exec/library/shared) with no measured loss.
Harness changes:
- New Agent variant: ClaudeCrossPrompt (cli.rs)
- Verify phase skipped (E4 prompt-sensitivity ablation)
- Battery resolves to "claude-cross-prompt" results dir
- dispatch_translate routes is_lib=true → translate-executable.md and
is_lib=false → translate-library.md (the swap)
- dispatch_translate_shared falls back to standard translate-shared.md
(cross-prompt has no analog for shared-source batteries; out of scope)
Harness bug fixes (also in this PR, discovered during E4 development):
- translate_one_independent skips post_process_independent for
ClaudeCrossPrompt. The lib-vs-bin override was forcing every Cargo.toml
to mismatch the agent's actual files (lib cases got [lib] when agent
wrote main.rs, etc.), causing 100% build failure that hid the actual
experimental result.
- [workspace] section is still added to each Cargo.toml so cargo doesn't
try to absorb each case into a parent workspace ("current package
believes it's in a workspace when it's not" error).
Cost: $93.66 over 208 sessions (4 fresh batteries + B01_synthetic v3 from
earlier patched run). Two earlier corrupted runs ($24.33 + $25.37) were
overwritten before commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
E4 —
claude-cross-prompt: directly answers Reviewer 2's verbatim question: "What happens whentranslate-library.mdis used to translate an executable?"Approach: SWAP project-type prompts. Libraries (cases ending in
_lib) gettranslate-executable.md; executables gettranslate-library.md. Same agent, same harness, just the "wrong" prompt for each project type.Scope: only meaningful on Test-Corpus batteries with lib/exec dispatch — n=214 cases (149 libs + 65 execs across B01_organic, B01_synthetic, B02_organic, B02_synthetic, P00_perlin). P01_sphincs uses
translate-shared.md(no lib/exec dispatch); CRUST datasets use a single prompt regardless of project type. Ablation does not apply there.Result — clean asymmetric finding
Per-battery breakdown
Headline finding
The project-type dispatch is not structurally necessary — only the FFI guidance is.
#[no_mangle]/extern "C"instructions don't break anything when the case is actually an executable; the agent still produces a working binary..sowith no exported C symbols. Runtime FFI tests fail todlopen+dlsym.Paper-level implication
We can publish a major prompt-simplification recommendation:
translate-executable.mdentirelytranslate-library.mdfor both libs and execslibrary/shared) replaces three-prompt (exec/library/shared) with no measured lossHarness changes
Agentvariant:ClaudeCrossPrompt(cli.rs)dispatch_translateroutesis_lib=true→translate-executable.mdandis_lib=false→translate-library.md(the swap)dispatch_translate_sharedfalls through to standardtranslate-shared.mdHarness bug fixes (discovered during E4 development)
translate_one_independentnow skipspost_process_independentforClaudeCrossPrompt. The lib-vs-bin override was forcing every Cargo.toml to mismatch the agent's actual files (lib cases got[lib]even when the agent wrotemain.rs), causing 100% build failure that hid the actual experimental result.[workspace]section is still added to each Cargo.toml so cargo doesn't try to absorb each case into a parent workspace ("current package believes it's in a workspace when it's not" error).Cost
Test plan
cargo build --releaseclean on tools/--helplists the new agent