feat: claude-no-subtask agent (E6 prompt-sensitivity ablation) (#37)

New Claude Code agent variant for the IEEE S&P 2026 paper revision,
testing whether the explicit subtask-decomposition guidance in the
shared-source prompt is load-bearing.

E6 — claude-no-subtask: engineered prompts minus the subtask-decomposition
block. Identical to `claude` except translate-shared.md is replaced with
translate-no-subtask-shared.md, which retains all structural guidance
(cmake-features, cdylib, FFI types, namespace warnings, byte-identical,
no-openssl) but drops the 17-line "create TODO list, work through subtasks
one at a time, do final feature-gate wiring" block.

Scope: only differs from `claude` on shared-source batteries. The only
one in our datasets is P01_sphincs_plus.

Result on P01_sphincs_plus: 116/129 (90%) ★

Comparison across ablations:
  claude-minimal      1/129  (0.8%)  — strips everything
  claude-no-features  5/129  (3.9%)  — strips ONLY cmake-features
  claude (engineered) 41/129 (32%)   — full prompt
  claude-no-iter     48/129 (37%)    — strips ONLY iteration loop
  claude-no-subtask 116/129 (90%)    — strips ONLY subtask decomposition

Surprising negative result: removing the subtask-decomposition guidance
*improves* P01 pass rate from 32% to 90% — a 58pp gain. The engineered
"create TODO list, work through subtasks" instructions over-prescribe in
a way that hurts. With them removed the agent tackles SPHINCS+ holistically
in 172 turns and produces a cleaner translation. Cost ($22.84) is also
lower than engineered ($27.39).

For the paper: well-intentioned prompt scaffolding can actively harm model
performance when the model has competent natural decomposition behavior.

Cost: $22.84, 1 LLM session, 172 turns, 57 min wall time.

Harness changes:
- New Agent variant: ClaudeNoSubtask (cli.rs)
- Verify phase skipped
- Battery resolves to "claude-no-subtask" results dir
- dispatch_translate routes independent cases to standard claude prompts
  (no subtask block in those anyway), shared cases to translate-no-subtask-shared.md
- run_crust / run_crust_blind use standard prompts (CRUST scaffolds don't
  have subtask blocks) — falls into default match arm

Co-authored-by: Benedikt Schesch <scheschb@amazon.co.uk>

Author: benedikt-schesch

prompts/claude/translate-no-subtask-shared.md

@@ -0,0 +1,35 @@
+<!-- markdownlint-disable MD041 -->
+Translate the C code in c_src/ to Rust that produces **byte-identical output** for the same inputs.
+Write Cargo.toml and src/ files in the current directory (NOT in c_src/).
+
+You MUST translate ALL C source files — no stubs, no placeholders, no empty
+functions. Every .c file MUST have a complete Rust equivalent. The binary MUST
+produce the same stdout as the C binary for the same inputs.
+
+This project has **build-time configurability** via CMake cache variables.
+Look at c_src/CMakeLists.txt — it uses variables to select which source files
+to compile and which parameter headers to include at build time.
+
+You MUST preserve this configurability using **Cargo features**. Each CMake cache
+variable value becomes a Cargo feature, using the **exact same name in lowercase**.
+Use `#[cfg(feature = "...")]` to conditionally compile modules and set constants.
+All combinations of features must compile.
+
+This project produces BOTH a shared library AND a binary executable.
+Your Cargo.toml must have both `[lib]` with `crate-type = ["cdylib"]` and
+`[[bin]]` with `name = "driver"` and `path = "src/main.rs"`.
+
+Requirements:
+- Do NOT use the `openssl` crate or any OpenSSL bindings. Use pure-Rust crates
+  instead (e.g., `aes` for AES-256-ECB, `sha2` for SHA-256)
+- All public C functions must use #[unsafe(no_mangle)] and extern "C"
+- Pay attention to C preprocessor macros that RENAME functions (e.g.,
+  `#define foo NAMESPACE(foo)` makes the linker symbol `PREFIX_foo`, not `foo`).
+  The Rust #[no_mangle] name must match the FINAL linker symbol, not the
+  source-level name. Check header files for namespace macros.
+- Preserve the exact C function signatures (use *const c_char, c_int, etc. from std::ffi)
+- Do NOT fix bugs in the original C code — reproduce behavior exactly
+- Use safe Rust internally where possible
+
+Run 'cargo build --release' and fix any errors until it compiles.
+Do NOT modify anything in c_src/.

results

@@ -548,6 +548,7 @@ impl Paths {
             Agent::ClaudeMinimal => "claude-minimal",
             Agent::ClaudeNoIter => "claude-no-iter",
             Agent::ClaudeNoFeatures => "claude-no-features",
+            Agent::ClaudeNoSubtask => "claude-no-subtask",
             Agent::C2rust => "c2rust",
             Agent::Laertes => "laertes",
             Agent::Kimi => "kimi",
@@ -570,7 +571,7 @@ impl Paths {
             ),
         };
         let prompts_dir = match agent {
-            Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures => match dataset {
+            Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures | Agent::ClaudeNoSubtask => match dataset {
                 Dataset::TestCorpus => repo_root.join("prompts/claude"),
                 Dataset::Crust | Dataset::BlindCrust => repo_root.join("prompts/claude/crust"),
             },

tools/src/battery.rs

@@ -27,6 +27,13 @@ pub enum Agent {
     /// Tests whether the cmake-features dispatch is what carries P01_sphincs_plus.
     /// Verify phase is skipped.
     ClaudeNoFeatures,
+    /// Claude Code with engineered prompts but no subtask-decomposition
+    /// guidance (E6 prompt-sensitivity ablation). Identical to `claude` except
+    /// the shared-source prompt drops the "create a TODO list, work through
+    /// subtasks one at a time" block. Tests whether explicit decomposition
+    /// guidance is needed for large multi-file projects (P01_sphincs_plus).
+    /// Verify phase is skipped.
+    ClaudeNoSubtask,
     C2rust,
     Laertes,
     Kimi,

tools/src/cli.rs

@@ -37,14 +37,15 @@ fn main() -> Result<()> {
 
             let tp = make_translate_plan(&paths, inner, include_regex.as_deref(), parallel, limit)?;
             // ClaudeCombined merges translate+verify into one prompt; ClaudeMinimal,
-            // ClaudeNoIter, and ClaudeNoFeatures have no verify by design (prompt-sensitivity
-            // ablations). All four skip the verify phase.
+            // ClaudeNoIter, ClaudeNoFeatures, and ClaudeNoSubtask have no verify by
+            // design (prompt-sensitivity ablations). All five skip the verify phase.
             let vp = if no_verify
                 || dataset == Dataset::Crust
                 || agent == cli::Agent::ClaudeCombined
                 || agent == cli::Agent::ClaudeMinimal
                 || agent == cli::Agent::ClaudeNoIter
                 || agent == cli::Agent::ClaudeNoFeatures
+                || agent == cli::Agent::ClaudeNoSubtask
             {
                 VerifyPlan::Skip
             } else {

tools/src/main.rs

@@ -262,6 +262,13 @@ fn dispatch_translate(paths: &Paths, battery: &str, name: &str, is_lib: bool) ->
             let prompt = std::fs::read_to_string(paths.prompts_dir.join(f)).unwrap_or_default();
             translate_case(paths, battery, name, &prompt)
         }
+        Agent::ClaudeNoSubtask => {
+            // E6: subtask-decomposition ablation only affects shared-source cases.
+            // Independent (executable/library) cases reuse the engineered claude prompts.
+            let f = if is_lib { "translate-library.md" } else { "translate-executable.md" };
+            let prompt = std::fs::read_to_string(paths.prompts_dir.join(f)).unwrap_or_default();
+            translate_case(paths, battery, name, &prompt)
+        }
         Agent::C2rust => translate_case(paths, battery, name, ""),
     }
 }
@@ -293,6 +300,11 @@ fn dispatch_translate_shared(paths: &Paths, battery: &str, name: &str) -> Result
             let prompt = std::fs::read_to_string(paths.prompts_dir.join("translate-no-features-shared.md")).unwrap_or_default();
             translate_case(paths, battery, name, &prompt)
         }
+        Agent::ClaudeNoSubtask => {
+            // E6: shared-source prompt without subtask-decomposition guidance.
+            let prompt = std::fs::read_to_string(paths.prompts_dir.join("translate-no-subtask-shared.md")).unwrap_or_default();
+            translate_case(paths, battery, name, &prompt)
+        }
         Agent::C2rust => translate_case(paths, battery, name, ""),
     }
 }
@@ -302,7 +314,7 @@ fn dispatch_translate_shared(paths: &Paths, battery: &str, name: &str) -> Result
 fn preflight_check(agent: Agent) -> Result<()> {
     let (cmd, version_args): (&str, &[&str]) = match agent {
         Agent::Kiro | Agent::KiroTranslate => ("kiro-cli", &["--version"]),
-        Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures => ("claude", &["--version"]),
+        Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures | Agent::ClaudeNoSubtask => ("claude", &["--version"]),
         Agent::C2rust => ("c2rust", &["--version"]),
         Agent::Laertes => ("docker", &["--version"]),
         Agent::Kimi => ("aws", &["sts", "get-caller-identity"]),
@@ -327,7 +339,7 @@ fn preflight_check(agent: Agent) -> Result<()> {
         println!("  {line}");
     }
 
-    if matches!(agent, Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures) {
+    if matches!(agent, Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures | Agent::ClaudeNoSubtask) {
         let stdout = String::from_utf8_lossy(&output.stdout);
         // Claude version output may be "2.1.150.280 ..." (older) or
         // "claude 2.1.158.312 ..." (newer). Match any line containing a digit-dot-digit pattern.
@@ -369,7 +381,7 @@ fn translate_case(paths: &Paths, battery: &str, name: &str, prompt: &str) -> Res
     let openssl_dir = std::env::var("OPENSSL_DIR").unwrap_or_else(|_| "/usr".into());
 
     let (work_dir, _tmp_guard) = match paths.agent {
-        Agent::Kiro | Agent::KiroTranslate | Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures | Agent::C2rust | Agent::Laertes | Agent::Kimi | Agent::Oneshot => {
+        Agent::Kiro | Agent::KiroTranslate | Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures | Agent::ClaudeNoSubtask | Agent::C2rust | Agent::Laertes | Agent::Kimi | Agent::Oneshot => {
             let tmp = tempfile::Builder::new()
                 .prefix("harvest-translate-")
                 .tempdir()
@@ -379,7 +391,7 @@ fn translate_case(paths: &Paths, battery: &str, name: &str, prompt: &str) -> Res
             std::fs::create_dir_all(&c_src)?;
             copy_dir_all(&input_test_case, &c_src)?;
 
-            if matches!(paths.agent, Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures) {
+            if matches!(paths.agent, Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures | Agent::ClaudeNoSubtask) {
                 let claude_dir = tmp.path().join(".claude");
                 std::fs::create_dir_all(&claude_dir)?;
                 let repo_root = paths.results_dir.parent().unwrap_or(Path::new("/"));
@@ -416,7 +428,7 @@ fn translate_case(paths: &Paths, battery: &str, name: &str, prompt: &str) -> Res
                 .status()
                 .context("invoking kiro-cli")?;
         }
-        Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures => {
+        Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures | Agent::ClaudeNoSubtask => {
             let settings_path = work_dir.parent().unwrap().join(".claude/settings.json");
             let _status = Command::new("bash")
                 .arg("-lc")
@@ -653,7 +665,7 @@ fn invoke_agent(agent: Agent, prompt: &str, log_path: &Path, work: &Path) -> Res
                 .status()
                 .context("invoking kiro-cli for CRUST")?;
         }
-        Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures => {
+        Agent::Claude | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures | Agent::ClaudeNoSubtask => {
             // Write a minimal settings.json so --settings can find one
             let claude_dir = work.parent().unwrap_or(work).join(".claude");
             std::fs::create_dir_all(&claude_dir)?;

tools/src/translate.rs

@@ -202,11 +202,12 @@ fn verify_case(case_dir: &Path, prompt_template: &str, cmake_flags: &str, config
                 .status()
                 .context("invoking claude for verification")?;
         }
-        Agent::C2rust | Agent::Laertes | Agent::Kimi | Agent::Oneshot | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures => {
+        Agent::C2rust | Agent::Laertes | Agent::Kimi | Agent::Oneshot | Agent::ClaudeCombined | Agent::ClaudeMinimal | Agent::ClaudeNoIter | Agent::ClaudeNoFeatures | Agent::ClaudeNoSubtask => {
             // ClaudeCombined: translate phase already did verify, skip this phase.
             // ClaudeMinimal: no verify phase (calibration baseline).
             // ClaudeNoIter: no verify phase (E3 prompt-sensitivity ablation).
             // ClaudeNoFeatures: no verify phase (E2 prompt-sensitivity ablation).
+            // ClaudeNoSubtask: no verify phase (E6 prompt-sensitivity ablation).
             // c2rust/laertes/kimi/oneshot: no verify phase by design.
             return Ok(true);
         }