Skip to content

upgrade_plan_experiment_resolution.md

Latest commit

 

History

History
103 lines (76 loc) · 5.27 KB

upgrade_plan_experiment_resolution.md

File metadata and controls

103 lines (76 loc) · 5.27 KB
Raw

Upgrade Plan: Experiment Resolution

What The Current Harness Already Does Well

  • It preserves the paper's main distinctions between surrogate utility, deployable utility, and structural/topology-sensitive reasoning.
  • It enforces explicit fixed-budget accounting over R_add = (C, M, L, V, H) and keeps W_max separate.
  • It includes a strong matched single-workspace baseline in the symbolic path.
  • It supports real local live execution through Ollama with gemma3:1b.
  • It logs machine-readable run records and event streams.
  • It already surfaces negative results instead of assuming scale-out wins.

Current Result Limitations

  • The live path does not yet create sufficiently different runtime information states across architectures.
  • Heterogeneous and homogeneous split prompts are too similar in practice.
  • Shared-memory and relay differences are mostly reflected in costs and proxy metrics, not strongly enough in actual prompt inputs.
  • Verification allocation is too coarse because many settings collapse into nearly identical behavior.
  • The single-workspace baseline does not yet face a rich enough candidate-selection problem in the live path.
  • Arithmetic-heavy live tasks can collapse into base-model weakness if tool usage is only conceptual.
  • Current summaries are architecture-level averages first; they are weak on task-level failure diagnosis and side-by-side engineering diffs.

Implementation-Induced vs Model-Capacity-Induced Issues

Mostly implementation-induced

  • Worker prompts not containing materially different subsets of evidence.
  • Shared-memory and relay paths not affecting downstream prompts strongly enough.
  • Lack of real local calculator execution despite tool-access concepts.
  • Verification actions being too discretized.
  • Weak task diff and failure-taxonomy reporting.
  • Single-workspace baseline not using a compact candidate pool plus selection.

Mostly model-capacity-induced

  • gemma3:1b on CPU can still fail on small multistep tasks even when the architecture is well designed.
  • Tiny models are brittle on exact formatting and on structured multi-fragment integration.
  • Some topology-sensitive live tasks may fail across the board if the task is above the model's solvable frontier.

Mixed

  • Weak live topology separation: partly due to prompt design, partly due to model limits.
  • Weak heterogeneous benefit: partly due to insufficiently differentiated context, partly due to the model not exploiting specialization strongly.

What Will Be Upgraded Now

  • Add module-specific live prompting so workers receive concrete partial evidence, not only role labels.
  • Make homogeneous vs heterogeneous split differ operationally in evidence, framing, and candidate generation.
  • Make relay workers consume bounded summaries rather than full original context.
  • Make shared-memory workers read and write a real shared scratchpad object that affects later prompts.
  • Add a deterministic local calculator tool path with explicit budget and logging.
  • Make verification-budget allocation more granular with light/deep checking behavior.
  • Strengthen the live single-workspace baseline by allowing a small candidate pool and explicit within-budget selection.
  • Add a task-level failure taxonomy and architecture diff reporting.
  • Add at least one engineer-facing structured task family.
  • Improve summaries with family-level and failure-mode views.

What Remains Future Work

  • Larger live task suites with stronger statistical power.
  • More realistic proxy-certification for structural quantities in live tasks.
  • Richer non-arithmetic tool use beyond a calculator while remaining local-only and safe.
  • More nuanced verifier models that do not rely on the same small base model.
  • Better live tasks that are solvable but still sharply topology-sensitive for gemma3:1b.

Upgrade Philosophy

The goal is not to make the small model look good.

The goal is to improve experimental resolution so that:

  • architectural differences become operational rather than merely decorative,
  • failure modes become legible,
  • and engineers can inspect why a given design failed under matched budget constraints.

Execution Status After Upgrade

The upgraded live pilots were executed on March 10, 2026 under local Ollama with gemma3:1b.

Implemented-and-executed now:

  • module-specific live prompting,
  • shared-memory and relay runtime information asymmetry,
  • deterministic calculator integration with budgeted logging,
  • finer-grained verification effort,
  • stronger single-baseline candidate pooling,
  • task-level failure taxonomy,
  • task-level architecture diff reports,
  • engineer-facing structured merge tasks,
  • updated pilot configs and summary artifacts.

Empirical status after execution:

  • verification and tool-confounding diagnostics became substantially sharper,
  • topology diagnostics are now informative but currently unfavorable to the shared-memory implementation,
  • the recalibrated engineer-facing tasks are now partially solvable, but still close to the reliable deployable frontier of the current model and budget.

Current run snapshot:

  • results/runs/20260310_170449_live_synthetic_v3
  • results/runs/20260310_170449_live_verification_v3
  • results/runs/20260310_170449_live_topology_v3
  • results/runs/20260310_170449_live_tool_v1
  • results/runs/20260310_170449_live_engineer_v2