metric_definitions.md

Metric Definitions

This file follows the paper's measurement separation rule. Each metric is tagged as observable, proxy-certifiable, or latent-unidentified in this repository.

Metric	Meaning	Repo Status	Notes
q_x(A)	tractable surrogate utility	observable relative to the declared surrogate pipeline	Used for explanatory comparisons inside the harness; its meaning depends on the declared scoring pipeline.
u_x^{dep}(A)	deployable task utility	observable on synthetic pilots	On benchmark-style tasks it can inherit proxy status through coverage.
coverage_kappa	candidate coverage kappa_x(A)	observable on synthetic and oracle-tagged samples	Must remain labeled proxy-certifiable when oracle correctness is unavailable.
conditional_selection_accuracy_nu	nu_x(A)	observable	Logged only when coverage occurs.
hijack_alpha	alpha_x(A)	observable on oracle-tagged tasks	A proxy on tasks where correctness is only proxy-certified.
gamma_proxy	decision-preserving fidelity Gamma(A,x)	proxy-certifiable	Implemented as a declared topology-sensitive fidelity proxy.
rho_observable_approx	signed shared-failure dependence rho(A,x)	observable approximation	Current implementation is a simple within-run approximation, not a full estimator.
decomposability_Pi_r	low-order interaction ratio Pi_r(x)	proxy-certifiable in general; oracle on synthetic generators	Synthetic tasks expose exact interaction structure unavailable on most real tasks.
decomposability_Theta_r	residual tail ratio Theta_r(x)	proxy-certifiable in general; oracle on synthetic generators	Same caveat as above.
signed_diversity_against_single	baseline-adjusted partition-induced diversity	observable relative to declared pipeline	Depends on evidence taxonomy and clustering choices.
latency_penalty_Lambda	latency penalty Lambda_x(A)	observable	Derived from critical path and deadline.
verifier_signal	candidate-side verifier support used in final selection	observable on oracle verifiers; proxy-certifiable on tool/heuristic verifiers	The run record labels verifier mode explicitly.
verification_label	operational verifier-budget label such as low, medium_low, balanced, or high	observable	This is a reporting/control label, not a scientific latent quantity.
verification_plan	mapping from verifier budget to light/deep checks	observable	Logged to make verification-share pilots interpretable.
tool_calls	local tool usage count	observable	Includes worker-side and verifier-side calculator calls.
shared_memory_note_count	number of shared-memory notes visible or written in a run	observable	Useful for diagnosing scratchpad growth and memory pressure.
shared_memory_read_events	total count of shared-memory read actions	observable	Counts explicit scratchpad reuse events in the runtime trace.
shared_memory_duplicate_ratio	fraction of shared-memory notes judged textually duplicate	observable	A lightweight redundancy diagnostic, not a latent structural estimator.
failure_label	task-level failure taxonomy label	observable relative to the declared taxonomy	Useful for engineering diagnosis; not itself a theorem.
local prompt settings	local Ollama generation parameters	observable	Logged for reproducibility when a model backend is configured.
recoverable critical interaction mass	latent structural quantity	latent-unidentified outside synthetic generators	Not directly identified by the current benchmark adapters.

Measurement Pipeline Dependence

The reported diversity and several proxy metrics depend on the declared measurement pipeline:

• evidence taxonomy version,
• claim clustering rule,
• verifier calibration rule,
• stratum definition.

Changing the pipeline can change the metric without changing the underlying execution trace. The pipeline therefore belongs in the run record, not a footnote.

Current Empirical Note

In the March 10, 2026 rerun snapshot under results/runs/20260310_170449_*, q_x(A) and u_x^{dep}(A) diverged most clearly in:

• the shared-memory synthetic and topology conditions,
• the engineer-style structured merge pilot.

The new shared-memory trace metrics were most useful exactly where deployable utility collapsed:

• shared_memory_stale_reuse dominated the topology pilot,
• shared_memory_prompt_bloat and shared_memory_stale_reuse both appeared in the engineer-task pilot.

That is a useful reminder that surrogate-side gains should not be read as deployable gains without coverage, selector success, and an inspectable runtime trace.