`fail_on_error=False` inflates accuracy by excluding errored samples from denominator

[xiangxl-a]

Problem

When fail_on_error=False, samples that error during execution are completely excluded from metric computation — both numerator and denominator. This inflates accuracy scores.

Example: 100 samples, 10 error, 80 of remaining 90 are correct → reported accuracy is 88.9% (80/90) rather than 80% (80/100).

Current behavior

1. task_run_sample returns None for errored samples
2. completed_scores filters to only isinstance(score_dict, dict) results
3. scorer_for_metrics further filters out NaN values
4. accuracy() computes sum(values) / len(scores) — only over successfully scored samples
5. A warning is printed to the display ("WARNING: N of M executed samples had errors and were not scored") but this is display-only and not reflected in the metric value

Suggested improvements

1. Add a task-level option that controls how errored samples are treated in metrics at eval time.
2. Document the current behavior explicitly — the Errors and Limits page documents fail_on_error but does not mention that errored samples are excluded from accuracy computation.

Environment

• inspect_ai version: 0.3.74
• Python: 3.12

[dragonstyle]

We can improve documentation around this, noting that since failed samples aren't scored, they aren't included in metric computation.

Aside from this, I think that conflating samples that fail due to an error with a specific score value is a pretty difficult problem given each scorer has a value that can be nearly any type of value. For a scorer that returns a dictionary of numbers, what would a failure return? Even in the case of correct/incorrect and accuracy, it is hard to determine universally how to handle this. Since the samples failed due to an error, it seems neither correct nor incorrect to me (e.g. the model might not have even had the opportunity to complete a response to the task, so whether it would've been correct or incorrect is unknown).

Thus I think treating the sample as unscored (and thus not participating in metrics computation) seems overall correct to me. This complexity is one of the reasons to not generally rely on fail_on_error=False, which IMO risks tolerating failures when those errors / failures should be treated as errors to be resolved in the task or infrastructure. I don't see how we'd configure this at a task level - I think we'd likely need to either somehow allow scorers to see 'failed' tasks so they could return a score value, or somehow specify a default score value for this case at the scorer level... I'll give it some thought but also open to suggestions if you have them.

[xiangxl-a]

It is indeed a difficult problem. In practice when benchmarking against hosted inference endpoints, inference errors can be specific to model providers, some of which cannot be easily resolved in the task or infrastructure (such as being blocked by certain content filters or model producing invalid tool call). For more complex benchmarks, sometimes we need to set fail_on_error=False in order to complete the run. The question is how to handle these failed samples from two runs on the same benchmark, each with certain failed samples, so that the final scores are "comparable". Excluding them from the dataset effectively changes the benchmark sample distribution, which could lead to an unfair comparison. On the other hand, as you called out, there is no uniform way to provide a default value when such inference errors occur. I agree the best option is to allow the task or the scorer to decide how to handle these failed samples (and the default option could be exclude to maintain backward-compatibility).

[jjallaire]

I have indeed heard demand for "marking errored samples as incorrect" -- I wonder if another approach would be to add --score-on-error which catches the error and then scores the sample in whatever state it is in (presumably leading to incorrect nearly all of the time). This could default to False for backwards compatibility.

[he-yufeng]

The --score-on-error approach cleanly sidesteps the "what default value?" problem by deferring to whatever the scorer returns on the partial state. Poked around the code to see how invasive the plumbing would be:

The returned-None path is at src/inspect_ai/_eval/task/run.py:1318-1319, and scoring itself happens inside the same outer try-block around line 1133 (score_result = await scorer(state, Target(sample.target))). Sketch:

1. New config flag: score_on_error: bool = False on TaskConfig (+ --score-on-error on _cli/eval.py).
2. Don't short-circuit on error: when score_on_error=True, instead of skipping straight to the return None path, run the same scorer loop against whatever partial state we have. The sample's existing error info already flows through create_eval_sample() at line 1237, so the log still records the error — this just adds a best-effort scoring pass on top.
3. Scorers can still error: if a scorer itself raises on partial state (e.g. state.output is missing), catch per-scorer and drop that one score — the sample's primary error stays the one reported to the user.
4. Metrics denominator: no further change in completed_scores / eval_results needed. Once the sample has a score dict it flows through the existing accuracy path and the inflation goes away.

On @dragonstyle's concern about scorer value types: with this design each scorer decides. For accuracy() with match()/includes() the scorer naturally returns INCORRECT on empty/no-output; for a dict-returning scorer, whatever its existing empty-output behavior is. No framework-level "default value" policy needed — the scorer owns it, which also keeps backwards-compat airtight since existing scorers see no new paths unless the user flips the flag.

Happy to send a PR if this direction looks right.

[jjallaire]

Score on error was implemented so this can now be closed.