Bound and pre-load the DeepSeek-R1 LiveCodeBench grader workers

[100-JM]

Problem

language/deepseek-r1/eval_accuracy.py grades LiveCodeBench in a ProcessPoolExecutor sized to multiprocessing.cpu_count(). load_lcb_benchmark() is lru_cached, but the cache lives per process, so every worker re-reads the full 400-problem benchmark on its first item. On a 256-core host (DGX B300) that is 256 concurrent loads; the OOM killer takes the pool down, every future fails, and both failure paths score the sample as wrong (except Exception: return False in the worker, prompt_accuracy = 0.0 around future.result()).

What a submitter sees is a run that finishes in seconds with livecodebench 0/335 and an overall exact_match of 74.11 instead of 81.02 — below the 80.5446 threshold — with nothing in the reported result indicating that the grader, not the model, failed. We spent two days on model/parallelism/kernels before grep "results:" on the grader log showed the per-dataset breakdown. Re-grading the same mlperf_log_accuracy.json with a bounded pool gave 302/335 and 81.02.

Changes

• Pre-load the benchmark in the parent before creating the pool, so forked workers inherit it copy-on-write. This is the pattern gpt-oss-120b/eval_accuracy.py already uses.
• Bound the pool with --num-lcb-workers (default 64, same as gpt-oss-120b; still capped by cpu_count and the number of samples).
• Stop scoring grader failures as wrong answers. The worker returns None (and logs the traceback) when the grader raises; the collector counts those rows instead of writing 0.0. If any remain, LiveCodeBenchEvaluationError is raised instead of printing an accuracy that silently under-counts. A BrokenProcessPool is reported once with a hint to lower --num-lcb-workers, rather than as 335 individual "Error evaluating row" lines.
• README: document the option and the failure behaviour.

Scores on the normal path are unchanged; this only changes resource usage and what happens when the grader itself fails. I considered making the failure a warning instead of an error — happy to switch if maintainers prefer, but for an MLPerf accuracy script a wrong number seems worse than no number.

Verification

Stubbed load_lcb_benchmark / evaluate_livecodebench and drove process_livecodebench_parallel directly (fork start method):

• normal path: scores identical, benchmark loaded exactly once in the parent;
• a grader that raises on one sample: LiveCodeBenchEvaluationError: 1 of 4 LiveCodeBench samples could not be evaluated …, that row left unscored;
• a worker that os._exits: single worker pool died … error instead of N zeros;
• autopep8 -a --max-line-length 79 clean.

Real-data confirmation from the incident above: the same log re-graded with a bounded pool moved from 0/335 to 302/335 on LiveCodeBench (other subjects unchanged).

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
❌ @100-JM
_{You can retrigger this bot by commenting recheck in this Pull Request}

[100-JM]

recheck