Fix

[anandhu-eng]

Removes llama2-70b-99 from the v6.1 config in tools/submission/submission_checker_old.py.

Why

#2651 removed llama2-70b-99 from the v6.1 block of the modular checker's constants.py, but left the legacy submission_checker_old.py untouched. The two configs then disagreed about whether the model exists in v6.1.

preprocess_submission.py does import submission_checker_old as checker, and its infer_low_accuracy_results pass copies a -99.9 result to -99 whenever the low-accuracy variant is in config.required. Against the legacy v6.1 config that condition still held, so preprocessing recreated the model directory:

[preprocess_submission.py:541 INFO] Division closed, submitter MLCommons,
  system SR680a_V3_B200SXMx8_TRT: copying llama2-70b-99.9 results to llama2-70b-99

The modular checker then walked that directory, Config.get_required() returned None for the unknown model, and lower_list(None) raised:

File "submission_checker/loader.py", line 249, in check_scenarios
    required_scenarios = lower_list(self.config.get_required(mlperf_model))
File "submission_checker/utils.py", line 119, in lower_list
    return [str(e).lower() for e in l]
TypeError: 'NoneType' object is not iterable

That crashed all six round=6.1 / folder_name=closed submission generation jobs on every open PR. Round 6.0 passes because it legitimately still has the model, and the 6.1 open fixtures never reach check_scenarios() — it only runs for closed/network divisions.

Change

Mirrors the #2651 deletions site for site — models, both required-scenarios-* and optional-scenarios-* dicts, accuracy-target, accuracy-upper-limit, performance-sample-count, dataset-size, latency-constraint, min-queries, and models_TEST06. Same 21 lines, v6.1 only. v6.0 and v5.1 keep the model.

Paired with a matching cleanup on the submission-generation-examples branch (14c5196), which dropped the stale llama2-70b-99 fixture from submission_round_6.1/closed/. Both are needed: the fixture removal stops the raw directory being copied in, and this stops preprocessing from recreating it.

Note

Worth considering as a follow-up: check_scenarios() could treat a None from get_required() as an "unknown benchmark" error rather than letting it become a traceback. A submitter with a retired or misspelled model directory currently gets an unreadable stack trace and no summary.csv, which then cascades into a second FileNotFoundError from generate_final_report.py.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅