Skip to content

fix(TEST01): NameError in verify_performance when the loadgen log reports errors - #2666

Open
Anai-Guo wants to merge 1 commit into
mlcommons:masterfrom
Anai-Guo:fix-test01-undefined-line
Open

fix(TEST01): NameError in verify_performance when the loadgen log reports errors#2666
Anai-Guo wants to merge 1 commit into
mlcommons:masterfrom
Anai-Guo:fix-test01-undefined-line

Conversation

@Anai-Guo

@Anai-Guo Anai-Guo commented Sep 9, 2026

Copy link
Copy Markdown

compliance/TEST01/verify_performance.py::parse_result_log formats a warning with a name that
does not exist in that scope:

    if mlperf_log.has_error():
        print(
            "WARNING: {} ERROR reported in {}".format(
                line.split()[0],        # <- NameError: name 'line' is not defined
                file_path))
$ pyflakes compliance/TEST01/verify_performance.py
compliance/TEST01/verify_performance.py:57:17: undefined name 'line'

line is a leftover from when this file scanned the summary as text — the sibling
compliance/TEST04/verify_performance.py:88 still does exactly that
(error = line.split(" ", 1)[0].strip() inside for line in test_file:). TEST01 was moved to
the parsed MLPerfLog object and the format argument was not moved with it.

The consequence is that TEST01 verification crashes only when the loadgen log actually
reports errors
— i.e. precisely the case this warning was written for. On a clean run
has_error() is False and the branch is skipped, which is why it has gone unnoticed.

Replaying parse_result_log (lifted out of the file with ast) against a stub MLPerfLog
for an Offline run whose log reports 3 errors:

### loadgen log reports 3 errors (mlperf_log.has_error() is True)
  [before] NameError: name 'line' is not defined
  [after]  WARNING: 3 ERROR reported in loadgen.txt
           returned ('Offline', 1234.5, None)

Fix

Use the log object's own error count, mlperf_log.num_errors() — the accessor
has_error() is itself defined in terms of (log_parser.py:160), and the one used for this
exact message everywhere else in the repo:

  • tools/submission/log_parser.py:214"- Number of errors: {:d}".format(mlperf_log.num_errors())
  • tools/submission/submission_checker/checks/performance_check.py:168
  • tools/submission/submission_checker/checks/accuracy_check.py:276

This keeps the rendered message identical in shape to TEST04's
("WARNING: " + error + " ERROR reported in ...").

🤖 Generated with Claude Code

parse_result_log formatted the warning with `line`, a leftover from when
the file scanned the summary as text (compliance/TEST04 still does). The
branch only runs when mlperf_log.has_error() is True, so TEST01
verification crashed exactly when the loadgen log reported errors.

num_errors() is what has_error() is defined in terms of and what the
submission checker uses for this same message.
@Anai-Guo
Anai-Guo requested review from a team as code owners September 9, 2026 04:22
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
@Anai-Guo
You can retrigger this bot by commenting recheck in this Pull Request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant