Skip to content

Default --audit-conf to audit.config in the reference entry points - #2672

Open
100-JM wants to merge 1 commit into
mlcommons:masterfrom
100-JM:fix/audit-config-default
Open

Default --audit-conf to audit.config in the reference entry points#2672
100-JM wants to merge 1 commit into
mlcommons:masterfrom
100-JM:fix/audit-config-default

Conversation

@100-JM

@100-JM 100-JM commented Sep 11, 2026

Copy link
Copy Markdown

Problem

Seven reference entry points default the LoadGen audit filename to audit.conf:

language/{llama3.1-8b,llama2-70b,llama3.1-405b,mixtral-8x7b,gpt-j}/main.py, language/bert/run.py, speech2text/reference_mlperf.py

but every compliance test under compliance/ ships the file as audit.config (renamed repo-wide in #2127), compliance/TEST06/README.md tells submitters to copy audit.config into the run directory, and both the loadgen Python binding (python_api.cc: audit_config_filename = "audit.config") and text_to_image/main.py already default to audit.config.

loadgen only applies the override when FileExists(audit_config_filename) (loadgen.cc:1232). So a compliance run that follows the README with one of these entry points runs as a plain performance run: no error, no Found Audit Config file line in mlperf_log_detail.txt, and a run_verification.py that then fails or verifies logs that were never produced under audit settings. We noticed this while setting up llama3.1-8b TEST06 — the README's advice to check for that log line is the only thing that exposes it.

Change

default="audit.conf"default="audit.config" in the seven files. Passing --audit-conf explicitly keeps working; this only changes what happens when the documented procedure is followed without it. No loadgen changes.

autopep8 -a --max-line-length 79 clean on all seven files.

🤖 Generated with Claude Code

…hipped file

Seven reference entry points defaulted the LoadGen audit filename to
"audit.conf", but every compliance test in compliance/ ships the file
as audit.config (renamed repo-wide in mlcommons#2127), the TEST06 README tells
submitters to copy audit.config into the run directory, and both the
loadgen Python binding and text_to_image/main.py already default to
audit.config.

loadgen only applies the audit override when FileExists(name) is true,
so with the old default a compliance run that followed the README ran
as a plain performance run: no error, no "Found Audit Config file" line
in mlperf_log_detail.txt, and a TEST06 verification that then fails or
verifies the wrong logs. Aligning the defaults makes the documented
procedure work without passing --audit-conf audit.config explicitly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@100-JM
100-JM requested review from a team as code owners September 11, 2026 10:45
@github-actions

Copy link
Copy Markdown
Contributor

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
@100-JM
You can retrigger this bot by commenting recheck in this Pull Request

@100-JM

100-JM commented Sep 12, 2026

Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant