Default --audit-conf to audit.config in the reference entry points - #2672
Open
100-JM wants to merge 1 commit into
Open
Default --audit-conf to audit.config in the reference entry points#2672100-JM wants to merge 1 commit into
100-JM wants to merge 1 commit into
Conversation
…hipped file Seven reference entry points defaulted the LoadGen audit filename to "audit.conf", but every compliance test in compliance/ ships the file as audit.config (renamed repo-wide in mlcommons#2127), the TEST06 README tells submitters to copy audit.config into the run directory, and both the loadgen Python binding and text_to_image/main.py already default to audit.config. loadgen only applies the audit override when FileExists(name) is true, so with the old default a compliance run that followed the README ran as a plain performance run: no error, no "Found Audit Config file" line in mlperf_log_detail.txt, and a TEST06 verification that then fails or verifies the wrong logs. Aligning the defaults makes the documented procedure work without passing --audit-conf audit.config explicitly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
|
MLCommons CLA bot: |
Author
|
recheck |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Seven reference entry points default the LoadGen audit filename to
audit.conf:language/{llama3.1-8b,llama2-70b,llama3.1-405b,mixtral-8x7b,gpt-j}/main.py,language/bert/run.py,speech2text/reference_mlperf.pybut every compliance test under
compliance/ships the file asaudit.config(renamed repo-wide in #2127),compliance/TEST06/README.mdtells submitters to copyaudit.configinto the run directory, and both the loadgen Python binding (python_api.cc:audit_config_filename = "audit.config") andtext_to_image/main.pyalready default toaudit.config.loadgen only applies the override when
FileExists(audit_config_filename)(loadgen.cc:1232). So a compliance run that follows the README with one of these entry points runs as a plain performance run: no error, noFound Audit Config fileline inmlperf_log_detail.txt, and arun_verification.pythat then fails or verifies logs that were never produced under audit settings. We noticed this while setting up llama3.1-8b TEST06 — the README's advice to check for that log line is the only thing that exposes it.Change
default="audit.conf"→default="audit.config"in the seven files. Passing--audit-confexplicitly keeps working; this only changes what happens when the documented procedure is followed without it. No loadgen changes.autopep8 -a --max-line-length 79clean on all seven files.🤖 Generated with Claude Code