Skip to content

llama3.1-8b: fix run scripts and the README accuracy flow - #2671

Open
100-JM wants to merge 1 commit into
mlcommons:masterfrom
100-JM:fix/llama3.1-8b-scripts-readme
Open

llama3.1-8b: fix run scripts and the README accuracy flow#2671
100-JM wants to merge 1 commit into
mlcommons:masterfrom
100-JM:fix/llama3.1-8b-scripts-readme

Conversation

@100-JM

@100-JM 100-JM commented Sep 11, 2026

Copy link
Copy Markdown

Problem

Following the llama3.1-8b README / helper scripts for an accuracy run fails before any inference happens, for three independent reasons:

  1. ${VAR:default} instead of ${VAR:-default} in run_accuracy.sh, run_offline.sh, run_server.sh. : without - is substring expansion, so with the variables unset CHECKPOINT_PATH and DATASET_PATH silently become empty strings and the scripts run with --model-path "" --dataset-path "". run_offline.sh/run_server.sh also pass --tensor-parallel-size "${GPU_COUNT}" with no default, which argparse rejects when GPU_COUNT is unset.
  2. run_accuracy.sh passes --mlperf-conf mlperf.conf, but main.py has no such argument (loadgen loads mlperf.conf itself; the FromConfig call is commented out), so the script dies in argparse.
  3. README accuracy examples set OUTPUT_LOG_DIR=… but pass --output-log-dir output, then read ${OUTPUT_LOG_DIR}/mlperf_log_accuracy.json. That file never exists, so the if [ -e … ] guard silently skips the evaluation step and the user gets no accuracy number and no error.

Changes

  • :- defaults in all three scripts; GPU_COUNT defaults to 1.
  • Drop --mlperf-conf from run_accuracy.sh.
  • README: pass --output-log-dir ${OUTPUT_LOG_DIR} in the three accuracy examples (Offline, Server, Edge).
  • README: one paragraph on evaluation.py --model-name, which defaults to the gated meta-llama/… Hub repo for the tokenizer and therefore fails with 401 Unauthorized after inference when running from a local checkpoint without Hub access; pass the same path as --model-path.

Verification

  • bash -n on the three scripts.
  • With the variables unset, the new lines expand to meta-llama/Meta-Llama-3.1-8B-Instruct / cnn_eval.json / 1; the old form expands to "" (checked on bash 3.2).
  • The --model-name note reflects an actual run: inference completed, then evaluation.py failed on the gated repo until the local path was passed.

🤖 Generated with Claude Code

… documented

Three problems in the documented accuracy flow, each of which stops it
before any inference happens:

- run_accuracy.sh, run_offline.sh and run_server.sh used
  "${VAR:default}" (substring expansion) where "${VAR:-default}" was
  intended, so with the variables unset CHECKPOINT_PATH and DATASET_PATH
  silently expanded to empty strings and the scripts ran with
  --model-path "" --dataset-path "". run_offline.sh and run_server.sh
  also passed --tensor-parallel-size "${GPU_COUNT}" with no default,
  which argparse rejects when GPU_COUNT is unset; they now default to 1.
- run_accuracy.sh passed --mlperf-conf mlperf.conf. main.py has no such
  argument (loadgen loads mlperf.conf itself, and the FromConfig call is
  commented out), so the script died in argparse. The flag is removed.
- The README's three accuracy examples set OUTPUT_LOG_DIR but passed
  --output-log-dir output, then read
  ${OUTPUT_LOG_DIR}/mlperf_log_accuracy.json, which therefore never
  exists and the evaluation step is silently skipped. They now pass
  ${OUTPUT_LOG_DIR} through.

Also documents evaluation.py's --model-name: it defaults to the gated
meta-llama Hub repository for the tokenizer, so runs from a local
checkpoint fail with 401 after inference unless the local path is given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@100-JM
100-JM requested review from a team as code owners September 11, 2026 10:45
@github-actions

Copy link
Copy Markdown
Contributor

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
@100-JM
You can retrigger this bot by commenting recheck in this Pull Request

@100-JM

100-JM commented Sep 12, 2026

Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant