llama3.1-8b: fix run scripts and the README accuracy flow - #2671
Open
100-JM wants to merge 1 commit into
Open
Conversation
… documented
Three problems in the documented accuracy flow, each of which stops it
before any inference happens:
- run_accuracy.sh, run_offline.sh and run_server.sh used
"${VAR:default}" (substring expansion) where "${VAR:-default}" was
intended, so with the variables unset CHECKPOINT_PATH and DATASET_PATH
silently expanded to empty strings and the scripts ran with
--model-path "" --dataset-path "". run_offline.sh and run_server.sh
also passed --tensor-parallel-size "${GPU_COUNT}" with no default,
which argparse rejects when GPU_COUNT is unset; they now default to 1.
- run_accuracy.sh passed --mlperf-conf mlperf.conf. main.py has no such
argument (loadgen loads mlperf.conf itself, and the FromConfig call is
commented out), so the script died in argparse. The flag is removed.
- The README's three accuracy examples set OUTPUT_LOG_DIR but passed
--output-log-dir output, then read
${OUTPUT_LOG_DIR}/mlperf_log_accuracy.json, which therefore never
exists and the evaluation step is silently skipped. They now pass
${OUTPUT_LOG_DIR} through.
Also documents evaluation.py's --model-name: it defaults to the gated
meta-llama Hub repository for the tokenizer, so runs from a local
checkpoint fail with 401 after inference unless the local path is given.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
|
MLCommons CLA bot: |
Author
|
recheck |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Following the llama3.1-8b README / helper scripts for an accuracy run fails before any inference happens, for three independent reasons:
${VAR:default}instead of${VAR:-default}inrun_accuracy.sh,run_offline.sh,run_server.sh.:without-is substring expansion, so with the variables unsetCHECKPOINT_PATHandDATASET_PATHsilently become empty strings and the scripts run with--model-path "" --dataset-path "".run_offline.sh/run_server.shalso pass--tensor-parallel-size "${GPU_COUNT}"with no default, which argparse rejects whenGPU_COUNTis unset.run_accuracy.shpasses--mlperf-conf mlperf.conf, butmain.pyhas no such argument (loadgen loadsmlperf.confitself; theFromConfigcall is commented out), so the script dies in argparse.OUTPUT_LOG_DIR=…but pass--output-log-dir output, then read${OUTPUT_LOG_DIR}/mlperf_log_accuracy.json. That file never exists, so theif [ -e … ]guard silently skips the evaluation step and the user gets no accuracy number and no error.Changes
:-defaults in all three scripts;GPU_COUNTdefaults to 1.--mlperf-conffromrun_accuracy.sh.--output-log-dir ${OUTPUT_LOG_DIR}in the three accuracy examples (Offline, Server, Edge).evaluation.py --model-name, which defaults to the gatedmeta-llama/…Hub repo for the tokenizer and therefore fails with401 Unauthorizedafter inference when running from a local checkpoint without Hub access; pass the same path as--model-path.Verification
bash -non the three scripts.meta-llama/Meta-Llama-3.1-8B-Instruct/cnn_eval.json/1; the old form expands to""(checked on bash 3.2).--model-namenote reflects an actual run: inference completed, thenevaluation.pyfailed on the gated repo until the local path was passed.🤖 Generated with Claude Code