Skip to content

CI: change SGLang plugin M3 nightly accuracy threshold#1529

Open
Phi-C wants to merge 3 commits into
ROCm:mainfrom
Phi-C:m3_ci_acc
Open

CI: change SGLang plugin M3 nightly accuracy threshold#1529
Phi-C wants to merge 3 commits into
ROCm:mainfrom
Phi-C:m3_ci_acc

Conversation

@Phi-C

@Phi-C Phi-C commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Motivation

For nightly gsm8k accuracy validation, its configuration is

lm_eval \
  --model local-completions \
  --model_args "model=amd/MiniMax-M3-MXFP4,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=65,max_retries=1,tokenized_requests=False,trust_remote_code=True" \
  --tasks gsm8k \
  --num_fewshot 5

which is different from configurations in recipe

BS=65

lm_eval \
    --model local-chat-completions \
    --model_args "model=${model_path},base_url=http://127.0.0.1:${PORT}/v1/chat/completions,num_concurrent=32,max_gen_toks=16384" \
    --tasks gsm8k \
    --num_fewshot 5 \
    --batch_size "${BS}" \
    --apply_chat_template \
    --fewshot_as_multiturn 2>&1 | tee minimax-m3-mxfp4-sglang-gsm8k.log

So, the accuracy threshold is changed to 0.88.

Technical Details

Test Plan

Test Result

Submission Checklist

Signed-off-by: Phi-C <chenxjhit@163.com>
@Phi-C Phi-C requested a review from zhuyuhua-v July 9, 2026 10:28
@zhuyuhua-v

Copy link
Copy Markdown
Collaborator

please refer atom's m3 acc test command and threshold, instead of lowering ours..

Phi-C added 2 commits July 10, 2026 00:41
Signed-off-by: Phi-C <chenxjhit@163.com>
@Phi-C

Phi-C commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

please refer atom's m3 acc test command and threshold, instead of lowering ours..

Okay, the test command is aligned with atom's m3 acc test now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants