Skip to content

Pin anthropic<1.0 in the DeepSeek-R1 evaluation requirements - #2669

Open
100-JM wants to merge 1 commit into
mlcommons:masterfrom
100-JM:fix/deepseek-r1-pin-anthropic
Open

Pin anthropic<1.0 in the DeepSeek-R1 evaluation requirements#2669
100-JM wants to merge 1 commit into
mlcommons:masterfrom
100-JM:fix/deepseek-r1-pin-anthropic

Conversation

@100-JM

@100-JM 100-JM commented Sep 11, 2026

Copy link
Copy Markdown

Problem

eval_accuracy.py imports lcb_runner.runner.scenario_router, which pulls in lcb_runner/prompts/__init__.py, which imports every prompt module — including test_output_prediction.py, which does from anthropic import HUMAN_PROMPT, AI_PROMPT at module level. Those two constants belong to the pre-Messages Text Completions API and were removed in anthropic 1.0.0 (last release that has them: 0.125.0).

LiveCodeBench's pyproject.toml only declares anthropic>=0.42.0, so a fresh install of docker/evaluation_requirements.txt today resolves to a 1.x SDK and the grader dies before evaluating a single sample:

eval_accuracy.py:496  from lcb_runner.runner.scenario_router import build_prompt_benchmark
  └ lcb_runner/prompts/__init__.py:3
      └ lcb_runner/prompts/test_output_prediction.py:3
          from anthropic import HUMAN_PROMPT, AI_PROMPT
ImportError: cannot import name 'HUMAN_PROMPT' from 'anthropic'

We only use LiveCodeBench for grading, never for generation, so the dependency is incidental — but it is on the import path. gpt-oss-120b/requirements.txt already pins anthropic==0.72.0, which is why that benchmark's grader survived; deepseek-r1 had no pin.

Change

Add anthropic<1.0 to language/deepseek-r1/docker/evaluation_requirements.txt with a comment explaining why. A range pin rather than an exact one so it keeps resolving alongside whatever else lands in the evaluation venv; happy to switch to ==0.72.0 to match gpt-oss-120b exactly if that is preferred.

Observed on the affected host: with anthropic 1.5.0 the import above fails; with a 0.x SDK the constants exist and load_lcb_benchmark() reports Loaded 400 problems.

🤖 Generated with Claude Code

LiveCodeBench's prompt package imports anthropic.HUMAN_PROMPT and
AI_PROMPT at module import time, and lcb_runner/prompts/__init__.py
imports every prompt module, so the constants are required even though
eval_accuracy.py only uses LiveCodeBench for grading. Both constants
were removed in anthropic 1.0.0 (they belong to the pre-Messages Text
Completions API). LiveCodeBench declares anthropic>=0.42.0 with no
upper bound, so a fresh evaluation_requirements.txt install today
resolves to a 1.x SDK and load_lcb_benchmark() fails with

    ImportError: cannot import name 'HUMAN_PROMPT' from 'anthropic'

before a single LiveCodeBench sample is graded. The gpt-oss-120b
reference already pins anthropic (0.72.0) for the same reason; this
gives deepseek-r1 the same protection with a range pin.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@100-JM
100-JM requested review from a team as code owners September 11, 2026 10:45
@github-actions

Copy link
Copy Markdown
Contributor

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
@100-JM
You can retrigger this bot by commenting recheck in this Pull Request

@100-JM

100-JM commented Sep 12, 2026

Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant