Pin anthropic<1.0 in the DeepSeek-R1 evaluation requirements - #2669
Open
100-JM wants to merge 1 commit into
Open
Pin anthropic<1.0 in the DeepSeek-R1 evaluation requirements#2669100-JM wants to merge 1 commit into
100-JM wants to merge 1 commit into
Conversation
LiveCodeBench's prompt package imports anthropic.HUMAN_PROMPT and
AI_PROMPT at module import time, and lcb_runner/prompts/__init__.py
imports every prompt module, so the constants are required even though
eval_accuracy.py only uses LiveCodeBench for grading. Both constants
were removed in anthropic 1.0.0 (they belong to the pre-Messages Text
Completions API). LiveCodeBench declares anthropic>=0.42.0 with no
upper bound, so a fresh evaluation_requirements.txt install today
resolves to a 1.x SDK and load_lcb_benchmark() fails with
ImportError: cannot import name 'HUMAN_PROMPT' from 'anthropic'
before a single LiveCodeBench sample is graded. The gpt-oss-120b
reference already pins anthropic (0.72.0) for the same reason; this
gives deepseek-r1 the same protection with a range pin.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
|
MLCommons CLA bot: |
Author
|
recheck |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
eval_accuracy.pyimportslcb_runner.runner.scenario_router, which pulls inlcb_runner/prompts/__init__.py, which imports every prompt module — includingtest_output_prediction.py, which doesfrom anthropic import HUMAN_PROMPT, AI_PROMPTat module level. Those two constants belong to the pre-Messages Text Completions API and were removed inanthropic1.0.0 (last release that has them: 0.125.0).LiveCodeBench's
pyproject.tomlonly declaresanthropic>=0.42.0, so a fresh install ofdocker/evaluation_requirements.txttoday resolves to a 1.x SDK and the grader dies before evaluating a single sample:We only use LiveCodeBench for grading, never for generation, so the dependency is incidental — but it is on the import path.
gpt-oss-120b/requirements.txtalready pinsanthropic==0.72.0, which is why that benchmark's grader survived; deepseek-r1 had no pin.Change
Add
anthropic<1.0tolanguage/deepseek-r1/docker/evaluation_requirements.txtwith a comment explaining why. A range pin rather than an exact one so it keeps resolving alongside whatever else lands in the evaluation venv; happy to switch to==0.72.0to match gpt-oss-120b exactly if that is preferred.Observed on the affected host: with
anthropic1.5.0 the import above fails; with a 0.x SDK the constants exist andload_lcb_benchmark()reportsLoaded 400 problems.🤖 Generated with Claude Code