sdk(llm): Stop model-name whack-a-mole: revert to core family substring matching #879
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…normalize usage - model_matches now does case-insensitive substring on full raw model - strip trailing '*' in patterns (migration aid) - pattern tables converted to plain substrings (no '*') - drop normalize_model_name and related tests - update tests to reflect substring semantics and Bedrock coverage Fixes #844 Co-authored-by: openhands <[email protected]>
Coverage Report •
|
…handling and empty-token skipping - Patterns are now used exactly as provided (lowercased/stripped) - No special handling for '*' or empty tokens Co-authored-by: openhands <[email protected]>
…eature detection - Validate provider-prefixed Bedrock ids and plain vendor-prefixed names - Ensure function-calling and prompt-cache features are enabled for Claude families Co-authored-by: openhands <[email protected]>
…edrock dotted vendor prefixes - Function-calling: adds claude-sonnet-4-5 and claude-sonnet-4.5, and us.anthropic.* examples - Prompt cache: keep only supported families; drop unsupported haiku-4.5 dotted vendor case Co-authored-by: openhands <[email protected]>
… extend tests with dotted vendor forms - Add claude-haiku-4.5 and claude-haiku-4-5 to PROMPT_CACHE_PATTERNS - Expand tests for us.anthropic.* and local names for Haiku 4.5 Co-authored-by: openhands <[email protected]>
enyst
added
the
integration-test
|
Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly. |
2 similar comments
|
Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly. |
|
Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly. |
🧪 Integration Tests ResultsOverall Success Rate: 0.0% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_deepseek_deepseek_chat
Failed Tests:
litellm_proxy_openai_gpt_5_mini
Failed Tests:
litellm_proxy_anthropic_claude_sonnet_4_5_20250929
Failed Tests:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
What & Why
Recent refactor introduced fnmatch-based globbing over a normalized basename. This unintentionally diverged from the prior V0 behavior where we effectively matched by substring on the full provider/model name. That change broke real-world cases, notably with AWS Bedrock where names embed dotted vendor prefixes and version suffixes inside the basename (e.g., bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0). Our patterns like 'claude-3-5-sonnet*' stopped matching after normalization and globbing.
This PR restores the durable invariant: if a meaningful family token (e.g., 'claude-3-5-sonnet', 'gpt-4o', 'o3', 'gemini-2.5-pro') appears anywhere in the model string, the feature applies. This eliminates the pattern maintenance whack-a-mole caused by dotted prefixes and provider-specific suffixes and aligns again with proven behavior in the wild.
Implementation Details
model_matches()
Pattern tables: remove '*'
normalize_model_name()
Tests
Outcomes
Checklist
Closes #844
@enyst can click here to continue refining the PR
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
golang:1.21-bookwormeclipse-temurin:17-jdknikolaik/python-nodejs:python3.12-nodejs22Pull (multi-arch manifest)
Run
All tags pushed for this build
The
60d1363tag is a multi-arch manifest (amd64/arm64); your client pulls the right arch automatically.Cross-repo impact: Fix: OpenHands/OpenHands#11248