Skip to content

Commit 98db09e

Browse files
committed
fix: model_id claude-sonnet-4-6 (no date suffix) + drop top_p (Sonnet 4.6 rejects both)
Two Anthropic API contract bugs surfaced by today's smoke after the auth + env-override fixes unblocked the path: 1. model_id "claude-sonnet-4-6-20260131" doesn't exist. /v1/models confirmed the current frontier IDs use no date suffix: opus-4-7, sonnet-4-6, opus-4-6. Only older models (sonnet-4-5-20250929 etc.) are dated. Updated endpoints.json + smoke.yaml + mve.yaml. 2. Sonnet 4.6+ returns 400 if both `temperature` and `top_p` are sent. We were sending temperature=0.0 + top_p=1.0 on every call. With temperature=0 the top_p is a no-op anyway; removed it from the API call AND from SamplingParams (recording top_p=1.0 in the Trial row while the API used Anthropic's internal default would be a reproducibility lie). After both fixes the first 5-trial smoke completed cleanly: 129 real MCP tool calls dispatched, $0.9052 spent, all 5 Trial rows valid (error_type=agent_gave_up — expected, git_mcp has no mounted volume).
1 parent d9d880f commit 98db09e

6 files changed

Lines changed: 23 additions & 7 deletions

File tree

CHANGELOG.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,17 @@ All notable changes to tool-crowding are documented here. Format follows [Keep a
44

55
## [Unreleased]
66

7+
### Fixed — model_id `claude-sonnet-4-6-20260131` did not exist; corrected to `claude-sonnet-4-6` (2026-05-26)
8+
9+
- `models/endpoints.json` + `configs/mve.yaml` + `configs/smoke.yaml` pinned the model as `claude-sonnet-4-6-20260131`. Anthropic returned 404 `not_found_error: model: claude-sonnet-4-6-20260131`. Probe of `https://api.anthropic.com/v1/models` confirmed the correct identifier is `claude-sonnet-4-6` (no date suffix). Sonnet 4.6 has not yet been promoted to a dated snapshot — older models like `claude-sonnet-4-5-20250929` use the dated form, current frontier models don't. Surfaced 2026-05-26 by the first live smoke after the auth + env-override fixes unblocked the path.
10+
- Caveat for reproducibility: `claude-sonnet-4-6` may be a floating alias rather than a frozen snapshot. The run_id chain captures the model_id string verbatim, so if Anthropic later releases `claude-sonnet-4-6-<date>`, our captured snapshot is the un-dated form. Migrate to the dated ID when one is published.
11+
12+
### Fixed — removed `top_p` from API calls + SamplingParams (Sonnet 4.6+ rejects both `temperature` and `top_p` together) (2026-05-26)
13+
14+
- `tcrun/agent.py::_invoke_api` sent both `temperature=0.0` and `top_p=1.0` on every API call. Sonnet 4.6 returns 400 `invalid_request_error: temperature and top_p cannot both be specified for this model`. Surfaced immediately after the model_id fix landed.
15+
- With `temperature=0.0` (our deterministic-dispatch default per pre-registration), `top_p` is a no-op anyway. Removed the `top_p` kwarg from the `client.messages.create` call. Removed the `top_p` field from `SamplingParams` (`tcrun/results.py`) so Trial rows don't record a value we never sent — recording `top_p=1.0` while the API used Anthropic's own internal default would be a reproducibility lie.
16+
- No new tests; the existing smoke + 257-test suite covers the schema migration (no tests referenced `SamplingParams.top_p`; the field was effectively dead code).
17+
718
### Fixed — `.env` now overrides shell-exported credentials, with a loud warning on drift (2026-05-26)
819

920
- `tcrun/cli.py::_load_env` previously called `load_dotenv(dotenv_path=env_path)` with `override=False` (python-dotenv's default). A stale `export ANTHROPIC_API_KEY=...` line in the user's shell rc (`~/.zshrc`, `~/.bashrc`) would silently shadow the value in `.env` — every subprocess inherits the shell env, dotenv refuses to overwrite, and Anthropic returns 401 against a key the dashboard swears is live. Surfaced 2026-05-26: a freshly-rotated key in `harness/.env` produced 401s for 30 minutes of debugging because `~/.zshrc:11` exported a long-revoked key into every shell. The harness's reproducibility chain depends on `.env` being the source of truth for credentials — silent shell-leak means two machines with different `.zshrc` files would produce different `run_id`s for the same checked-in config.

harness/configs/mve.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ distractors:
3535

3636
N: [1, 5, 10]
3737
runs_per_cell: 3
38-
model: claude-sonnet-4-6-20260131
38+
model: claude-sonnet-4-6
3939
host: claude-desktop
4040
seed: 42
4141

harness/configs/smoke.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ distractors: []
3030

3131
N: [1]
3232
runs_per_cell: 1
33-
model: claude-sonnet-4-6-20260131
33+
model: claude-sonnet-4-6
3434
host: claude-desktop
3535
seed: 42
3636

harness/models/endpoints.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,10 @@
22
"schema_version": "1.0",
33
"models": [
44
{
5-
"model_id": "claude-sonnet-4-6-20260131",
5+
"model_id": "claude-sonnet-4-6",
66
"provider": "anthropic",
77
"api_url": "https://api.anthropic.com/v1/messages",
8-
"checkpoint_identifier": "claude-sonnet-4-6-20260131",
8+
"checkpoint_identifier": "claude-sonnet-4-6",
99
"default_temperature": 0.0,
1010
"default_max_tokens": 4096,
1111
"system_prefix_template": "You are a code-retrieval assistant. [nonce: {nonce}]",

harness/tcrun/agent.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -643,7 +643,6 @@ async def _invoke_api(
643643
model=inputs.model_snapshot_id,
644644
max_tokens=inputs.sampling_params.max_tokens,
645645
temperature=inputs.sampling_params.temperature,
646-
top_p=inputs.sampling_params.top_p,
647646
system=system_prompt,
648647
messages=messages,
649648
tools=tools,

harness/tcrun/results.py

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,10 +77,16 @@ class ServerEntry(BaseModel):
7777

7878

7979
class SamplingParams(BaseModel):
80-
"""Model sampling parameters (SPEC.md §4)."""
80+
"""Model sampling parameters (SPEC.md §4).
81+
82+
`top_p` was removed 2026-05-26: Sonnet 4.6+ rejects requests that specify
83+
both `temperature` and `top_p` (400 invalid_request_error). With
84+
`temperature=0.0` (our deterministic default), `top_p` has no observable
85+
effect anyway. Recording a `top_p` value in the Trial row that we never
86+
actually sent to the API would mislead downstream reproductions.
87+
"""
8188

8289
temperature: float = 0.0
83-
top_p: float = 1.0
8490
max_tokens: int = 4096
8591

8692

0 commit comments

Comments
 (0)