PAL-BUG-3 (NEW W23): `qwen/qwen-2.5-coder-32b-instruct` returns content but OR `/v1/generation/{id}` returns all-null fields

[primovera12]

Reporter: rabih@ultratms.com (via /eval-models routing review W23)
First observed: 2026-W22 + 2026-W23 — 2 consecutive weeks
Severity: medium-high (model is in 3 picks-table rows; swap already applied this week)
Likely venue: OpenRouter issue tracker, NOT PAL — but capturing here because it surfaced through PAL eval flow

TL;DR

Calls to qwen/qwen-2.5-coder-32b-instruct via OpenRouter exhibit two related anomalies:

1. Premature truncation under codereview-style prompts. Output ends mid-token with finish_reason: null AND native_finish_reason: null. 2 consecutive weekly runs (W22: 57 tokens, 1/6 issues; W23: 97 tokens, 1/6 issues). Same fixture both weeks.
2. All-null OR /v1/generation/{id} response under precommit-style prompts. PAL response contains valid 456-token content with 4/6 issues identified; OpenRouter's API for the same gen-id returns {"model": null, "tokens_prompt": null, "tokens_completion": null, "total_cost": null, "generation_time": null, "finish_reason": null} even after 20-second wait for eventual consistency.

Both shapes suggest a provider-routing issue specific to this slug — possibly one OpenRouter provider in the routing pool is returning malformed or incomplete metadata back to OpenRouter.

Reproduction A — codereview truncation

Fixture: ~/.claude/openrouter-models/test-fixtures/codereview-buggy.ts (116-line TS, 6 planted issues).
Prompt: "Focus on bugs, security, perf. Skip style. Format: file:line — issue — fix. One row per issue."
Call: pal__chat with model=qwen/qwen-2.5-coder-32b-instruct, temperature=0.2.

Week	gen-id	Output tokens	Issues found	finish_reason	native_finish_reason
W22	gen-1779687xxx (from W22 report)	57	1/6	null	(not recorded)
W23	gen-1779733527-doXyUhFdAIBQsJ9SqS9E	97	1/6 (truncated mid-issue-2 at "file:4")	null	null

Both runs returned actual content (output was syntactically valid prose, just cut off mid-sentence) but with no completion signal.

Reproduction B — null OR metadata

Fixture: same codereview-buggy.ts.
Prompt: precommit-style "Flag blockers only" (different system-prompt-modifier, max_tokens=2048).
Call: pal__chat with model=qwen/qwen-2.5-coder-32b-instruct, temperature=0.2.
gen-id: gen-1779735114-PJPfstG1DnH7d7LXGC9v (epoch 2026-05-25T15:51:54Z)

PAL response: valid content, 4 issues identified, normal MCP response shape, content_type="text", model_used="qwen/qwen-2.5-coder-32b-instruct".

OpenRouter /v1/generation/{id} lookup (immediate + 6s retry + 20s retry):

{
  "model": null,
  "tokens_prompt": null,
  "tokens_completion": null,
  "total_cost": null,
  "generation_time": null,
  "finish_reason": null,
  "native_finish_reason": null,
  "error": null
}

Same symptom shape as W22 PAL-BUG-2 case (pal__analyze empty_response) but here the PAL response is intact — only the OR metadata is missing. The W22 PAL-BUG-2 case had both PAL empty + OR null. This case has PAL OK + OR null.

Cross-pattern observation

Out of 51 unique models tested in the W23 routing review, only qwen/qwen-2.5-coder-32b-instruct exhibited any of these anomalies. All 50 other models — including other Qwen 2.5 family members, other 32B-class models, and other code-specialized models from competing providers — returned normal finish_reason: "stop" and complete OR metadata.

The pattern is slug-specific, not size-specific or family-specific.

Hypothesized cause

OpenRouter's provider-routing pool for qwen/qwen-2.5-coder-32b-instruct likely has at least one provider that:

• Returns content prematurely (no [DONE] marker on the stream)
• Fails to report completion metadata back to OpenRouter's billing/telemetry pipeline

The intermittent vs systematic split between reproductions A and B may reflect different OR routing decisions for the two prompt shapes (different max_tokens, different system_prompt budget).

Recommended action

1. For Master-Kit picks-table: swap applied immediately — codereview + precommit rotated to x-ai/grok-build-0.1. Done in ~/.claude/openrouter-models/current-picks.md 2026-05-25.
2. For OpenRouter: file an issue including both gen-ids and the all-null metadata response. Ask which provider in the routing pool serves this slug — they may need to be removed from rotation or flagged for upstream investigation.
3. For PAL maintainers: consider adding a sanity check — when OR /v1/generation/{id} returns all-null fields for a gen-id whose PAL response was non-empty, log a warning. This would have surfaced this bug a week earlier.

Cross-references

• W22 routing review: ~/.claude/openrouter-models/results/2026-W22.md (codereview truncation noted as F3)
• W23 routing review: ~/.claude/openrouter-models/results/2026-W23.md Section A1 + A7, Section B (PAL-BUG-3)
• Picks-table state: ~/.claude/openrouter-models/current-picks.md (post-W23 swap)

[miamilabs]

Looks liek the project is dead. No maintanance anymore. I did implement new models and fixes by myself