Grammar-constrained tool calls

[M-ballabio1]

The README lists this as the research claim, and it is not built.

Today. Small models are asked politely to produce well-formed tool calls. When the arguments come back malformed, the result is a tool error the model can retry from. That works often enough to demo and not often enough to promise.

Wanted. Tool arguments constrained by grammar at decode time, so a malformed call is not possible rather than recoverable. This is what makes the claim falsifiable: on a small local model, the tool-call error rate should go to zero, not down.

Notes. docs/reference/typing-debt.md points out that Tool.execute has a dishonest base signature and that the fix is most likely a typed per-tool argument model — which is the same artefact grammar compilation needs. The two should probably land together.

Source: README, "What is built, and what is not".

[M-ballabio1]

Before any grammar work, this claim needs a baseline, and right now the code cannot produce one.

The promise here is falsifiable in a specific way — "the tool-call error rate should go to zero, not down". That is only checkable against a measured before. There is a place to measure it: tests/test_live_local_model.py already runs the real loop against a real Ollama model, defaulting to qwen2.5:3b and parameterised by ANNONA_LIVE_MODEL, so 3b / 14b / a frontier substrate is a matter of environment rather than new scaffolding.

The obstacle is that the loop cannot currently tell malformed apart from refused. runner/agent/loop.py:216-226 records every outcome through one flag:

result = self._executor.invoke(call) if self._gate.permits(call) else _denial(call)
invocations.append(ToolInvocation(..., error=result.is_error))

So error=True means any of three different things: the perimeter denied the call, the tool ran and failed, or the model produced arguments the tool could not accept. Only the third is what grammars fix. Measured as it stands, a policy that denies a lot would look like a model that cannot form a call, and a grammar that worked perfectly would move a number that is mostly made of denials.

So step zero is a distinction, not a feature: separate denied from failed from malformed in ToolInvocation. Small, useful on its own — it is also the difference an operator wants when a run went nowhere, and today annona why cannot tell them which of the three happened.

Then the order is:

1. Split the three outcomes.
2. Publish the baseline malformed-call rate per model. It will be embarrassing for 3b and that is the point; the interesting number is the gap between 3b, 14b and a frontier model, because the claim is that grammars close it.
3. Typing debt: Tool.execute has a dishonest base signature #8 — a typed per-tool argument model. That is the schema grammar compilation compiles.
4. Constrained decoding against it, and the same measurement re-run.

Steps 1 and 2 are worth doing even if nobody ever writes step 4: they turn "small models are bad at tool use" from folklore into a number this project measured, which is the same move #4 makes for the leak rate.

On step 4, note the substrate ceiling: GBNF-style constrained decoding is available where we control the decoder — Ollama and vLLM — and is not available uniformly across the frontier kind: anthropic / kind: openai-compatible backends, which offer their own constrained-output mechanisms with different semantics. The claim only needs the local case to hold, since that is where the gap being closed lives, but the placement layer should not pretend the guarantee is universal — a run routed to a frontier substrate gets whatever that provider offers, and the ledger should not imply otherwise.

Kept help wanted: steps 1 and 2 are a genuinely good entry point for someone who wants to work on the research claim without needing decoder internals.