Tool calling with small local models: why the Qwen3.5-9B failed a task built for the LFM2.5-230M (and the thought field fix)

I was trying to make the LFM2.5-230M do one task: fetch the weather in two cities. It failed.

So I started reworking the context, trying to explain the failure. The sequence was:

1. The original tool-calling format. Failed.
2. A reworked context-building phase. Failed.
3. A two-step approach (pick a tool, then fill it). Failed.

At that point I tested other models as a sanity check. The Qwen3.5-9B failed too. That was the eureka moment. My note at the time, verbatim: "Only huge model got that right. Even Qwen 3.5 9b failed, this is strange."

A 9B model failing a task I was trying to give a 230M model is not a model problem. It is a harness problem. So I scrapped everything, went back to the native conversation format, used the bigger models as a baseline (they solved it), and worked my way down to smaller and smaller models.

That is where this note comes from.

The task

"What's the weather in Paris and New York?"

It looks trivial. It is not. It is the minimal dual-op probe: two sequential tool calls with different arguments, then a synthesized final answer. It isolates multi-step agency from single-call correctness. A model can be excellent at "make ONE correct tool call" and still be unable to "notice what step 1 did and choose step 2."

Three local tools: CityWeather, Calculator, ConvertTemperature, plus a FinalAnswer action. Every model ran through llama.cpp on GGUF, greedy decoding, same task, same tools.

What I tested

Three architectures, compared on the same task:

	Architecture
A	Two-phase progressive disclosure + rebuilt plain-text history + full guardrails (my LFM2.5-230M agent)
B	Bare loop: one forced union call, native conversation history, no guardrails
C	Bare + a thought field (constrained chain-of-thought)

The model zoo: LFM2.5-230M, Qwen3.5-0.8B, LFM2.5-2.6B, Gemma-4-e2b-it, Qwen3.5-9B, Ministral-3-14B, LFM2.5-8B-A1B, Qwen3.8-27B, Qwen3.6-35B-A3B.

Finding 1: format beats size

The rebuilt plain-text history in architecture A silently destroyed multi-step competence in my mid-size models. Same task, same tools, same decoding:

Model	A (rebuilt context)	B (native format)
Qwen3.5-9B	❌ repeats Paris, disclaimer answer	✅ Paris → New York → full answer, 3 turns
Ministral-3-14B	❌ repeats Paris	✅ solved
Qwen3.8-27B	✅ solved	(expected ✅)
LFM2.5-2.6B / LFM2.5-230M	❌	❌ (capability wall either way)

The Qwen3.5-9B and Ministral-3-14B failures were not capability walls. They were format walls.

Instruct models are trained on standard tool-loop conversations. When I replaced that history with a "cleaner" plain-text digest, I moved them out of distribution. They could read a normal conversation and infer "next: New York." They could not do it from my digest, which resembled nothing in their training mix.

The rule I now follow: manipulate what is shown (which schemas, which enum values, which feedback lines), never the conversational skeleton itself.

Finding 2: reasoning inside the grammar

Chain-of-thought is not new. Kojima et al. showed in 2022 that a model reasons better when asked to "think step by step" before answering (Large Language Models are Zero-Shot Reasoners). What is not obvious is that you can trigger that reasoning while the output stays 100% grammar-constrained.

With a forced tool call, the grammar masks everything except the JSON, so the model has literally no room to reason before acting. Small and mid-size models then fall back on pattern-copying: repeat the last call that worked.

The finding here is that triggering reasoning while keeping constrained grammar sampling is possible, using a thought field. Put thought first in the step schema:

class AgentStep(msgspec.Struct):
    thought: str   # "what the history already provides, what is still missing"
    call: CityWeather | Calculator | ConvertTemperature | FinalAnswer

JSON key order is generation order. So the reasoning is generated before the action, and it stays inside the grammar. The thought value is free text (the one unconstrained part), but the call that follows is still masked to the schema. You get chain-of-thought and guaranteed-valid JSON in the same forward pass.

The measured impact:

Model	without thought	with thought
LFM2.5-2.6B	infinite repeat of Paris	solved, 3 turns
Qwen3.5-0.8B	failed	solved, 3 turns
Ministral-3-14B	failed (under A)	solved, 3 turns
Qwen3.5-9B	solved	solved, cleaner reasoning
LFM2.5-230M	loops	verbalizes the plan, still loops

The turn-2 thought is the mechanism: "I've already retrieved Paris… now I need New York" → correct action. The reasoning does the bookkeeping the model was missing, inside the same forward pass. It also breaks repetition loops, because the reasoning prefix varies the continuation.

This is the single highest-leverage addition from 0.8B up. It lowered the floor to 0.8B.

Finding 3: below ~2.6B, mechanics beat instructions

The LFM2.5-230M ignores every sentence I write. It violated an explicit one-line output protocol every turn. When I challenged it, it confabulated an explanation. Correction hints ("fix the arguments, choose a different tool") got the identical failed call repeated 5× in a row.

But the same model obeys instantly when the fix is mechanical:

• Remove the failed tool from the enum → different choice.
• city: Literal["Paris","Tokyo","New York","London","Sydney"] → "Londres" becomes "London" at sampling time. The invalid string is unsampleable.

So below ~2.6B, every guarantee has to come from the grammar, the enum contents, the validator, or the loop controller, never from a sentence hoping the model complies. The guardrail stack that works:

1. Grammar enums for every enumerable domain
2. FinalAnswer removed from the enum until ≥1 tool succeeded
3. A tool with ≥2 failures is banned from the enum (mechanical loop exit)
4. Provenance validation of numbers (below)
5. Repeat handling: 2nd repeat of a succeeded call → forced final
6. Scope caps: single/dual ops only
7. Honest failure: max turns exhausted → exit 1, never an invented answer

Above ~8B, most of this tax disappears. The Qwen3.5-9B needs none of it in the native format.

What grammar constraints buy you (and what they don't)

llama.cpp compiles a JSON Schema into a GBNF grammar and masks every token that would leave the schema, at each sampling step. The output is guaranteed parseable. No repair loops, no "please output valid JSON."

What it does not buy you:

• It masks; it does not choose well. Which enum value it picks is still model behavior.
• It leaves zero room for reasoning outside the schema (hence the thought field).

And it has silent traps. The one that is easy to miss: tool_choice="auto" applies no grammar and parses no tool calls. The "most basic grammar-constrained function calling" in llama.cpp is therefore a single forced union tool, not the OpenAI-style auto loop.

Success is not the same as grounded

A small model will call a perfectly-shaped tool with a fabricated argument. I watched the LFM2.5-230M call ConvertTemperature(value=25.0) where 25.0 appears nowhere in the task or any tool result. The tool "succeeded," which unlocked FinalAnswer, and the agent produced a fluent answer grounded in a hallucinated input.

The fix: keep a set of every legitimate number, seeded from the user request, grown by every successful tool result, and reject anything ungrounded. The grammar guarantees shape. Provenance guarantees the numbers are real.

One detail worth keeping: the provenance error message is the one textual message that steered the LFM2.5-230M. "Use CityWeather to fetch a city temperature first" → its next pick was CityWeather(New York). Text fails as a rule. It works as tool feedback, because it arrives in-distribution, an API telling you what's missing, instead of as a nag.

The loop: same context, same output, forever

Under greedy decoding, the output is a pure function of the rendered prompt. If a turn changes nothing in the context, the next turn is byte-identical and produces the identical output. I proved it with a prompt spy: prompts #4 and #6 were identical (154 tokens each), and both outputs were identical, turn after turn.

So any loop-breaking measure must change the context or the grammar, mechanically. A "retry" that appends nothing is dead code, an infinite loop wearing a for-loop disguise.

The design rule: before adding any retry path, ask what is different in the next prompt? If the answer is "nothing," don't write it.

How I debug: the prompt spy

With chat-template bindings you never see what the model actually receives. The template injects the tool list, renders history its own way, and may silently drop or mangle parts. Every "why did it do that?" in this project was answered by looking at the rendered string, not the messages list.

The technique: intercept create_completion and detokenize the exact token list with special tokens visible.

def install_prompt_logger(llm):
    original = llm.create_completion
    def spy(prompt, **kwargs):
        tokens = prompt if isinstance(prompt, list) else llm.tokenize(prompt.encode(), add_bos=True, special=True)
        text = llm.detokenize(tokens, special=True).decode("utf-8", errors="replace")
        print(f"PROMPT ({len(tokens)} tokens)\n{text}")
        return original(prompt, **kwargs)
    setattr(llm, "create_completion", spy)

It found the schema duplication that bloated my prompts from 782 to 366 tokens, the exact trained format, the loop proof, and the token budgets. Reading the raw prompt is the single highest-yield debugging habit for local-model agents.

The capability wall, measured

The LFM2.5-230M's own benchmarks predict the shape of everything:

Benchmark	Score	Reading
BFCLv3 (single call correctness)	43.26	single-shot tool calls are a strength
τ²-Bench (multi-turn agency)	5.26	multi-step "what next" is near zero

Our task-level results track it exactly. Where the wall sits:

Band	Dual-op	What works
230M	❌ never	full guardrail stack → grounded partial answer + honest failure
0.8B–2.6B	family-dependent	constrained CoT, when the family has tool grounding
9B–14B	✅ native format	standard loop + optional CoT
27B+	✅ format-robust	anything reasonable

Size bands are a heuristic, not a law. The Qwen3.5-0.8B outperformed the Gemma-4-e2b-it on this probe. One caveat I have not closed: every run used greedy decoding, but Gemma recommends temperature 1.0 / top_p 0.95 / top_k 64 and is documented to degrade under greedy. Its failure, inventing a hypothetical weather_api and answering with a statement of intent, may be partly a sampling artifact. I am retesting before I call it a wall.

What I would ship now

For a local demo or an on-device agent, the default is C: native conversation format + a constrained thought field. Add the provenance check and honest failure when the model is small or unproven. Skip the two-phase disclosure and the rebuilt context, they doubled my calls per turn and cost me the Qwen3.5-9B / Ministral-3-14B band.

Scope small-model agents to what the benchmarks certify: single (or guarded dual) operations. Spend the complexity budget on mechanics, enums, bans, provenance, forced finals, not on prompts.

Why this matters (for small teams)

Current agent harnesses are built for big cloud models. They assume the model will follow instructions, recover from a bad tool call, and keep a long context coherent. None of that holds below 1B parameters.

Sub-1B LLMs need a custom harness. And building one is where the interesting engineering is: the model will not obey a sentence, so every guarantee has to be mechanical (enums, bans, provenance, forced finals). The context has to stay in the format the model was trained on. The loop has to fail honestly instead of looping forever. These are real design problems, and they do not show up in the cloud-model benchmarks.

Fine-tuning is the other lever, and the obvious next step, but that is a note for another day.

FAQ

Does a small model need guardrails?

Below ~2.6B, yes, the full mechanical stack. At 9B+ in the native format, no. The thought field and provenance are the only two that stay useful across the board.

Is grammar-constrained generation enough?

No. It guarantees shape, not truth, and it leaves no room to reason. Pair it with a thought field and provenance validation.

Can I just use tool_choice="auto"?

In llama.cpp, no, it applies no grammar and parses no tool calls. Force a single union tool.

Why did the Qwen3.5-9B fail?

Out-of-distribution context. I replaced the native tool-loop history with a plain-text digest. Native format fixed it in one change.

Read more

• How to run OpenCode with LM Studio locally (Qwen3.5 on a 6-year-old MacBook M1)
• CRM migration with local coding agents: why the M5 Max MacBook Pro matters
• The New Hires Never Sleep