Appearance
Three local models, one 96-minute session: the 35B corrupted the file, the 27B fixed it, and the 9B distill shipped 70%
MiMo-V2.6-Distill-Qwen-9B is a 9B agentic model released by Xiaomi MiMo on Hugging Face. It is a supervised fine-tune of Qwen3.5-9B on MiMo-generated data, covering coding, general agent tasks, visual coding, and cybersecurity. Xiaomi released it as the SFT checkpoint from the MiMo-V2.6 technical report, and the published numbers are interesting for local work: relative to the base Qwen3.5-9B, the fine-tune lifts Terminal Bench 2.1 from 27.0 to 37.1 and SWE Pro from 32.0 to 44.6, while staying small enough to run on a MacBook.
I wanted to see how it behaves as a coding agent in a real setup, not on a benchmark, so I gave it a task with a nice property: I used the model itself to set itself up on another machine. The job was to wire MiMo into my llama.cpp model router and into OpenCode on this Mac, and then to replicate the whole llm-serve stack, including the model's own chat template and router entry, over SSH onto a Mac mini. The model did the setup work, and the target of the setup was the model's own deployment.
The session ran 96 minutes, and I ended up running three models on it in sequence: Qwen3.6-35B-A3B, then Qwen3.8-27B, then the 9B distill itself. I switched because the first model broke things, not because I was planning an experiment. That made it one anyway: same task, same repo, same human, three local models, and the results did not line up with size at all.
The task, precisely
For context, my daily driver for local models is a llama.cpp model router: a single llama-server process that loads several GGUFs at once and routes requests by alias, configured through an INI preset file. OpenCode talks to that one endpoint, so adding a model means adding a router entry, not standing up a new server. The task was:
- Add the MiMo GGUF to a llama.cpp
--models-presetINI router, with its chat template and multimodal projector. - Register it in OpenCode's llama.cpp provider.
- Replicate the entire llm-serve stack (router script, INI, templates, models) onto a Mac mini over SSH, and get the router serving there.
The task is mostly file copies, config edits, and work on one remote machine. The only subtle part is that the chat template is a Jinja file that must be byte-exact, because it is the serialization contract between the harness and the model.
The cast
| Model | Phase | Steps | Output tokens | My stops/corrections |
|---|---|---|---|---|
| Qwen3.6-35B-A3B (MoE) | Initial setup | 29 | ~13k | 3 |
| Qwen3.8-27B (dense) | Recovery + verification | 34 | ~13.5k | 1 |
| MiMo-9B (distill) | Config Q&A + remote replication | 60 | ~44k | 7 |
All three were served locally through llama.cpp, all through the same OpenCode session. The export records the model ID per message, so the phase boundaries are exact: I switched at the correction points, which means each phase starts with a model responding to a complaint about the previous model.
Finding 1: size did not predict reliability
When I sorted the three phases by what actually happened, the 35B MoE was the least reliable, the 27B dense was the most reliable, and the 9B distill sat in between. If you had sorted them by parameter count or benchmark score, you would have predicted the wrong order.
What the 35B did, in 29 steps:
- Created a JSON config I had explicitly declined. I wanted "just the ini." It heard a request and returned a project.
- Wrote the Jinja chat template four times from memory. Each rewrite "corrected" special tokens it had misremembered, and the file ended with corrupted placeholder tokens.
- Claimed, confidently, that the INI preset format "doesn't support
chat_template_file." That was false. The next model proved it the most expensive way possible: instead of reading the docs or running--help, it parsed llama.cpp'spreset.cppsource, where the key maps 1:1 to the CLI flag. The answer was one flag away, and the model paid for it in tokens and wall time. It got the right answer, which is the worst part. - After the fourth failed rewrite, it ended the turn mid-confusion instead of re-fetching the source.
- Later, it wrote
context: 128000in the OpenCode config, while the real value is 262144. It wrote the number with total confidence, which is how you know it had never checked.
When I switched from the 35B to the 27B, I nudged the new model with "i think you made mistakes, check", and it spent its 34 steps doing exactly that:
- Re-fetched the template from the source, compared, and stopped retyping. It ran
curlto a temp file andcpinto place, byte-exact. - Verified the template by rendering a real conversation through
llama-cliand inspecting the serialized prompt, and confirmed the thinking prefill was there. - Verified the preset by starting
llama-serverwith--no-models-autoloadon a scratch port and readingGET /modelsfor the resolved flags. - Updated the repo's AGENTS.md to document the new key.
- Zero tool errors.
What the 9B distill did, in 60 steps:
- Asked three clarifying questions before touching anything, the only one of the three that did, and then produced a clean phased plan.
- Executed the remote SSH work well once the path was explicit.
- Copied configs I did not ask for, probed the remote machine's
--helpinstead of copying the router script that already existed locally, hit the same scp-into-a-missing-directory failure three times, misread its own test output, and tried tocurl | shsoftware that was already installed. That is the agentic equivalent of re-keying a lock that already opens. - 11 of its 60 turns (18%) stalled and needed a stop or a re-prompt.
Finding 2: retyping is a failure mode
The template corruption is the incident of the session, and it is worth dissecting, because it is not a "small model hallucinated" story. It is a 35B model.
The sequence was:
- The model was asked to add the template to the repo. It did not fetch the file. It wrote it from memory.
- I checked. A placeholder token was wrong. It rewrote the whole file from memory, "correcting" it.
- Another token was wrong. Rewrite number three.
- Rewrite number four. The file got worse, not better, because each pass corrected tokens it had misremembered and introduced new errors in tokens it had remembered.
- It stopped, mid-confusion, and told me the INI format could not even take a
chat-template-filekey.
The failure is not that it forgot the template. The failure is that it treated a byte-exact artifact as a generation task. A chat template is not prose. It is a contract. Regenerating it is like regenerating a TLS certificate from memory: the output looks plausible, it differs in the places that matter, and the model has no internal signal that it is wrong.
The fix was two shell commands:
bash
curl -fsSL <source-url> -o /tmp/mimo.jinja
cp /tmp/mimo.jinja models/templates/mimo-v2.6-distill-qwen-9b.jinjaNo model in the loop. The 27B model's contribution was deciding that the file had to move by copy, and then verifying the copy.
The rule I now apply to any artifact that must be byte-exact, such as chat templates, config files, keys, or pinned manifests, is that it moves by copy, never by model regeneration. If a model needs to produce one, the source must be in its context or on its disk, and the operation is cp, not a completion.
Finding 3: verification is the skill that separated the phases
When I looked at what every recovery in the session actually was, each one turned out to be a verification act, not a generation act: re-fetch the source and compare, render the template with llama-cli and read the serialized prompt, start the server with --no-models-autoload and read GET /models for the resolved flags, re-read the file after an edit.
The model that skipped verification (the 35B) produced the corruption. The model that verified (the 27B) fixed it. The 9B performed verification in the sense that it ran the commands and narrated "let me check", but twice it checked something that could not fail, or read the result wrong.
The two checks that were actually decisive, for the record:
bash
# Render check: does the template serialize as intended?
llama-cli --model mimo-v2.6-distill-qwen-9b-q8.gguf \
--chat-template-file models/templates/mimo-v2.6-distill-qwen-9b.jinja \
-ngl 0 -c 512 -n 1 --no-display-prompt -p "Say hi" </dev/null
# Preset check: did the INI parse into the flags I expect?
llama-server --host 127.0.0.1 --port 7999 \
--models-preset models-router-mtp.ini --no-models-autoload &
curl -fsS http://127.0.0.1:7999/models | jq '.[] | .alias, .metadata'The --no-models-autoload flag matters, because the server starts without loading weights, so the check costs seconds instead of a model load. The GET /models endpoint returns each model's resolved flags (chat-template-file, mmproj, input_modalities), which is exactly the question "did my preset do what I think it did?"
A verification step that cannot fail is not a verification step. The 9B's test -e X && echo exists || ln -s …; echo 'symlink created' is the canonical example from this session: the trailing echo runs unconditionally, so the output proves nothing. The whole line is a ritual, not a check. It read "exists" plus "symlink created" as "symlink works", moved on, and the next command failed.
Finding 4: what distillation transfers
This is the part I came for, and the answer is narrower than I hoped.
The MiMo distill is a fine-tune of Qwen3.5-9B trained on the behavior of a larger MiMo model, and the session showed me what that buys you in practice.
What transferred was step-following. The 9B asked clarifying questions first, produced a clean phased plan, and once a step was explicit ("copy this file, run this command"), it executed it well over SSH. That is a real, usable skill at 9B, locally, and it is consistent with the published numbers, where the fine-tune's biggest gains are on agentic benchmarks like Terminal Bench and Toolathlon rather than on raw reasoning.
What did not transfer was judgment under repeated failure. The same scp failure happened three times, and each time the model patched the symptom (mkdir, retry) instead of asking why the same assumption kept failing. It did not notice it was repeating, and it did not flag the stall: 18% of its turns simply ended mid-task and waited for me to notice.
This is also a data point for a question I have been chasing since the 230M tool-calling experiment. There, the LFM2.5-230M showed what I would call the "lexical" approach: it pattern-matched the shape of the task to the shape of the response, made perfectly-shaped tool calls with fabricated arguments, and repeated the same failed call five times in a row. It generated the right tokens without any deep "understanding" of why they were right. The 9B distill in this session shows the same shape at higher resolution: it follows explicit steps well, but it does not build a model of the system it is working on, so the third scp failure looks like a new failure, not a repeat.
What I am interested in is the threshold where this behavior stops. Is it a size threshold, where the model finally has enough capacity to hold the task state in working context? Is it an architecture threshold, where dense and MoE behave differently at the same parameter count? Or is it a training threshold, where enough diverse agentic data replaces pattern-matching with something closer to understanding? The 230M article gives me the low end, and this session gives me a point at 9B: lexical-plus, it follows steps but still repeats failures. I need more points before I can draw the line.
So the honest summary is that distillation transfers the teacher's skills and the base model's fragility, but not the teacher's judgment. The 9B followed explicit steps better than the 35B MoE did, and it self-corrected worse than the 27B dense did. If you are evaluating a distill, test the second failure, not the first task, because the first task is the demo.
Finding 5: supervision cost is the honest metric
I stopped counting "did it finish" and started counting interventions, because completion rate hides the real operating cost. For this session, the interventions were:
| Intervention | Type |
|---|---|
| Repeated an instruction that got no response | re-prompt |
| "I didn't ask for opencode; just the model router" | scope-creep stop |
| "You don't change the file on this mac. Create a new file then copy it" | method stop |
| "Tailscale is already installed and configured" | fact injection |
| "Use /opt/homebrew/bin/tailscale" | fact injection (path) |
| "Before proceeding with tailscale, test the llama-server router" | sequencing stop |
| "Instead of rediscovering, copy the llm-serve-router-mtp from this mac" | method stop |
That is eleven stops and corrections in 96 minutes, across three models that are all, by any vendor's description, capable. It works out to roughly one human intervention per ten minutes of wall time, for work that was about 70% complete when the session ended, since the final symlink verification on the mini was still failing at export time.
How I read the four types:
- Re-prompts measure attention retention: did the last instruction survive in working context?
- Fact injections measure world-knowledge gaps the model refused to probe. It could have checked whether tailscale was installed. It assumed.
- Method stops measure judgment: the model chose a path that violated a constraint I had not restated.
- Stalled turns measure self-termination: the model stopped mid-task without saying so.
None of these show up in a benchmark, and completion rate hides them all. If you are deciding what to hand to a local model, supervision cost per unit of finished work is the number that predicts your actual operating cost.
What I would do now
For the next setup session like this one:
- Copy, never retype. Byte-exact artifacts move with
curl+cp(orscp), and the model's job is to orchestrate the copy and verify the result. - Force a verification step after every mutating command, and make sure the check can fail. Read the file back. Hit the endpoint. Run the binary.
- Ban re-discovery when the artifact exists locally. If the router script is on this machine, the instruction is "copy it", full stop. Probing the remote's
--helpis how three scp failures happen. - Measure supervision cost from the session export. The model ID is recorded per message, so per-model stops and stalled turns are reconstructable after the fact. One export, one spreadsheet, one honest number.
- Do not benchmark a model on its demo task. Test the second failure.
Key takeaways
- Size did not predict reliability in a real agentic session. The 27B dense model was the most reliable phase, the 35B MoE was the least, and the 9B distill was in between.
- Byte-exact artifacts (chat templates, configs, keys) must move by copy, never by model regeneration. The model's job is to fetch, copy, and verify, not to reproduce.
- Verification is the skill that separated the phases. Every recovery in the session was a verification act, and every corruption came from a skipped one.
- Distillation transfers skills, not judgment. The 9B distill followed explicit steps well and never self-corrected a repeated failure.
- Supervision cost is measurable from session exports, and it is the number that predicts operating cost: about one intervention per ten minutes here, for work that ended 70% complete.
A 9B distill on a Mac mini is not an employee. It is a fast, cheap, occasionally confident junior who follows explicit steps well, does not notice when it is repeating a failed action, and will happily regenerate a certificate from memory if you let it. The 27B dense model was closer to a senior, but it only arrived after I complained.
None of that is a reason to stop. It is a reason to design the harness around the failures you now have names for: retyping, unverifiable checks, re-discovery, silent stalls. Those are mechanical problems with mechanical fixes. The model size is the part you cannot fix.
FAQ
Why did I switch models mid-session?
Because the 35B had corrupted the template and made a false claim about the INI format, and I wanted a different model to check its work. That turned out to be the cleanest possible control: each new model started by auditing the previous model's damage.
Was the 9B distill a failure?
No. It asked better questions than either Qwen model, planned cleanly, and executed explicit steps well over SSH. It stalled 18% of its turns and never self-corrected a repeated failure. For explicit, well-scoped, copy-and-run work it is a solid local executor. For open-ended debugging it needs a supervisor, which is what I was.
Is the 35B MoE bad?
It is not bad. It is a 35B model that was asked to produce a byte-exact artifact from memory and did what any generative model does in that situation: it hallucinated plausibly and could not tell. The failure was the task assignment, not the model. The 27B with the same task and a "check" instruction did it right.
How do I get the supervision-cost number for my own sessions?
Export the session (opencode records model ID, tool calls, and aborts per message), then count: user messages that correct or repeat, tool calls that failed and were retried, and assistant turns that ended without a tool call or a stated next step. Divide by wall time.