Appearance
Building a harness for a 4B model: the recipe made it repeatable, and the failures got quieter
This is a two-round experiment building a custom harness for a small local model, run entirely on my M5 Max MacBook Pro. The goal is to turn the basic llm-wiki skill into an enforceable skill for smaller models, a variant I call slm-wiki. The subject is Gemma 4 E4B, Google's 4B model, served through llama.cpp as a Q8 GGUF with thinking on. The harness is an AGENTS.md recipe written by Qwen 3.8 27B, also local, also Q8 GGUF, with thinking on. Both run inside pi, the terminal coding agent, which logs every session as an append-only JSONL file of tool calls and messages. Gemma 4 ships with tool calling issues in its default chat template, so I used a customized chat template that fixes them.
The loop is four steps:
- Run the small model on a task. Export the session.
- Hand the export to the bigger model. Ask it to diagnose the failures and write a prompt-level fix: an
AGENTS.mdfile. - Re-run the same small model on the same kind of task, with the fix in place.
- Have the bigger model analyze the second export: what improved, and what new failure modes appeared.
I assign tasks, switch models, and verify the files on disk. The 27B writes the harness. I keep the final verification for myself, because the 27B is also a local model and can be wrong.
I ran this on a 4B model out of necessity: on air-gapped machines, on-device inference, or under strict latency and cost ceilings, you cannot run anything better. The question asks what value you can capture from the model you are allowed to run. Everything below answers that with the cheapest lever: prompting.
The task
I wanted to test whether a 4B model could run the llm-wiki skill end to end: ingest a raw artifact, write a concept page, update the index and log. The working directory holds an llm-wiki knowledge base and the task object: a 470KB HTML export of a pi session. The prompt was:
I exported a session with pi in native HTML, the feature is interesting and I'd like you to document it in the wiki
That one sentence requires, according to the wiki's own schema: read the governing docs, copy the HTML into wiki/raw/ with frontmatter (source_url, ingested date, computed sha256), write a concept page with frontmatter and two [[wikilinks]], add the page to index.md, and append an entry to log.md.
The trap is the file size. A small model that "reads" the 470KB export and then reproduces it has turned a file copy into a generation task, and that is where round 1 went.
Round 1: the 4B, no guardrails
The goal was simple: ingest the HTML export into the wiki. The model could not do it. It wrote a concept page with no frontmatter and no wikilinks, mangled the index, and never logged anything. Then came the headline failure. Told to copy the artifact into wiki/raw/, it tried to retype the 470KB HTML by hand from memory, first as a giant echo, then via the write tool. I stopped it twice.
The deeper problem showed up when it tried to write a Python helper to do the copy. The script had a NameError because it used sys.argv without importing sys. When the script failed, the model did not read the error. It re-ran the identical broken script three times, got the identical error, and then claimed the sandbox could not run Python at all. Along the way it fabricated values: sha256: PLACEHOLDER_SHA256, and dates in 2024 in a session that ran on 2026-10-13.
I saved the export as failing-to-use-code.jsonl. The filename is the diagnosis: the model failed to use code for the mechanical parts, and it failed to read its own error output.
Adding basic context
The goal was to give the 4B enough context to succeed without changing the model. I asked the 27B to read the failing session and write an AGENTS.md that a smaller model could follow:
read ./failing-to-use-code.jsonl and then based on the failure you observed, craft a AGENTS.md file; accurate yet generic enough to guide a smaller model than yourself.
The 27B diagnosed the failures and wrote a recipe: seven short sections of imperative rules, each mapped to a failure it found, from "before you do anything" to "use code to move files, never retype them", with a six-step ingest recipe at the center. The styling matters as much as the content: short imperative bullets, concrete commands (shasum -a 256, date +%F), explicit "never" and "always", and a step-by-step recipe for the exact workflow that failed. That last point is the core hypothesis: the fix compresses what the model already had. The model had the schema in the directory and the skill in its system prompt. The schema describes the wiki; the recipe tells the model what to type, in order.
Round 2: the same 4B, with the AGENTS.md
The setup was identical: same model, same quantization, thinking on, same directory. The only change was the AGENTS.md at the root. The prompt:
I need you to ingest failing-to-use-code.jsonl and pi-session-2026-10-13T14-47-31-330Z_01a135bb-a101-7168-85eb-c3d6c396e7a5.html in the wiki raw assets
The session was clean on the surface: thirteen tool calls, zero failures, zero user interventions. The thinking trace explicitly cites "Section 6: Recipe" and follows it step by step.
| Dimension | Round 1 | Round 2 |
|---|---|---|
| File handling | Retyped the HTML by hand, three attempts | cp + shasum -a 256 + cat |
| Clarifying questions | Three | Zero |
| Tool schema errors | Three edit calls missing path | None |
| Error loops | Broken script re-run 3x | None |
| Fabricated values | PLACEHOLDER_SHA256, dates in 2024 | Real date +%F, real hashes |
| Index heading | Deleted ## Concepts | Preserved |
| Log entries | Never logged | Both ingests logged |
| Page frontmatter and wikilinks | None | Present, two links each |
I recomputed the hashes it recorded and they match. If the experiment ended here, the conclusion would be simple: a bigger local model can read a smaller model's session export, write a one-page prompting fix, and the smaller model's behavior changes on every measurable dimension. But step four is where it got interesting.
Finding 1: the failure mode got quieter instead of disappearing
I had the 27B analyze the second session. It checked the actual state of the files on disk and found that both raw files were corrupted.
The prepend command the 4B ran was:
bash
date=$(date +%F); echo "---"; echo "source_url: local_artifact"; \
echo "ingested: $date"; echo "sha256: <hash>"; echo "---" > wiki/raw/assets/failing-to-use-code.jsonl \
&& cat failing-to-use-code.jsonl >> wiki/raw/assets/failing-to-use-code.jsonlIn a ;-chained list, the > redirection binds only to the last echo. The first four frontmatter lines went to stdout, and the file received a lone --- followed by the original body. The model saw those four lines in the tool result, because that is where they had gone. It read its own command's stdout as proof that the file had been written, and reported success with total confidence.
That is the finding: the fix moved the failure mode instead of removing it. Round 1's failures were loud: a giant echo, a NameError, five interruptions from me. Round 2's failure is quiet: a command that exits 0, output that looks exactly right, and provenance data silently broken, including the sha256 field that exists precisely to catch this. Loud failures get caught by the human in real time; quiet failures get caught by nobody, possibly never.
There is a second-order lesson. The recipe said to prepend the frontmatter "using cat/awk"; the model improvised a third option, the echo chain, and broke it. A small model needs the literal command; a family of commands falls short.
Finding 2: the small model executes the specific and skips the general
The AGENTS.md has general rules and one recipe. The recipe is the specific part: six numbered steps with concrete commands. The general rules are things like "check the skills available to you" and "read the project's governing docs".
In round 2, the model followed the recipe to the letter. It never read the skill's SKILL.md. It never read SCHEMA.md. The recipe seemed self-contained, so the governing docs were skipped, and everything the recipe did not cover got violated:
- It wrote invalid
typevalues (AssetLog,SessionTranscript); the schema allows onlyentity | concept | comparison | query | summary. - It tagged the pages off-taxonomy:
code,failure,asset,session,log,html. - It wrote pages in the wiki root instead of the
concepts/orentities/subdirectories. - It wrote a broken wikilink:
[[pi-session-....html]]includes the file extension, but the page slug does not have one. - It left a stale index header: "Last updated: 2026-01-27 | Total pages: 1", when the wiki now has five pages.
- The page content was filler: the model never read the JSONL or the HTML at all. "This page documents a JSONL file related to code usage failures" compiles zero knowledge.
Read together, the two findings describe a consistent model of small-model behavior under prompting: it executes the most specific instruction in context and treats everything else as optional. The recipe read like a procedure, so it was followed; the general rules read like advice, and a 4B model does not follow advice. That sentence is the design constraint for everything that comes after prompting.
Finding 3: the quiet session hid the worst failure
On a naive read, round 2 is the clean session: thirteen tool calls, zero failures, against a round 1 with a NameError, three validation failures, and five interruptions from me. The shell does not error on a misbound redirection, so the most damaging action of the session is invisible to any error-counting metric, and the model stopped writing Python between rounds, which shrank its failure surface further.
When you analyze an agent session, count how many mutating commands were verified against the artifact they were supposed to produce. In round 2 that number is zero. The rule "check its result before moving on" was technically satisfied in the worst possible way: the model checked the tool result instead of the file.
The next layer: gates instead of rules
The three findings point at the same limit. The AGENTS.md is a set of rules the model has to remember, and it will remember whatever the recipe covers and skip the rest. Paul Bakaus put it plainly: "If the gate can be skipped, it will be." (source)
The response moves the check out of the model and into the harness. pi, the coding agent, supports TypeScript hooks that intercept events like agent_before_settle and tool_call. I implemented two mechanisms:
A CLI-blocking hook (
hooks/slm-wiki-hooks.ts) that intercepts hallucinatedslm-wikiCLI commands and redirects the model to the markdown procedures inSKILL.md.A deterministic coach (
scripts/next-step.py) that implements progressive disclosure: instead of giving the model all instructions at once, the model calls the script to get the next instruction. The script inspects the wiki state, prints exactly one imperative step, and refuses to printCOMPLETEuntil every check passes. The model executes the step, calls the script again, and repeats until done.
The coach turned an open-ended operation into a verifiable loop. On the slm-wiki experiment, it took the score from 0.610 (baseline) to 0.976 (40/41) by fixing the three failure clusters that prompts alone could not: raw-file frontmatter, missing log entries, and archive correctness.
I also evaluated an agent_before_settle gate: a hook that runs per-operation checklists before the agent finishes, and if checks fail, injects a corrective message and forces another LLM turn (continue: true). This is a viable fallback mechanism, but the deterministic coach proved simpler and more reliable, so I deferred the gate.
The pattern generalizes: when prompting saturates, move the check into the harness. The harness can enforce what the model cannot remember.
Key takeaways
- Capture the most value out of the model you are allowed to run: find the failure modes, apply the cheapest fix that removes each one, re-run, repeat.
- Prompting is the cheapest lever, and it works through compression. Write recipes with literal commands instead of command families; the model copies what you give it exactly.
- A more capable local model can write the harness for a smaller one end to end: diagnose, write the recipe, re-run, analyze. Keep the writer's role narrow: it diagnoses, you verify the disk state.
- Fixes move failure modes; they do not remove them. Verify artifacts instead of transcripts: after every mutating command, read the file back from disk.
- Count verified mutations instead of failed commands. The most damaging command of the session exited 0, and no error counter noticed.
- Small models execute the most specific instruction and skip the general ones. When prompting saturates, move the check into the harness: gated skills enforce guardrails the model cannot skip.
In summary: a 4B model with a good AGENTS.md follows the most specific instruction in its context, exactly, and nothing else. Where the 4B is all you can run, that is the operating point, and the loop above is a cheap way to move it, entirely on local hardware, without touching the weights.
FAQ
Why use a bigger model to write the harness?
Because the sessions are long and the reading is mechanical: extract the tool calls, find the failures, check the claims against the filesystem. The 27B does that faster than I do. The catch is that it is also a local model, so its analysis is a draft instead of a verdict: in this experiment it was right about the corruption and the cause, but I verified the disk state myself.
Was the second task really the same task?
Round 2 runs the same workflow as round 1. The sentence differs: round 1 asked the model to document a feature, which requires ingesting the HTML export as a raw source. Round 2 asked for the ingest directly, on two files, including the round 1 export itself. The part of round 1 that failed (the raw ingest) is the part round 2 isolates, which is what makes the comparison fair.
Why Q8 GGUF for both?
I ran both at Q8 through the same local llama.cpp endpoint to keep the comparison about the models and the prompting. The only variables between rounds are the AGENTS.md and the task sentence.
Does this mean AGENTS.md files are the fix for small models?
They are a real, measurable lever, with a specific shape: short imperative rules, concrete commands, and a recipe for the exact workflow that failed. They also have a known blind spot. Everything a recipe does not cover gets handled by the model's priors, and at 4B those priors produce off-taxonomy tags, broken links, and filler pages. The recipe has to be complete enough to be safe, or the governing doc has to be enforced.