I run a pair of report pipelines that read a long transcript, draft a structured summary, and pick the better of two candidate drafts. For months the picking was done by a cloud model. That model is competent and expensive, and every run pays the toll. I wanted to move the judging to a model on my own machine. I had already benched three local candidates on a ground-truthed corpus and picked a winner, so I took one production pair the cloud judge had already scored, handed it to that winning local model on the prompt it ships with, and asked the same question. The local model picked the other candidate.
That inversion is the whole story, and I nearly drew the wrong lesson from it. My first read was that the local model was too weak to judge. The bench said otherwise: this was the model that had scored highest. The real read is that I had given it half the job.
The prompt was missing, not the model
The candidates in that pair were not written blind. Each drafting run receives a checklist and a set of rules before it starts. The cloud judge had context I never passed to the local one. So the local model, on a bare prompt, failed in both directions at once. It missed decimal-scale errors where a candidate had printed a crypto level at ten to a hundred times its true value, the kind of typo that turns a plausible number into nonsense. And it flagged injected data as hallucination, because the numbers looked unsupported to a judge that had never seen the checklist the candidate was working from. Two opposite failures, one cause: information asymmetry.
The fix was scaffolding, not a swap. I rebuilt the judge prompt to carry two things the bare version lacked. First, a compiled defect taxonomy, a five-class cheat sheet of the mistakes these drafts make, held to an 8000-character budget so it stays inside the context that matters. Second, the run's own checklist, with an explicit instruction to judge the summary against the checklist rather than against the raw transcript. That second clause is small and it closes the exact gap that caused the false positives: data consistent with the checklist is not a hallucination, and the prompt now says so in plain words.
I fed the same production pair back through the local model with the enriched prompt. It reproduced the cloud judge's verdict. Same winner, the same scale errors cited, zero false positives on the checklist-backed numbers. One pair is one data point, so I hold it as an existence proof rather than a rate. But it told me where the missing half was.
I almost missed the taxonomy, because it was sitting in plain sight. A script compiles it deterministically, no model involved, from 30 mined prompt rules, 48 per-run improvement ideas, and 12 validator findings. Months of accumulated evidence about how these drafts go wrong, living in ledgers and validator output, that no judge had ever been shown. The judge was flying blind. Size was never the problem; I had never handed it the notes.
Benching the judge before trusting it
A single reproduced pair is not enough to swap a production role, and the bench that picked this model ran before I wired anything into production. That order was written into the tooling on purpose: the production judge wiring stayed deferred until the taxonomy-armed judge had been validated against the ground-truthed bench. The bench generates a 40-variant, defect-labeled corpus from the golden dataset of my open-source llm-judge-evals repo, so each variant has a known set of planted defects and the judge's answers can be scored against ground truth.
I did not want to trust my own labels either. The corpus generator ends with a self-check that re-reads every variant it wrote, re-derives the defects from the actual file bytes, and exits nonzero on any mismatch between what the file contains and what its manifest claims. The scorer is built to resist gaming: defect matching is one-to-one, so a judge that spams defect types or vague ticker mentions loses precision instead of buying recall. And the bench itself came out of a multi-agent build with adversarial reviewers and independent label auditors, with 13 review findings fixed before I let the corpus grade anything.
On that bench, the local model I now use won. It scored 0.95 verdict accuracy, 1.00 on clean variants, 0.933 on defective ones, and a 0.0 false-positive rate. Two other local candidates landed at 0.90. Every local model I benched caught 100% of the planted decimal-scale and invented-number defects, which is the failure class that hurts most in a numbers-heavy summary. Speed was not the constraint: the winner averaged 3.76 seconds and about 37 completion tokens per verdict, so a full 40-variant sweep is a couple of minutes of local compute. These are my own numbers off local artifacts, not a citable public benchmark, so read them as first-person reporting.
I keep one honest weakness in view. Format-violation recall was poor for every local judge, the winner catching 2 of 10 and the others none. I did not paper over that by leaning on the model harder. Structural format checks stay in deterministic code, where a regex is more reliable and cheaper than a language model. The LLM judges what code cannot: whether a number is a scale error, whether a claim has support. Code judges what it is good at.
What actually shipped
With the bench passed, I swapped both pipelines to the local model as primary, with the cloud model kept as a declared fallback lane in the model registry. Even then I staged the swap rather than hot-swapping it, because the consumer code was still shaped around the cloud judge and a scheduled morning run was close. The registry entry documents the rationale and the bench numbers, so the routing decision carries its own audit trail instead of living in my head. The production judge runs deterministically: temperature zero, JSON mode, one retry. And the enriched prompt is mandatory, written into the tooling as a standing rule. I never run the local judge on this pipeline without it, because I have seen what the bare prompt does.
The prompt does more than describe defects; it enforces consequences. Any severe decimal or price-scale error caps a candidate's score at 79 no matter how complete the draft is. Multiple severe scale errors cap it at 69 and trip a model-removal signal. The judge is handed the rubric with the arithmetic already in it, so it is not asked to feel its way to a grade.
Two clarifications, because the tidy version of this story would overstate it. The summary pipeline is triggered on demand through a chat command, not on a schedule. The genuinely recurring pieces are an overnight tuning loop, which uses the second swapped judge, and a daily regression bench I added after serving-layer changes had silently degraded my local models three times in a single week. On its first completed overnight run, that second judge did the thing I most wanted to see: it rejected its own candidate rule rather than rubber-stamping it, with a validation delta a hair above zero and two regressions. A judge that only ever approves is a rubber stamp with a latency cost, and this one discriminated.
The pattern here rhymes with something I wrote about earlier, on blind LLM judging and knowing when to stop: the model doing the evaluation is only as good as the independence and the context you give it. I run this stack on a 64GB Mac Studio under MLX and llama.cpp, the same rig behind my Cerebellum router, and the takeaway generalizes past my desk. Judge quality is mostly prompt scaffolding. The model matters less than the rubric you hand it and the checklist you let it see. And you bench a judge before you trust it, because a confident wrong verdict looks exactly like a confident right one until you check it against ground truth.