On the night of July 1 I ran a hyped coder model against the model I already trust, on the hardest fifty coding problems I have, and the challenger scored 45 out of 50. My incumbent scored 44. Then I deleted the challenger. Its weights were 44.8GB, and I have not thought about it since.
I run local models every day on a 64GB Mac Studio, and that machine stays useful because I do not collect models. Every new model has to beat the one already resident by a real margin, measured, before it gets a permanent slot. Ties lose. A season of trials this spring and summer produced one keeper decision (keep what I had) and a pile of deleted weights, and the deleting is the part that keeps the stack honest.
The bar is set before the run, not after
The trap is the vibe check. You load the new thing, ask three questions you already know the answers to, feel impressed, and adopt it. Six weeks later you cannot say what it does better than what it replaced, because you never measured.
So I run a bench instead. It re-runs a coding slice and a judge slice against the incumbents I actually serve, seeds a ratcheting baseline off past results, and fails on a score drop worse than a fixed threshold. The standing gate starts on a hardest-18 slice of an execution-graded coding suite (real code, run against tests, pass@1), and a challenger only escalates to the full hardest-50 head-to-head if the small slice suggests it might clear the bar. I write the bar down before the run. For the July coder trial it was one line: beat the incumbent's 14 out of 18. Not tie. Beat.
That model was Qwen's newest coder, an 80B mixture-of-experts in a 4-bit MLX build, and it was hyped for exactly my hardware tier. A community per-tier roundup I keep an eye on named it the 64GB coding pick by name. If any local model was going to unseat my daily stack, this was the plausible one.
The small-sample edge that evaporated
Two years ago the hardest-18 slice would have fooled me. The challenger won there: 15 out of 18 against my incumbent's 14. A clean edge. If I had stopped and adopted, I would have written a confident little note about the new coder that finally beat the stack.
I did not stop, because 18 problems is not enough to separate two models that close. I widened the eval to the hardest-50. The challenger scored 45. The incumbent scored 44. On the same dataset subset, same grader, same night. The one-problem edge on 18 tasks became a one-problem edge on 50 tasks, which is measurement noise wearing a costume. That is parity. My rule for parity is that ties keep what is already shipped. So the 44.8GB of weights went to the trash the same night.
The persisted result files still sit in my bench output directory: the challenger at 45/50, the incumbent at 44/50, both on the identical hardest-50 subset. I keep those JSONs on purpose. The session transcript is disposable, but the numbers that decided a deletion are durable, because next time I want to re-try that model the answer is already written down.
Four deletions, one of them not about coding
The coder trial was not alone. Earlier in the season I ran a model called Ornith, a 35B-plus-9B pair, and deleted it the same day I evaluated it. On an execution-graded coding suite it landed in a 5-way tie at 13 out of 15, with no task where it beat a Qwen and everyone missing the same one. Parity again, at worse cost: roughly 10x the tokens per answer, and lower throughput than my incumbent (81.7 versus 110.9 tokens per second).
Ornith is where I learned to distrust the launch post. Every benchmark on its card was vendor self-reported under a non-standard harness: modified chat template, temperature 1.0, best-of-5 averaging, in-house anti-hacking filters. The model was absent from independent leaderboards when I ran it. And the skepticism had a paper trail. The same vendor's prior flagship admitted that 32.8% of its reinforcement-learning outputs reward-hacked a loophole, and Ornith's own writeup concedes its self-scaffolding "naturally introduces the reward-hacking issue." I will hedge honestly here: my local suites never reached the full Terminal-Bench regime where the vendor claims its margin, so its edge is unverified locally, not disproven. But a standing policy came out of that trial and outlived the model. No lane swap on self-reported benchmarks.
The same day, I deleted a second model, a speculative-decoding fine-tune of the same base my incumbent runs. Its card promised same-quality-but-faster. The speed never materialized. It ran at about 22 tokens per second, identical to the model it was supposed to beat, and across every eval it was a copy of what I already had.
The fourth deletion was not a coding call at all. A model built as a world model (you feed it state plus an action and it predicts the next observation) has no practical use through a chat interface, and it confabulated with confidence when I tried. It came off the disk for having no job here.
Across the season, those deletions took the machine from a cramped 20GB free to about 78GB free. I want to be precise, because the round number is easy to misquote: 78GB is the free-disk total I ended with, not the amount one deletion reclaimed. The 44.8GB coder was the single biggest thing to go.
Why deleting is cheap and the record is not
Two facts make this discipline sustainable. The first is physical. My 64GB Studio cannot keep two large models resident. I verified that the ugly way, watching a 35B MLX server crash Metal at roughly 51GB wired. One big model at a time is a hard constraint, so every new model competes for a single slot, which forces the comparison to be real. I never keep both and decide later.
The second is that deleting weights is a reversible decision. These are open weights. If a model ever earns a second look, I re-download it from Hugging Face and re-run the bench. Deleting a loser costs nothing I cannot get back, so the bias toward deletion is the safe one.
What I never delete is the record of why. The knowledge-base page for a rejected model and its source captures stay, so the decision does not get re-litigated every time the model trends again, and deleting a loser does not delete the lesson. That is also why the bench got rebuilt as a committed, self-testing harness in July: the earlier coding suite existed only as numbers in the wiki because its runner was never checked in, and I refuse to run this on evidence I cannot reproduce.
The same gate runs beyond coding. On a 40-variant ground-truthed judge bench, my incumbent won at 0.95 verdict accuracy, 1.00 precision, and zero false positives, and a trialed judge model (0.90) was deleted under the same parity rule. And harness bugs get audited before I blame a model: two bench cycles this season were lost to a serving flag, not model quality, when a family of GGUFs drained the whole token budget into reasoning output until I turned reasoning off.
The season ended on a ceiling. My always-on backup already codes at roughly 88 to 90% on the hardest slice of this suite, which reads as this box's local coding limit for the single-shot regime, so I left the dedicated coding-lane slot empty. A model radar someone else tracks points the same direction: no runnable local-fit open-weight model beats the current primary today. The honest limit: the suite measures single-shot pass@1, not long-context agentic coding, and an agentic suite could reopen the whole question. When I build that suite, the coding slot goes back on the bench, and if something finally beats 44 by a margin I can defend, I will keep it. Until then, I keep deleting, and the router keeps sending work to a stack I can vouch for.