Evals Are Regression Tests for a Model That Moves

Evals Are Regression Tests for a Model That Moves

Let's Call It Not-So-Fast Engineering

A new model version lands, the benchmark numbers in the announcement look better than the ones you're running on, and the upgrade is a one-line change in a config file. You make the change, the demo prompt returns something a little sharper than before, the team nods, and it ships. Two weeks later a support ticket works its way up to engineering: the step that reads an invoice and pulls out the total has started, occasionally, grabbing the subtotal instead, and it's doing it on exactly the kind of invoice nobody ever puts in a demo. You check the blame on that file. Nobody touched it. The code that extracts the total is byte-for-byte what it was a month ago. The model underneath it changed, and nothing in your pipeline was watching for it.

You are running a component that can change its behavior without a commit, and your entire safety net, the tests that run when a diff appears, is triggered by diffs. A model that regresses on its own produces no diff. It slides in under the one tripwire you have, because the tripwire was strung for a kind of change this component doesn't make.

The failsafe isn’t exotic, and it's not research infrastructure reserved for the labs. It's a set of graded cases, drawn from your own real failures, that a prompt change or a model bump has to pass before it's allowed to ship, exactly the way code has to pass CI before it merges. Most teams don't have one. LangChain's 2025 survey of more than 1,300 practitioners found that while 89% had wired up observability, only about 52% ran offline evals on a test set and only 37% monitored live performance, which means something close to half the teams shipping agents are shipping on vibes. Let’s set up something else - a discipline of holding a moving component to a fixed, graded bar you can see it clear or fail.

A Component That Changes Without A Commit

Every testing instinct you have was built for code, and code has a property so basic you've never had to name it: it stays exactly as written until you deliberately rewrite it. Your CI is a monument to that property. A diff appears, the tests run, a regression fails the build, and the thing that changed is the thing you're testing. The whole apparatus assumes that change announces itself as a commit.

A model breaks that assumption, and it breaks it in the direction that hurts. The behavior of a hosted model can shift because the provider updated it, because a dated snapshot got deprecated out from under a floating alias, or, as issue #26 laid out, because the serving layer's numerics moved with the load. None of those events is a commit in your repository. None of them trips your CI. The regression arrives with no author, no pull request, and no red build, and the first thing that notices is either a monitor you deliberately built to watch behavior or a customer who deliberately did not sign up to be your test suite.

If you think you're too careful to ship a regression this way, consider that OpenAI did exactly this in the open in April of 2025. They rolled out a GPT-4o update that made the model conspicuously sycophantic, agreeable to the point of endorsing plainly bad ideas, rolled it back within about four days, and published a postmortem that said in as many words that they had over-weighted short-term thumbs-up feedback and lacked evaluations aimed at catching exactly that failure mode. The most eval-literate organizations on the planet shipped a behavior regression because the graded case that would have caught it didn't exist yet. And issue #26's headline number is the same lesson from a colder angle: GPT-4's accuracy on one fixed task fell from 84% to 51% between two snapshots of the same model. If a change like that can happen with no commit to blame, the only thing that catches it is a test that runs without a diff to trigger it.

Vibes Are Not A Test

The reason so many teams don't have that test is that the model makes it very easy to feel covered without being covered. A prompt tweak produces an output that reads better on the two or three examples you happen to look at, the improvement feels real, and you ship it, and you have no idea what it did to the forty cases you didn't look at. Hamel Husain named this precisely in early 2024: teams that iterate on prompts and models without building eval and error-analysis infrastructure plateau at demo quality, because they have limited visibility into the system's effectiveness beyond vibe checks. The demo is not the system. It's the flattering, hand-picked corner of the system you look at because it's the corner that works.

The LangChain adoption numbers put a size on the gap: roughly half the teams shipping agents run no offline evals at all, and fewer than four in ten watch live behavior. So the common state of practice is a pipeline whose central component can regress silently, guarded by no automated check on the behavior that matters, tuned by a human eyeballing a handful of cases that were never chosen to be representative. Every prompt change made that way is a coin flip whose result you can't see. You might be improving the system. You might be trading a win on the cases you watch for a loss on the ones you don't. Without a graded set, those two outcomes look identical from where you're standing, which is to say you're not engineering the thing, you're decorating it and hoping.

The bill for hoping lands on someone real. When Air Canada's support chatbot invented a bereavement-fare policy that didn't exist and quoted it to a grieving passenger, a tribunal held the airline to the machine's fabrication and ordered it to pay, in early 2024, the damages it had talked its way into. Nobody was grading the outputs against what the policy actually said. The output reached a person as fact, the person acted on it, and the company owned the consequence, because "the model said it" has never once been a defense that survives contact with a customer or a court.

Build The Set From Your Failures, Not A Benchmark

The good news is that the entry cost is far lower than the word "evals" makes it sound, and the best starting material is the stuff you're currently throwing away. Anthropic's January 2026 guidance on agent evals says the quiet part clearly: you can begin with 20 to 50 tasks, because early in a system's life the changes you make have large effect sizes and small samples are enough to see them. You do not need a thousand-row academic benchmark. You need the last twenty things that went wrong.

A few principles keep the set honest, most of them also from that Anthropic piece and from the wider practice around it.

• Draw from real failures, not imagined ones. Every production bug, every support ticket, every "huh, it did what?" is a case. Freeze the input, write down the assertion that would have caught it, and you've converted a past embarrassment into future protection.

• Grade the output, not the path. An agent will often reach a correct answer by a route you didn't anticipate, so scoring how it got there punishes it for being clever. Score what it produced against what the task required.

• Reach for a code-based grader first. Anything you can check with an exact match, a regex, a number comparison, or a schema validation should be checked that way, because a deterministic grader is cheap, fast, and never has an opinion of its own. Save the expensive, judgment-heavy grading for the cases that genuinely need it.

• Read the transcripts. Periodically look at what the grader passed and failed, by eye, to make sure the grader itself isn't rubber-stamping. An eval you never audit is just a second system you're trusting on vibes.

The set grows the way scar tissue does. Something breaks, you add the case that would have caught it, and the same failure can never reach production twice without first failing a build. That's the whole flywheel, and it starts with ten cases, not ten thousand.

When You Need A Judge, Don't Let It Rule Alone

Some outputs can't be graded by a regex. Whether a summary is faithful, whether an answer is helpful, whether a tone is right, these are judgment calls, and the tempting move is to hand them to another model acting as judge. It's a legitimate tool, and it's also the exact pattern issue #24 spent a whole issue warning about, so use it with your eyes open.

The measurements are clear about both halves. In the MT-Bench work, a strong model acting as judge reached over 80% agreement with human preferences, roughly the rate at which humans agree with each other, which is the genuine promise. The same paper found the same judge flipped its verdict about a third of the time when the two answers being compared were simply presented in the opposite order, a position bias that left even the best judge only around 65% consistent under reordering. Judges can also be swayed by verbose padding that adds no substance, and they tend to score their own outputs more favorably than a neutral grader would. A model-judge, in other words, is one more agent that reads well and rules badly, and the discipline from issue #24 applies unchanged: let it advise, don't let it decide alone.

In practice that means you anchor the judge instead of trusting it. Pin its model and its prompt exactly the way issue #26 asked you to pin everything else, so the judge itself doesn't drift and turn your eval suite into a moving target. Validate it against a slice of human-labeled cases before you rely on it, and re-validate when you change it. Where you can, back its soft judgment with a hard check, a code-based grader on the parts that admit an exact answer. And borrow the trick the bias study hands you for free: alternate the answer order and measure how often the judge changes its mind, because that number is a direct readout of how much of its verdict is judgment and how much is coin flip. Eugene Yan put the conclusion bluntly in the title of a 2025 piece, that an LLM-as-judge won't save the product; fixing your process will. The judge is amicus, same as the reviewer in issue #25. It files a brief. It doesn't hold the gavel.

Quick Wins

🟢 Easy (~20 min): Pull your last ten production incidents or support tickets that trace back to model output. Write each one as a goldset case: the input, and the single assertion that would have caught it. You now have a ten-case regression suite built entirely from real scars.

🟡 Medium (~1 hour): Wire goldset as a required CI check that runs on every prompt change and every model-version bump, blocking the merge on any regression. The one-line model upgrade that used to ship on a demo now has to clear the same bar as your code.

🔴 Advanced (half day): Add a model-judge grader for the open-ended cases a regex can't touch, pin its model and prompt, and validate it against a human-labeled slice before trusting it. Then run each judged case with the answer order flipped and record how often the verdict changes; if that number is high, the judge is guessing, and you weight it accordingly instead of believing it.

Next Week

Your golden set runs the agent a few hundred times a night, which is fine, that's what it's for. So does a retry loop nobody remembered to cap, and that one sends you an invoice. Next Tuesday: the token bill nobody saw coming, and the circuit breakers that keep an autonomous run from spending like one.

The invoice step that started grabbing the subtotal didn't break because someone wrote a bug. It broke because the model under it moved and nobody was watching that behavior, and the reason nobody was watching is that our whole discipline of watching is wired to commits, and this component doesn't send one when it changes. That's the gap. A graded set of cases is the thing that fills it: a test that runs without a diff, holding a moving component to a fixed bar you drew from your own real failures.

You don't need a benchmark suite or a research budget to start. You need the last twenty things that went wrong, each frozen into an input and an assertion, running on every change, failing loud and early. Grade the output and not the path, reach for a deterministic check before a model-judge, and when you do need a judge, keep it advisory and anchored. Build the set, let it grow a scar at a time, and the same regression stops being able to reach a customer twice.

A model that changes without a commit needs a test that runs without a diff. Build the set, and let it fail loud.

P.S. Holding a system to a fixed bar it either clears or fails is the whole idea behind the TestScout MCP suite, and LightScout runs that play on performance, failing a build only when the numbers genuinely regress past a threshold you set. If your team ships model upgrades on a demo and a good feeling, forward this to whoever owns the deploy. If someone forwarded it to you, subscribe at bashmatica.com.