Temperature Zero Is Not a Lock

Temperature Zero Is Not a Lock

Let's Call It Zero-Point Engineering

You go back to reproduce yesterday's result, because a colleague flagged something odd in the output and you want to see it again before you touch anything. Same prompt, same model name in the config, temperature pinned to zero, the setting everyone reaches for when they want the model to stop being creative. You run it. The answer comes back different. Not wildly different, not gibberish, just different enough that the field you were chasing now reads correctly, and you're left holding a genuinely unsettling question: did you fix something between yesterday and today, or did the ground move under you while you were looking away?

The reflex answer is that temperature zero makes a model deterministic, but that reflex isn’t true. In September of 2025, Thinking Machines Lab published the clearest account of why. They ran Qwen3-235B a thousand times on an identical prompt at temperature zero and got eighty distinct completions, with the first divergence appearing as early as the hundred-and-third token. The culprit isn't the sampler, which temperature zero pins. It's that production inference servers batch concurrent requests together, the batch size shifts with real-time load, and the floating-point reductions inside the model aren't associative, so (a + b) + c and a + (b + c) land on different bits. The same prompt gets a slightly different numerical answer depending on who else happened to be in its batch, which is to say depending on how busy the server was the moment you hit it.

The model won't hold still, the knob you were told would freeze it doesn't, and on a hosted endpoint you don't even control the one variable that actually moves the output, which is the request batch you land in. Stop trying to make the model deterministic. Start engineering the pipeline around a model that isn't. Pin what you can pin, snapshot what you can't, and test the behavior relentlessly so drift is caught by a machine on your side and not by a customer on theirs.

Temperature Zero Is Not A Lock

The reason this matters more than a curiosity is that a great deal of production code is built on the unexamined belief that temperature zero buys reproducibility, and it doesn't buy reproducibility on any hosted API you're likely using. The Thinking Machines work is the cleanest demonstration, but the mechanism generalizes to every transformer served at scale: the numerics depend on batch shape, batch shape depends on load, and load is the one input to your output that you have no visibility into and no control over. Your prompt is deterministic. The server's traffic is not, and the server's traffic is now part of your function signature whether you acknowledge it or not.

There is a real fix, and its cost tells you why you don't already have it. Thinking Machines open-sourced batch-invariant kernels that make the reductions independent of batch size, and running them, the same thousand prompts collapsed to a thousand bit-identical outputs. LMSYS reproduced the approach in the SGLang engine shortly after. The catch is throughput: the deterministic kernels ran on the order of 1.6 to 2.1 times slower in the original benchmarks, with SGLang's optimized integration bringing the overhead down but not to zero. That price is exactly why the major hosted providers don't ship batch-invariant inference by default, and why "just call the API at temperature zero" will keep handing you eighty flavors of the same answer for the foreseeable future. The determinism is buildable. It is not free, and it is not on.

None of this means the model is broken. It means the model is a probabilistic component sitting behind a serving layer that adds its own noise, and if your architecture assumed a pure function where there's actually a load-dependent one, the assumption is the bug, not the model.

The Model Name Is Not The Model

Batch noise is the fast, within-a-second version of the problem. There's a slower version that's arguably worse, because it moves on a timescale where you'll have forgotten you were exposed to it: the model behind a given name changes over time. OpenAI's own cookbook says it directly, that even with a fixed seed and matching parameters, "determinism is not guaranteed," and that the system_fingerprint field exists precisely so you can detect when the backend configuration shifted underneath you. Notice what that field is: a tool for detecting drift, not for preventing it. The provider is telling you the ground can move and handing you a seismograph, not a foundation.

The most rigorous public measurement of this remains the 2023 study by Chen, Zaharia, and Zou. On the fixed task of identifying whether a number is prime, GPT-4 scored 84% in its March 2023 snapshot and 51% in its June 2023 snapshot. Same name, same API, a thirty-three point collapse on identical inputs, with no version bump a caller would have noticed. Later that year came the episode where users reported GPT-4 had grown "lazier," and OpenAI responded that it hadn't updated the model since November 11th and that the behavior shift wasn't intentional, which is its own kind of unsettling: the weights held still and the behavior drifted anyway. "We pinned gpt-4" pins a string. It does not pin a distribution of outputs, and the distribution is the thing your pipeline actually consumes.

The counterintuitive part, for anyone who assumes a bigger model is a steadier one, arrived in a 2025 study of output drift across model architectures on regulated financial tasks. The smallest models tested held 100% output consistency at temperature zero, while a 120-billion-parameter model held only 12.5%, and retrieval-augmented tasks drifted somewhere between 25 and 75 percent depending on the model. Scale does not buy you stability. Sometimes it costs you stability, and the report that runs on last night's model isn't guaranteed to agree with the one that runs on tonight's, even when nothing in your repository changed.

Consistency Is Not Correctness

Here's the trap waiting for anyone who takes the first half of this issue too much to heart and sets out to make the pipeline perfectly reproducible as an end in itself. Reproducibility is not accuracy. A pinned, seed-locked, batch-invariant pipeline that returns the identical answer every single time can be returning the identical wrong answer every single time, and its beautiful consistency will do nothing but make the error look trustworthy.

A large medical study in late 2025 made this concrete at a scale worth respecting, generating on the order of nine hundred thousand model outputs across hundreds of clinical cases run a hundred times each. The finding that matters for us is that a model's repeatability, how consistently it gave the same answer, didn't reliably predict its diagnostic accuracy across most of the prompting styles tested. Consistent and correct are simply different axes. You can have either without the other, and a system optimized only for the first is a system that has learned to be confidently, repeatably mistaken.

So the point of pinning was never to be right. The point of pinning is that when you ARE right, you stay right until you deliberately change something, and when you do change something, the change is the only variable in the room. That's not a philosophical nicety. It's the entire precondition for the regression testing this newsletter turns to next week, because you can't detect a regression against a baseline that's drifting on its own.

Pin What You Can, Snapshot The Rest, Test The Behavior

The discipline that falls out of all this has three moves, in ascending order of how much it protects you.

1. Pin what the provider lets you pin. Use dated model snapshots everywhere you currently use a floating alias, so a silent server-side model swap can't reach you without a config change you made on purpose. Log the system_fingerprint on every call so a backend shift is visible in your telemetry the day it happens. Understand the expiration date on this protection: OpenAI commits to roughly six months' notice before retiring a generally available model, Anthropic to a minimum of sixty days, and Google to as little as two weeks for preview models. Pinning defers the drift and buys you a scheduled migration instead of a surprise one. It does not abolish the drift.

2. Snapshot what you can't pin. You can't pin the batch you land in or the exact numerics of a hosted endpoint, so capture the behavior instead. Freeze a set of representative inputs and the outputs you expect from them, stored in your repository next to the code, as an artifact a human reviewed and blessed.

3. Test the behavior, and assert only on what should be stable. Run the snapshot set on every model bump and every prompt change, and assert on the parts that must not move, the extracted value, the classification, the schema, the sign of the number, while tolerating the cosmetic churn in phrasing that was always going to wobble. This is the deterministic wall from issue #12, moved around the whole model: the probabilistic core produces its output, and a plain, non-probabilistic check on your side decides whether that output is inside the band you signed off on.

That third move is where the real protection lives, because it's the only one that catches a regression the provider introduced without telling you. Pinning and logging make drift visible. A behavior test makes drift fail a build, which is the only form of "visible" that reliably stops a bad output before a customer meets it.

Quick Wins

🟢 Easy (~20 min): Take one prompt your pipeline depends on and run holdstill on it fifty times at temperature zero. Count the unique outputs. If the number is greater than one, and it usually is, you now have direct evidence that your "deterministic" call isn't.

🟡 Medium (~1 hour): Replace every floating model alias in your codebase with a dated snapshot, and add the returned system_fingerprint to the structured logs on each call. The next time a provider shifts the backend under you, it shows up in your telemetry as an event with a timestamp instead of a mystery in your metrics.

🔴 Advanced (half day): Build a behavior snapshot over 20 to 50 representative inputs, asserting only on the fields that must stay stable, and wire it to run on every model or prompt change. Set a drift tolerance, fail the change when the tolerance is exceeded, and require a human to re-bless the snapshot before it can move. You've built the wall the next issue is going to lean on.

Next Week

A behavior snapshot over a handful of inputs is where an eval suite is born. Next Tuesday: how to grow it into something that catches a regression the night it lands, before it's the customer telling you.

You went back to reproduce yesterday's result and the answer had moved, and the honest reading of that isn't that your setup is broken. It's that you were counting on a lock that was never a lock. Temperature zero pins the sampler and nothing downstream of it; the batch you land in moves with the server's load, and the model behind the name you pinned drifts on a slower clock of its own. The variance is real, it's structural, and no knob in the API dashboard turns it off.

What you can do is refuse to build on the fiction. Pin the snapshots the provider offers and log when they shift, capture the behavior you can't pin into a set you reviewed, and put a plain deterministic check on your side of the boundary so a drifted output fails a build instead of reaching a person. You can't nail the model to the floor. You can build a floor solid enough that when the model moves, it can't fall through.

The model won't sit still. Make the pipeline that doesn't ask it to.

P.S. A deterministic gate that never guesses is the spine of the TestScout MCP suite, and LightScout is that idea made concrete: a check that fails a build only when the numbers genuinely miss a fixed threshold, the same steady wall this issue is asking you to build around your models. If measuring your own wobble surprised you, forward this to the engineer who swears the call is deterministic. If someone forwarded it to you, subscribe at bashmatica.com.