AI engineering

Non-Determinism Is a Bug, Not a Feature

JSON schemas were bolted onto models that write. Jev, and the decision models that followed it within weeks, return the answer instead.

“Non-Determinism Is a Bug, Not a Feature” sat in my Medium drafts for over a year before Jev came onto the scene. Anyone who was early to building agentic harnesses learned that half the battle was getting tool calls over JSON to work at all.

Gemini, circa 2024 and 2025, was notorious for it. Its API has a finish reason for this exact failure, MALFORMED_FUNCTION_CALL, and developers have been filing reports about it from Gemini 2.0 Flash onward. The model would decide to call a tool, and instead of a JSON object it would return Python, or nothing, with no error raised.

The standard workaround in one agent framework after another was to send the same request again and hope. Gemini helped many of us realize that most of the value of working with LLMs was using an LLM to infer a decision, not to write one. The writing was overhead we’d added so the decision could travel back to our code.

That’s the problem Jev was built for, and it’s why I think the last month will matter more than most model launches. For the parts of a system that make decisions, non-determinism was never a feature. It was a bug we’d learned to wrap. It’s also the thesis behind PWTL, the task language we built at GARDE1, where the AI chooses rather than writes at the level of the task. Jev arrives at the same idea one layer lower, in the model itself.

Strings All the Way Down

A language model produces a string. Software needs a type. Everything we’ve built to connect the two exists because the thing doing the deciding only knows how to write, and it’s worth looking at what that everything actually is, because each layer is a patch on the one before it.

The first layer is the schema in the prompt. You paste a JSON Schema into the instructions and ask nicely. That’s a request, not a guarantee. By OpenAI’s own account, its trained model alone matched complex schemas 93% of the time, which sounds high until you remember it means seven failures in every hundred calls (OpenAI). The second layer is JSON mode, which guarantees the output parses but, in OpenAI’s words, “does not guarantee that the model’s response will conform to a particular schema.” The third is tool calling, where a model fine-tuned to emit function arguments hands them to the provider to parse, which is exactly the step Gemini kept failing.

The fourth layer is constrained decoding, and it’s the one that finally made the guarantee real. The provider compiles your schema into a grammar and, at every step of generation, masks out any token that would break it. Outlines did this in 2023 by turning a schema into a finite-state machine; XGrammar extended it to full context-free grammars in 2024, checking a vocabulary that can run to 128,000 tokens at every single step. OpenAI shipped it as Structured Outputs in August 2024, and on its own eval of complex schemas, gpt-4o-2024-08-06 with Structured Outputs scored 100%, against less than 40% for gpt-4-0613 without it. That comparison mixes a newer model with the new technique, but constrained decoding became the standard. The fifth layer is the one every team adds anyway, which is to validate the result in code and retry when it fails.

So what did all of that fix? Notice that it’s only the shape of the output. The model underneath still writes a string one token at a time, still samples, and still has no idea it’s making a decision rather than composing a paragraph. Every layer was bolted on from the outside.

The Pitfalls of a Bolt-On

The first pitfall is that a valid shape says nothing about a valid answer. A schema can guarantee that "category" is one of five allowed strings. It can’t guarantee it’s the right one, and because the output is a single sampled answer, it can’t tell you how close the second choice was. Nothing in a JSON object says “I’m 51% sure.” OpenAI’s own launch post says Structured Outputs doesn’t prevent mistakes in the values themselves.

The second is that the constraint can make the model worse at the part that matters. In Let Me Speak Freely?, presented at EMNLP 2024, researchers found that format restrictions, and JSON mode in particular, degraded reasoning, with stricter constraints causing larger drops. In one task the JSON parsing error rate was only 0.148%, yet accuracy fell by 38 points, which means the damage came from the constraint itself, not from broken output. The same study found structured output helped on classification, and critics have argued its prompts weren’t perfectly matched across conditions. There’s a mechanical version of the same problem. The grammar thinks in characters and the model thinks in sub-word tokens, and when the two don’t line up, the mask forces the model into tokenizations it would never naturally produce. Researchers at ETH Zurich showed at ICML 2024 that many constrained decoding methods “significantly impair task accuracy” for exactly this reason. Forcing a writer to write in a cage changes what it writes.

The third is that the schema isn’t really JSON Schema, and every provider supports a different piece of it. OpenAI’s strict mode requires every field to be marked required and additionalProperties: false on every object, caps a schema at 5,000 properties and 10 levels of nesting, and doesn’t support allOf, not, if, then or else (OpenAI). Azure runs the same OpenAI models with tighter rules, allowing 100 properties and five levels of nesting, and no pattern, format, minimum, maximum, minItems, maxItems or uniqueItems. Structured outputs on Azure don’t work with parallel function calls at all, or with Assistants, Foundry Agents or bring-your-own-data, and some newer chat models haven’t supported it (Microsoft). A schema that works against OpenAI can come back from Azure as a 400.

Amazon Bedrock didn’t add structured outputs until February 2026. It compiles each new schema into a grammar and caches it for a day, and it doesn’t support recursive schemas, external references, or numeric and string-length constraints (AWS). In practice it still fails in ways the guarantee doesn’t cover. Some models on the platform reject the structured output field outright, as Llama 4 does (aws-samples). Popular client libraries have silently dropped the schema on the way to Bedrock and gotten unconstrained free text back (LiteLLM, Bifrost). And a response that hits its token limit comes back truncated and won’t parse, which is why AWS’s own examples check the stop reason before parsing anything.

A team supporting three providers maintains three dialects of the same contract, and each one fails differently. How many of those teams have a test that catches when one of them drifts?

The fourth pitfall is the one Gemini taught us. When the bolt-on fails, it fails as an empty response, a malformed call, a refusal or a truncation, and the fix is a retry. Retries turn one call into a variable number of calls, with variable latency and variable cost. The decision might be deterministic in the end, but the system that produces it isn’t.

Non-Determinism Was Never the Feature

We tend to treat variation in model output as a creative property. For writing, sometimes it is. Ask for five taglines and you want five different taglines. In chat, it usually isn’t. The only chats where variation is worth anything are the creative ones. Marketing copy, a sales pitch, brainstorming names, a wedding toast, role-play and interactive fiction, rehearsing a hard conversation, practicing a language, therapy, journaling, companionship, or just shooting the shit are all conversations where hearing the same reply twice would feel canned. A customer who asks a support assistant the same question twice should get the same answer twice, and a question about a refund policy shouldn’t have a lucky answer and an unlucky one. People hold each other to the same standard. A husband who tells his wife one thing at breakfast and something else at dinner isn’t being creative, and anyone who’s been married knows how that conversation goes. And most of what an agent does isn’t chat or writing. It’s routing a ticket, choosing the next tool, deciding whether a task is done, deciding whether a claim meets a policy. For those, the same input should produce the same decision, every time, or you can’t test it, audit it or trust it.

Turning the temperature down to zero doesn’t get you there. In September 2025, Horace He and Thinking Machines Lab showed that LLM inference is non-deterministic even at temperature zero, because the GPU kernels serving the model aren’t batch-invariant: results shift with how many other requests happen to be in the batch (Thinking Machines). On a standard vLLM setup, 1,000 identical requests produced 80 different completions. Their fix made the outputs identical, at a performance cost.

For a writer, 80 versions of a paragraph is a curiosity. For a decision, it’s 80 chances to route the same customer two different ways.

That’s the bug in the title. A decision system that can give two answers to one question is broken in the same way a function that returns two values for one input is broken, and we spent two years treating it as the price of intelligence. It never was. The feature was inference: for the first time, a machine could come close to understanding something the way a person would, in human language. The variation came along with the way we sample from it, and we mistook the side effect for the point.

System One

Jev, released by TypeSafe AI on September 15, 2026, starts from the other end. It doesn’t write. You send it the state of your program, unstructured text like a customer message or a proposed tool call, along with typed questions: pick one of these options, score this against these levels, answer yes or no. It returns a typed value for each question, with a calibrated probability, in a single parallel pass that TypeSafe says takes 70 to 500 milliseconds. The company calls it a System One model, after Daniel Kahneman’s Thinking, Fast and Slow, where System 1 is the fast, intuitive mode of judgment and System 2 the slow, deliberate one.

The structural claim is the important one. Because Jev chooses from an answer space you define, it can’t return a value outside that space. TypeSafe’s 0% error rate on structured output isn’t a benchmark result. It’s true by construction, and TypeSafe says so. What it measures empirically is accuracy. On TypeSafe’s own four-workflow evaluation, Jev scored 67.8%, level with GPT-5.6 Terra at 67.9%, and about six points behind GPT-5.6 Sol and Claude Opus 5, at a fraction of their cost and latency.

The company is unusually candid about the limits of that claim. Its team wrote the workflows, it ran the speed tests “from our laptops on the West Coast,” and on price it says, “We can’t prove it isn’t subsidized.” There’s no independent reproduction yet. The claim worth building around isn’t the benchmark. It’s that the output is a decision with a probability attached, which is the thing software wanted from the start.

DataCamp’s review says the name is also a nod to William Stanley Jevons. If so, it’s apt. In The Wrong Layer of Abstraction, I used Jevons’ coal paradox to argue that cheaper software doesn’t mean less software. Cheaper decisions work the same way. When a judgment call costs a fraction of a cent and a few hundred milliseconds instead of ten seconds and several cents, it becomes worth putting a decision at every branch of a workflow that used to be a hard-coded if.

Three Weeks

What happened next is the best evidence that the idea was waiting to be built. Upstage’s Solar Decide reached OpenRouter on September 28, serving the same request shape as Jev. On September 29, at DevDay, OpenAI announced a Decisions API that focuses GPT-6 Luna “on a specific set of user-defined questions with finite pre-defined answers,” and opened it as a public beta on October 6. On October 1, Cloudflare released Clef and Clef-flash, open-weight decision models under Apache 2.0, compatible with Jev’s API. On October 9, Microsoft added Microsoft-Decision-1 to Foundry, built on a 9-billion-parameter Qwen model.

In under four weeks, a new category went from one startup to the largest model lab, two hyperscalers and an open-weight alternative, and several of them chose to speak Jev’s API rather than invent their own. Categories don’t usually standardize that fast, so why did this one? Because every team building agents already had the problem, and had been solving it with retries.

What Typed Decisions Don’t Fix

None of this makes a decision model correct. It makes it well-formed, and fast enough to use everywhere, which is a different property. The first serious adversarial study arrived nine days after launch. In JevOut, researchers added short, natural-sounding context to inputs, background information rather than instructions, and redirected 61% to 73% of four decision systems’ correct answers, Jev included, toward a wrong option chosen in advance. In blinded review, annotators judged more than 90% of those additions natural and free of anything that should have changed the answer.

The flipped answers often came back with high confidence. A calibrated probability on a manipulated input is still a confident wrong answer.

Jev has practical limits of its own. By default it reads at most 32,000 tokens per question and 64,000 per request (TypeSafe), and Cloudflare and Vercel list it at 32,000. An opt-in mode raises both to 262,000, but TypeSafe itself warns it’s “slower and less accurate.” That rules out handing it an 80-page claims file and asking for a decision, which is why the evidence still has to be prepared and broken into requirements first, the work I described in Prompt Engineering Is Finding the Seams. We also don’t know what Jev is. TypeSafe describes only “a new model architecture, a parallel sampler, and a training method.” Early guesses held that it was a small BERT-style encoder, which would have meant giving up most of what frontier pretraining buys. The outside evidence points the other way. Working from Archer Hume’s reconstruction, built on roughly 10,000 API probes, one detailed analysis concluded that Jev’s base is “almost certainly a pretrained causal transformer,” the same family as frontier LLMs, reading the state once and answering each question from it (deep dive). Its answers also shift with the order of the options, which fits that picture. Until TypeSafe publishes more, how much frontier intelligence sits underneath it is an inference, not a fact.

There’s also the question of why. Jev returns, in DataCamp’s phrase, “a number, not a rationale.” For routing a support ticket, that’s fine. For anything someone has to sign, a coverage decision, a compliance finding, a hiring screen, a number isn’t enough. The decision has to cite the evidence it rests on and show how it got there. That’s the problem we built PWTL around at GARDE1: AI that chooses rather than writes, and shows its work when it does.

The honest counterargument is that decision models are only half of a system. They can’t write the reply, generate the code or explain themselves, and the vendors say as much. The pattern most of them recommend is to let a decision model route and gate the workflow, then let a language model write whatever has to be written. I think that’s right, and it’s also the point. Writing and deciding were always different jobs. We just had one kind of model, so we asked it to do both, and then built two years of scaffolding to make its writing look like deciding.

Variety is a feature of writing. In a decision, it’s a bug.