Decision Models and Jev

Most AI work is choosing, not writing

Chatbots made AI famous by writing. But look inside almost any company’s software and the jobs you would hand to AI are overwhelmingly choices: which queue gets this ticket, is this payment fraud, may the agent run this command, does this passage answer the question.

Each of those is a decision with a small, known set of answers. Software wants the answer as a value it can branch on, not a paragraph. And it wants to know how sure the model is, because “billing, 97% sure” and “billing, 51% sure” should lead to very different actions.

This lesson is about models built for that job. It starts with old ideas that matter more than ever (discriminative models, calibration, cost-sensitive thresholds) and ends with one of September 2026’s biggest AI stories: Jev, a model from TypeSafe AI that does not generate text at all and instead returns typed decisions with calibrated probabilities.

What is a decision model?

A decision model takes some input and returns one answer from a set you define in advance, along with a probability or confidence. The answer can be yes or no, one category from a list, or a score on a scale. What it never returns is free-form text, so there is nothing to parse and nothing outside the list it can say.

How does it fit into a system?

The model makes the judgement; your code keeps control. Code builds the input, asks the question, reads the typed answer and its probability, and then decides: act automatically, ask a person to confirm, or escalate. The whole pipeline is testable like any other function call, which is exactly what is hard to achieve with a chatbot in the loop.

Why does it matter now?

Two trends collided in 2026. Teams building agents discovered that most agent steps are small decisions (which tool? is this done? is this safe?), and that paying a large language model to write out every one is slow and expensive. At the same time, the gap between how confident models sound and how often they are right became a practical problem for automation. Decision models target both.

Key takeaways

• Much real-world AI work is picking from a known set of answers, not generating text.
• Software needs the answer as a typed value plus an honest measure of certainty.
• Decision models return exactly that, leaving the final action to ordinary code.

Generative and discriminative models

There are two classic ways to build a classifier. One learns how each class produces data and works backwards. The other learns the boundary directly. The difference explains a lot about 2026’s model landscape.

What are the two families?

A generative model learns P(x | y), how data looks within each class, plus how common each class is. To classify, it applies Bayes’ rule to flip that into P(y | x). Naive Bayes is the textbook example; a large language model is a generative model of text.

A discriminative model learns P(y | x) directly: given this input, how likely is each answer? Logistic regression is the textbook example, and so are most image classifiers and, in spirit, decision models like Jev. It never models what inputs look like, only how to tell answers apart.

Generative: model the data for each class, then invert with Bayes’ rule. Discriminative: fit the right-hand side directly.

How do they differ in practice?

A generative classifier is only as good as its story about the data. If you assume each class is a bell curve and one class is actually skewed, the posterior you derive can be badly wrong, even in regions you have plenty of data for. A discriminative model makes fewer assumptions about the inputs, so it is more robust to that kind of mistake. The flip side, shown by Ng and Jordan, is that generative models can reach their best performance with fewer examples, while discriminative models tend to win once data is plentiful.

With both classes Gaussian, the generative model’s assumption is correct and it tracks the truth closely even with few examples. Give class B a cluster of outliers and the Gaussian fit to B balloons; Bayes’ rule then predicts class B again far to the left, where there is no class B data at all. Logistic regression never makes that mistake because it never modelled the data’s shape.

Why does this matter for decision models?

Chat LLMs are generative: trained to produce likely text, then adapted to be helpful. You can coax a classification out of them, but the label is a by-product of writing. A decision model is discriminative by design: its output is the distribution over your answers and nothing else. That makes it natural to train and measure the thing you care about, whether its probabilities match reality. For the generative side of the story see Generative AI; for classic discriminative methods see Machine Learning.

Key takeaways

• Generative models learn how data is produced and use Bayes’ rule; discriminative models learn P(answer | input) directly.
• Wrong assumptions about the data can distort a generative classifier’s probabilities even where data is plentiful.
• Decision models are discriminative by design, so their probabilities are the product, not a by-product.

Why not just ask a chatbot?

The quickest way to build a classifier in 2026 is to ask a chat model “Which of these categories is this?”. It often works. It also brings four problems that get worse at scale.

What goes wrong?

Parsing. The answer arrives as text, so you need code to turn “Billing.” or “**Category:** Technical” into a label. Structured output modes help a lot, but you are still generating tokens and validating them afterwards.

Cost and latency. Every label is produced by sampling tokens one at a time from a large model, often after the model writes some reasoning first. That is fine for ten tickets a day and painful for ten thousand agent steps a minute.

Confidence. If you ask the model how sure it is, the number is just more generated text. Xiong et al. found LLMs tend to be overconfident when they verbalise confidence. Token probabilities are a better signal, but chat tuning distorts them too: the GPT-4 technical report shows the pre-trained model was well calibrated on a multiple-choice benchmark and that post-training “hurts calibration significantly”.

Consistency. Sampling means the same input can get different answers, and a small prompt change can shift behaviour in ways that are hard to test.

No parser wins. Exact matching is safe but brittle. Keyword search reads “Not billing… this is an account issue” as billing, the worst outcome because it fails silently. JSON parsing works only when the model complies, and even then its “confidence”: “high” is not a number you can threshold. A decision model avoids the whole row of problems because the answer space is fixed before the call.

Key takeaways

• Using a chat LLM as a classifier means parsing text, paying for tokens and trusting a confidence that is often inflated.
• Silent mis-parses are more dangerous than loud failures.
• The trade-off is flexibility versus control; decision models try to keep the flexibility and add the control.

Calibration: probabilities you can trust

A weather forecaster who says “70% chance of rain” is well calibrated if it rains on about 70% of those days. The same test decides whether a model’s probabilities are worth anything to software.

What is calibration?

A model is calibrated when, among all the cases it gives probability p, the answer is “yes” about p of the time. Calibration is separate from accuracy. A model can be 90% accurate and wildly overconfident, or it can be perfectly calibrated and useless (always predicting the base rate is calibrated). You want both: sharp predictions that are also honest.

How do you measure it?

Sort predictions into bins by probability, say 0 to 0.1, 0.1 to 0.2 and so on. In each bin compare the average prediction with the fraction that actually came true. Plot one against the other and you get a reliability diagram; a calibrated model sits on the diagonal.

The expected calibration error (ECE) summarises the diagram as the average gap, weighted by how many cases fall in each bin, as used by Guo et al.

B bins, n_b cases in bin b, N cases in total. A second common score is the Brier score, the mean of (p - y)², which rewards calibration and sharpness together.

Overconfidence has a signature: an S-shaped curve and a histogram piled up at 0 and 1. Change the number of bins and ECE moves even though the model has not changed. That is a real weakness of the metric; report the bin count with any ECE you publish, and look at the diagram, not just the number.

Why does it matter, and can it be fixed?

Every downstream use of a probability assumes it is calibrated: choosing a threshold from costs, deciding when to defer to a person, combining several model outputs. Guo et al. showed that modern deep networks, unlike older and smaller ones studied by Niculescu-Mizil and Caruana, are often overconfident. The good news is that calibration can be repaired after training on held-out data. Temperature scaling divides the logits by one fitted number T; Platt scaling fits a sigmoid with a slope and an offset; isotonic regression fits any monotone curve.

Key takeaways

• Calibrated means: of the cases given probability p, about p turn out positive.
• Reliability diagrams show calibration; ECE summarises it, but depends on the binning.
• Overconfidence is common in modern networks and can often be fixed after training on held-out data.

System 1, System 2 and a borrowed metaphor

TypeSafe calls Jev a “System One model”. The name comes from psychology, and it is worth knowing what the original idea says, and what it does not, before accepting the analogy.

What are System 1 and System 2?

In Thinking, Fast and Slow, Daniel Kahneman describes human judgement as the work of two characters, using terms from psychologists Keith Stanovich and Richard West. System 1 is fast, automatic and effortless: recognising a friend’s face, sensing anger in a voice, reading a word. System 2 is slow, deliberate and effortful: multiplying 17 by 24, filling in a tax form, checking a logical argument. Much of the book is about System 1’s predictable errors, such as overconfidence and substituting an easy question for a hard one.

Why do AI companies borrow it?

The metaphor gives a neat story for two product categories. Reasoning models and agents that generate long chains of intermediate steps are pitched as System 2. A model that reads the input once and returns an answer in a single pass is pitched as System 1. TypeSafe’s documentation says the System One name comes from Kahneman’s concept and that “the emphasis is on fast, focused judgments”. The same framing has been used by researchers for years to describe what deep learning does well (fast pattern recognition) and what it historically struggled with (deliberate multi-step reasoning).

What are the caveats?

First, it is a metaphor. Kahneman himself describes the two systems as useful fictions rather than parts of the brain. A neural network is not doing “intuition” in any psychological sense.

Second, parts of the book have not aged well. A replicability analysis by Schimmack and colleagues found the social priming studies in its fourth chapter were weakly supported, and Kahneman replied that he had “placed too much faith in underpowered studies”. The broad fast/slow distinction survives; some of the famous examples do not.

Third, and most importantly for engineers, the analogy cuts both ways. Kahneman’s System 1 is famous for being confidently wrong. A “System One” model is only useful if it does not share that flaw, which is why the claim that matters is calibration, and calibration can be measured. Treat the name as marketing and the calibration claim as a hypothesis to test.

Key takeaways

• System 1 is fast and intuitive, System 2 slow and deliberate; the terms describe human judgement, not machines.
• Vendors use the pair to position single-pass decision models against slower reasoning models.
• Human System 1 is known for overconfidence, so a “System One” model must prove its calibration rather than assume it.

Jev: what we actually know

Jev is a proprietary model from TypeSafe AI, a San Francisco company founded in 2024. It was released in limited early access on 15 September 2026. Here is what the vendor says, what others have measured, and what nobody outside the company knows yet.

What is it?

Jev returns typed values with probability estimates instead of text, and its output is meant for software rather than people (Wikipedia). You send a block of state (a string, JSON object or array of text) and one or more typed questions, and every question is answered against the state in one parallel pass. The official documentation defines three question types, which it calls primitives:

How was it built?

TypeSafe says Jev is trained with Reinforcement Learning for Calibrated Decisions (RLCD), in which probabilities are optimised against outcomes rather than against human preference, the target that shaped chatbots through RLHF. Its documentation states the goal plainly: across many predictions, outcomes assigned 0.8 should happen about 80% of the time, and it adds that calibration describes groups of predictions, not a guarantee about any single answer.

Beyond that, little is public. According to press coverage summarised on Wikipedia, TypeSafe describes the model as transformer-based and trained only on synthetic data, but has not published its architecture, weights or a technical paper; outside observers have suggested it may be built on an open-weight LLM. Sebastian Raschka speculates it is likely a small encoder-style model and argues that “the secret sauce is probably more in the data than in the training algorithm”. Treat both as informed guesses.

The name refers to the economist William Stanley Jevons, whose paradox says making a resource cheaper to use can increase total consumption. TypeSafe’s bet is that very cheap machine judgement will be used far more widely.

Vendor claims

In its launch post, TypeSafe reports end-to-end responses in 70 to 500 ms and says Jev was 40 to 200 times faster and 40 to 400 times cheaper than frontier LLMs on comparable tasks. The same post is candid about the limits of that comparison: the workflows were written by TypeSafe’s own team, and the reference answers came from frontier LLMs (GPT-6 Astra and Fable 5.1), which makes the score a measure of agreement with those models rather than with ground truth. The price listed in the docs is $0.042 per million input tokens, with output tokens free, and input is text only, up to 64k tokens per request.

Adoption

According to Vercel, nearly 13% of paid teams on its AI Gateway were using Jev within 24 hours of launch, twice the share of the GPT-5.6 family and more than six times Fable 5.1’s over the same period. That is a fact about developer interest, not quality; Vercel also notes Jev is free to use on its gateway until 25 September.

Independent measurements so far

Two community evaluations with published raw data give a first, partial picture. Neither is peer reviewed, and both are small.

priorbench ran 5,721 calls across 21 experiments. On its 400-item benchmark Jev scored 95.9% zero-shot, against 77.2% for hand-written keywords and 66.0% for a supervised TF-IDF plus logistic regression baseline. But it always answered: 0 of 30 out-of-scope messages were flagged, and a cake recipe was classified as a technical issue with 0.94 confidence. Accuracy above a confidence threshold was roughly flat from 0.50 to 0.95 and jumped to 100% only at 0.99.

scienthoon measured ECE of about 0.02 to 0.03 on three public benchmarks, which may overlap with training data, but 0.107 on 900 rule-generated support tickets the model cannot have seen. Miscalibration differed by question type on the same inputs: yes/no answers were underconfident while choice and score answers were overconfident. The author also notes that Jev’s probabilities are rounded to 0.01, often to exactly 0 or 1.

The honest summary: strong zero-shot accuracy on some tasks, calibration that looks good on familiar benchmarks and noticeably worse off them, and not yet enough independent evidence to know which pattern will dominate. That is normal for a nine-day-old model.

Key takeaways

• Jev answers typed questions (Noul, Choice, Score) with probabilities and never generates text.
• Speed, cost and calibration claims come from the vendor; architecture, weights and a paper are not public.
• Early independent tests show strong accuracy but calibration that varies by task and question type, so measure on your own data.

Evaluating a decision model yourself

Whatever a vendor claims, the only evaluation that matters is on your data with your costs. The toolkit is old, well understood and cheap to run.

What should you measure?

Start with a few hundred examples of your real traffic, labelled by people who know the task, including awkward cases and inputs that fit none of your categories. Then measure three things. Accuracy through a confusion matrix, which shows which mistakes the model makes, not just how many (see Evaluating Models and the Confusion Matrix Lab). Calibration through a reliability diagram and ECE on the same set. And behaviour at the edges: out-of-scope inputs, adversarial text, very long inputs.

How do you turn probabilities into decisions?

Here calibration pays for itself. If the probabilities are honest, the best threshold follows from the costs alone. Acting on a case with probability p is worth it when the expected cost of acting is lower than the expected cost of not acting:

C_FP: cost of a false alarm. C_FN: cost of a miss. This is the classic result in Elkan’s treatment of cost-sensitive learning; it only holds if p is calibrated.

The formula comes from Elkan. With a $400 miss and a $20 false alarm, you should block anything above 20 / 420, about 0.048, far below the 0.5 most people reach for.

With a calibrated model, the formula lands on the bottom of the cost curve without any search. Make the model overconfident and the formula’s threshold misses, because p no longer means what the formula assumes; you would have to tune the threshold by trial and error on labelled data, and retune it whenever the model changes.

Why not act on every case?

A system can also abstain: act automatically on confident cases and send the rest to a person. This trades coverage for accuracy, a setting known as selective classification (Geifman and El-Yaniv). TypeSafe’s confidence guide recommends exactly this, with three bands (act, confirm, escalate) and stricter thresholds for riskier actions. Calibration is what lets you pick those bands from a target error rate. The Calibration Lab’s “decide or defer” panel shows the promised and delivered accuracy side by side.

Key takeaways

• Evaluate on a labelled sample of your own traffic, including out-of-scope and adversarial inputs.
• With calibrated probabilities, the optimal threshold is C_FP / (C_FP + C_FN), with no tuning needed.
• Abstaining on low-confidence cases trades coverage for accuracy, and calibration makes that trade predictable.

Decision models in agents, and open questions

The most interesting use of a fast decision model may be inside an LLM agent: the slow, articulate model plans and writes; the fast one checks and routes; code decides.

How do they combine?

An agent makes many small decisions per task: which tool to call, whether a result answers the question, whether an action is safe, whether it is finished. Each can be a typed question with a probability, answered in a fraction of the time an LLM would take to write out its reasoning. TypeSafe’s documentation describes patterns such as intent routing (classify a request and send it to code, a specialist LLM or a person) and guardrails that screen messages going into and out of an LLM app.

What are the limits?

TypeSafe publishes an unusually frank list of known failure modes for jev-1.13. It reads instructions literally; it does not count or do arithmetic reliably (“keep the arithmetic in code”); it compares dates poorly; it struggles with double negatives and multi-hop questions; accuracy falls as the input fills with irrelevant detail; adversarial text in the input can move the answer; and it cannot generate text at all.

One example from that list is worth remembering. Asked “Is the customer asking for a refund?” and “Is the customer asking for something other than a refund?” as two yes/no questions about the same ticket, it returned 0.72 and 0.47, which sum to 1.19. Each answer can be individually reasonable, but separately asked questions are separate predictions, and nothing forces them to be logically consistent.

What is still unknown?

• Calibration under shift. Calibration measured on one distribution often degrades on another, a pattern Ovadia et al. documented across many deep learning methods. Early community tests suggest the same may apply here.
• Knowing what it does not know. A Choice picks the best of the options offered. Without an explicit “none of these” option or a separate yes/no check, out-of-scope inputs get confident answers.
• Transparency. Without a paper, weights or third-party audits, claims about RLCD and training data cannot be checked, and version changes behind an alias can move your thresholds. The docs recommend pinning a version if you tune thresholds against it.
• Scale effects. If Jevons is right, cheaper judgement means far more automated decisions, which raises the stakes of every error. The AI Ethics lesson covers how to check that errors do not fall unevenly on different groups.

Key takeaways

• LLMs plan and write; decision models make fast typed checks; code keeps control with risk-scaled thresholds.
• Keep arithmetic, dates and logical identities in code, and give every Choice an escape option.
• Calibration off the training distribution, transparency and version drift are the big open questions.

Check your understanding

Seven scenarios. Each asks what you would actually do with a decision model, not what a term means.

Your fraud model outputs P(fraud). A missed fraud costs about $300; wrongly blocking a good payment costs about $15 in support time and lost goodwill. The probabilities are well calibrated. Roughly where should you set the blocking threshold?

References and further reading

Research papers first, then sources on Jev. Vendor material (TypeSafe’s blog and docs) and community evaluations (the two GitHub projects) are labelled as such; weigh them accordingly. Sources on Jev were accessed in September 2026.

References

1. [1]

   On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes(opens in a new tab)

   Andrew Y. Ng, Michael I. Jordan, 2001

   NIPS 2001. Generative models can approach their best error with fewer examples; discriminative models usually win asymptotically.

2. [2]

   Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs(opens in a new tab)

   Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, Bryan Hooi, 2024

   ICLR 2024. Finds that LLMs verbalising their confidence tend to be overconfident.

3. [3]

   GPT-4 Technical Report(opens in a new tab)

   OpenAI, 2023

   Figure 8: the pre-trained model is well calibrated on MMLU; post-training hurts calibration significantly.

4. [4]

   On Calibration of Modern Neural Networks(opens in a new tab)

   Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, 2017

   ICML 2017. Shows modern deep networks are overconfident and that temperature scaling fixes most of it.

5. [5]

   Predicting Good Probabilities with Supervised Learning(opens in a new tab)

   Alexandru Niculescu-Mizil, Rich Caruana, 2005

   ICML 2005. Compares the calibration of ten learning algorithms and of Platt scaling versus isotonic regression.

6. [6]

   Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods(opens in a new tab)

   John C. Platt, 1999

   In Advances in Large Margin Classifiers (MIT Press). Introduces fitting a sigmoid to classifier scores.

7. [7]

   Thinking, Fast and Slow(opens in a new tab)

   Daniel Kahneman, 2011

   Farrar, Straus and Giroux. Popularised the System 1 / System 2 description of human judgement.

8. [8]

   TypeSafe AI documentation: System One, Confidence, Models, Jev 1.13 jaggedness(opens in a new tab)

   TypeSafe AI, 2026

   Official docs (accessed 24 September 2026): primitives, API shape, pricing, limits, and a list of known failure modes for jev-1.13.

9. [9]

   Reconstruction of a Train Wreck: How Priming Research Went off the Rails(opens in a new tab)

   Ulrich Schimmack, Moritz Heene, Kamini Kesavan, 2017

   Replicability analysis of the priming studies in chapter 4 of Thinking, Fast and Slow, with a reply from Kahneman in the comments.

10. [10]

    Jev (AI model)(opens in a new tab)

    Wikipedia contributors, 2026

    Summarises press coverage (Forbes, TechCrunch, The Register, SiliconANGLE) of the launch, including what has not been disclosed.

11. [11]

    Introducing System One Models & Jev(opens in a new tab)

    Diogo Almeida (TypeSafe AI), 2026

    Vendor launch post, 15 September 2026. Source of the speed, cost and calibration claims, with the vendor’s own caveats on its evaluations.

12. [12]

    It’s Easy to Dismiss Jev as Just a Classifier(opens in a new tab)

    Sebastian Raschka, 2026

    Independent commentary, 20 September 2026. Speculates on architecture and argues the data is the main ingredient.

13. [13]

    Jev is the fastest-adopted model in AI Gateway history(opens in a new tab)

    Amelia Charles, Harpreet Arora, Eric Dodds (Vercel), 2026

    Vercel blog, 18 September 2026. Adoption figures from Vercel’s AI Gateway.

14. [14]

    Independent, pre-registered evaluation of TypeSafe AI’s Jev(opens in a new tab)

    priorbench (GitHub), 2026

    Community evaluation, 20 September 2026: 5,721 calls, 21 experiments, raw data published. Not peer reviewed.

15. [15]

    Independent calibration test of TypeSafe’s Jev (jev-ood-calibration)(opens in a new tab)

    scienthoon (GitHub), 2026

    Community evaluation on 900 rule-generated support tickets plus three public benchmarks, with ECE and temperature refits. Not peer reviewed.

16. [16]

    The Foundations of Cost-Sensitive Learning(opens in a new tab)

    Charles Elkan, 2001

    IJCAI 2001. Derives the optimal decision threshold from misclassification costs, given correct probabilities.

17. [17]

    Selective Classification for Deep Neural Networks(opens in a new tab)

    Yonatan Geifman, Ran El-Yaniv, 2017

    NeurIPS 2017. Trading coverage for accuracy by letting a network abstain on low-confidence inputs.

18. [18]

    Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift(opens in a new tab)

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, Jasper Snoek, 2019

    NeurIPS 2019. Calibration of many uncertainty methods degrades as the data shifts away from the training distribution.