You want to know whether your agent actually did what each user asked.
A large model can read every chat and grade it. But you pay for every grade, all day, and each one takes a while.
Operant gives each conversation an outcome score: did the agent get the task done? Today a large model produces that score. A model that grades your chats like this is called a judge.
So here's a research question: could a small, fast model do your grading instead? This is research, and it isn't part of the product yet.
The four judges
Four judges got the same chats, the same question and the same options:
- Jev is a hosted model from TypeSafe, version
jev-1.13.0, built to pick between options. It writes no text. It answers a typed question with a probability for each option. - Kev-9B is an open version of Jev on Hugging Face. It ran locally on a Mac with
MLX, so no text left the network. - Haiku 4.5 is a small general model.
- Sonnet 5 is a stronger general model.
The question was "Did the agent do what the user asked?" The options were success, failure and unclear.
The chats
The test used 240 runs from tau2-bench, an open benchmark where agents handle airline and retail tasks. Half the runs passed and half failed, by the benchmark's own reward. That reward checks the final state of the task, the database and the policy, not just the words. Each judge read each full chat.
Test 1: small judge versus big judge
A small judge is only useful if it agrees with the large one it would replace. So for the first test, Sonnet 5's verdicts were the answer key. Sonnet gave a verdict on 231 of the 240 runs.
- Agreement: Jev 93.9%, Kev-9B 89.6%, Haiku 4.5 83.5%.
- Balanced accuracy: Jev 89.8%, Kev-9B 79.3%, Haiku 4.5 66.3%.
- Failures caught: Jev 82.4%, Kev-9B 60.8%, Haiku 4.5 35.3%.
- Cohen's kappa: Jev 0.819, Kev-9B 0.683, Haiku 4.5 0.515.
Jev also gives a probability. So you can let it decide only when it's sure, and send your other chats to a large judge. With a floor of 0.7, Jev decided 85.7% of the runs on its own. It agreed with Sonnet on 98.5% of those.
Test 2: what really happened
For the second test, the benchmark reward was the answer key. Against it, every judge was close to a coin flip:
- Balanced accuracy: Jev 49.2%, Kev-9B 42.1%, Haiku 4.5 45.8%, Sonnet 5 47.5%.
- Failures caught: Jev 20.0%, Kev-9B 10.8%, Haiku 4.5 8.3%, Sonnet 5 21.7%.
AUROC: Jev 0.536, Kev-9B 0.465, Haiku 4.5 0.531, Sonnet 5 0.572.
Each judge called most of the 120 failed runs a success. That's because a failed run usually reads fine. The user says thanks, but the agent wrote the wrong change to the database or broke a policy. A judge that reads only the chat can't see that.
Here's one of them, from the public tau2-bench results. A traveler asked the airline agent to remove one passenger from a booking. The airline's rules don't allow that, and they didn't allow cancelling this booking either. The agent cancelled the whole booking anyway, then booked the traveler a new trip. The traveler signed off with "Thanks for your help!" All four judges called the chat a success. The benchmark marked it a failure.
Against the large judge, small Jev agreed on 93.9% of the chats (full table).
The limit is the second test. Against what really happened, every judge, large or small, was near chance (full table). A score from a chat tells you whether your user's goal looks met. It doesn't prove your final state is right. That matters most when a system picks "good" chats to learn from.
The next research step is to run a small judge next to the large one, without changing any score. That isn't shipped. Every number here was recalculated from the study's committed row files.
So a small, fast judge could grade your chats almost as well as a large one.
See every number on the results page.