We Put a Hallucination Detector on a Live Quiz Generator. It Caught 6 of 9.
Two players wrote in to say the answers were wrong. They were right. This is what happened when the questions started being checked by a second model before anyone played them — including the one defect the check cannot see, which is the part most write-ups leave out.
TL;DR
- Sixteen questions graded by hand against sources. Nine were defective.
- A gate on min(answer score, explanation score) caught 6 of the 9, with zero false positives.
- It cannot see a single fact flipped inside a coherent story. One scored 0.77 and went straight through.
- Threshold came from a sweep, not the docs: 0.35 caught one defect, 0.6 caught six, 0.7 started eating good questions.
- It fails open on every path, costs about 900ms on a round of twelve, and it does not make a round correct.
The two rounds that started it
AI-written quiz questions are mostly fine, which is exactly what makes the failures expensive: nobody is checking, because ninety per cent of the time there is nothing to find. The signal that something was wrong did not come from monitoring. It came from two people who played a game and said the answers were wrong.
Both sessions were then pulled up and graded question by question against sources. Not sampled — every question, by hand.
That second one is the interesting failure, and we will come back to it. A single wrong premise did not produce one wrong question; it produced a coherent, internally consistent round that was wrong all the way through.
What the second model is actually asked
The obvious approach is to ask a language model “is this question correct?” and read the reply. That gives you a paragraph to parse and a confidence you invented yourself. What is used here instead is a model built to answer typed questions about a piece of state and return probabilities rather than prose — TypeSafe's Jev, the first of what they call System One models. Two questions per quiz question, both boolean:
- Is the marked answer factually the correct answer? The state it sees is the question, the options, and which option the quiz has marked correct.
- Is every factual claim in the explanation true? Asked only when there is an explanation. Asking about one that is not there invites a guess, and the guess comes back as a number the gate would then act on.
The gate is applied to the lower of the two. That was not a design preference, it was the measurement: either signal on its own performed worse. The 100-Skulltula question scored 0.77 on the answer and 0.45 on the explanation — the answer score alone would have passed it. The Apple Cup chronology scored 0.27 and 0.32.
Across all 16 graded questions, gating on the minimum caught 6 of the 9 defects and flagged nothing that was correct.
The threshold is the whole product
A confidence number is useless until you decide what to do at each value, and that decision cannot be taken from anyone's documentation — it is a property of your data. Three settings were swept against the graded questions:
0.6 is not a round number someone liked. It is the widest setting that had not yet started costing anything. If you are wiring up a gate like this, the sweep is the work — the integration is an afternoon.
The failure it cannot see
“Washington won 40–13” scored 0.77 and passed the gate. The score was real, the coach was right, the running back was right. Only the winner was swapped.
This is the honest limit of the whole approach, and it generalises past quizzes: a confidence signal detects uncertainty, and a confidently held wrong premise does not read as uncertain. There is nothing in that sentence that looks false. The questions built on top of it looked even less false, because they were consistent with the premise.
The sentence worth keeping: this gate lowers the error rate on a round. It does not make the round correct. Two thirds of known defects, and blind to the third that is a coherent falsehood. Any tool in this category that is described as “verification” is being oversold, including this one if we described it that way.
Four rules for putting a second model in a user's path
These are the parts that took longer than the integration, and they are the parts that transfer to any pipeline where one model checks another's output in a request a person is waiting on.
Fail open, on every path
Timeout, non-200, malformed body, missing key, missing field — every one of them returns the round untouched.
Silence from a third-party API is not evidence against a question. The alternative is that their outage becomes your empty round, which is a worse product than an occasional wrong answer.
Over-ask, then trim
With the gate on, the generator asks the writing model for 1.5x the questions it needs and trims back to the target afterwards.
A gate can only drop what there is a spare of. Ask for exactly eight, drop two, and the round is short — the player sees the gate. The two graded sessions were 4-of-10 and 5-of-6 defective, so bad rounds are bad in bulk, and 1.25x would not have covered it.
Pin the model id
The request names an exact published model, not a moving "latest" alias.
Every threshold in this post is a property of one model version. A silent upgrade upstream moves the number the gate is built on, and nothing in the logs would say so.
Give it a budget it cannot exceed
Six-second hard timeout per call, four calls in flight. A round of twelve comes back in about 900ms.
Generation takes about seven seconds. A verification pass is worth roughly one of those. Anything that cannot promise that belongs in a background job, not in the path a host is waiting on.
There is a fifth, less flattering one. If the gate drops everything across every attempt, the ungated round is played anyway, with a warning in the logs. A player who gets an empty screen has a worse evening than a player who gets a wrong question about Hyrule, and pretending otherwise is how a safety feature becomes an outage.
What it costs
Generating a round of eight questions takes just under seven seconds. A verification pass was budgeted at roughly one of those, and four calls in flight brings a round of twelve back in about 900 milliseconds. The per-call timeout is six seconds, which is less a latency target than a promise that a slow dependency cannot hold a lobby open.
The gate also costs questions, which is why the generator asks for 1.5× what it needs whenever the gate is on and trims back afterwards. Dropping two questions from a round of twelve is invisible; dropping two from a round of eight is a product the host can see going wrong.
Questions people ask
What is actually being detected here?
Not "hallucination" in the abstract — two specific, typed questions about one quiz question. First: is the answer the quiz marks as correct actually the correct answer? Second: is every factual claim in the one-line explanation true? Both come back as a probability rather than as prose, which is the part that makes it usable in code: there is nothing to parse and nothing to regex out of a paragraph.
Why gate on the minimum of the two scores?
Because either signal alone was measurably worse. A question about a reward in Ocarina of Time scored 0.77 on the answer and 0.45 on the explanation; a question about a football chronology scored 0.27 and 0.32. Taking the lower of the two caught 6 of the 9 known defects with zero false positives across all 16 graded questions. Taking the answer score alone would have let the first one through.
Where did the 0.6 threshold come from?
A sweep, not a guess. At 0.35 the gate caught 1 defect of the 9. At 0.6 it caught 6 and still flagged nothing that was correct. Above 0.7 it starts eating real questions: across 151 questions from graded sessions, a 0.7 floor flagged 11 of 25 known-good pairs. 0.6 is the widest setting that had not yet started costing anything.
What does it not catch?
A single fact flipped inside an otherwise coherent story. A question asserting that Washington won the 2021 Apple Cup scored 0.77 and sailed through — Washington State won 40-13. A football result with a plausible score, the right coach and the right running back has nothing in it that reads as false. Four further questions in that session were built on top of that one flipped fact, which is the real shape of the problem: the model was not unsure, it was confidently working from a wrong premise.
So does this make the questions correct?
No, and that is the sentence worth keeping. It lowers the error rate on a round. It caught two thirds of the known defects in the sessions that were graded by hand, and it cannot see the third that is a coherent falsehood. Anyone selling a gate like this as "verified" is describing a different product than the one that exists.
Is a calibrated probability the same as a correctness guarantee?
No. Calibration is a property of groups of predictions — across many answers, the stated probabilities line up with how often those answers are right. It says nothing about whether one individual answer is correct. That distinction is why the threshold had to be swept against graded data instead of picked from the documentation.
Try to break it
The most useful thing you can do with this post is falsify it. Pick a topic you know properly — a game, a season, a discography — generate a round, and read it against what you know. The failure mode described above is still in there somewhere.
Generate a QuizFree to host • No account for players • Up to 20 in a free room
The generator on the other side of this gate is described in AI Quiz Generator: how it works and where it fails.