I ran the same annotation audit three times with three large language models. The defect rate moved by 2.9×. Here is why a single-judge number is not a measurement.
DROID annotation audit · 72,851 episodes · 3 judges. Replication code and data are on GitHub — every figure below is reproduced by analyze.py from the released judge outputs.
There is a tempting move in dataset quality work: point a capable language model at your data, give it a careful rubric, and report back “X% of this dataset is defective.” The number looks authoritative. It came from a frontier model reading every example. I ran an experiment, and the only conclusive thing I found is that a defect rate, taken from any single judge, tells you at least as much about the judge as about the data.
Setup
DROID is a large robot-manipulation dataset where each episode is labelled by three people with a short natural-language instruction. Those three descriptions are supposed to describe the same video. Often they don’t. One says “push the cup left,” another “push it right”; one says “pick up the marker,” another “pick up the marker and put it in the cup.” I built an LLM-as-judge pipeline to flag two failure modes: direction conflicts (incompatible spatial claims) and truncation (one annotation describing only part of what another describes). One rubric, one schema, ~73,000 episodes judged by every model.
I ran the whole thing three times with GLM, DeepSeek, and Mistral, and compared the data. I have a few observations.
1. The rate reported is a property of the judge LLM
Had I run only one model and reported its conflict rate, here’s the number I’d have published — depending entirely on which model I happened to choose:
A 2.9× swing is not a rounding difference. If you’re deciding whether a dataset is clean enough to train on, or which labs’ data to trust, the model you reached for first sets the answer. That should bother you.
2. The honest objection: is this just noise?
The obvious rebuttal is that the models are simply unreliable or guessing, and three guessers naturally disagree. If that were true, none of this would be interesting. So I tested it: I shuffled each model’s verdicts while keeping its overall rates fixed, simulating three judges with the same biases but no shared signal, and measured how much agreement survives.
The models agree far more than chance. Cohen’s κ says the same without the theatrics: every pair lands between 0.40 and 0.49, which is chance-corrected “moderate.” There is a real signal here. Which makes the next finding the uncomfortable one.
3. The agreement is real — and it’s hiding where it’s cheap
The comfortable “45% unanimous” number is dominated by the easy calls. Nearly 60% of all three-way agreement is the models jointly deciding an episode has no direction language at all — agreeing there’s nothing to judge. Strip those out, then narrow to the episodes where at least one model actually cried foul, and the agreement collapses:
Noise averages out; add more judges and random errors cancel. This doesn’t, because it isn’t random: it’s three systematically different thresholds for what counts as a defect. Adding a fourth model of the same class won’t converge them; it’ll add a fourth threshold. On the episodes that matter, roughly three out of four have a dissenter.
The single sharpest disagreement — 8,204 episodes are ones GLM calls perfectly clean and DeepSeek calls a conflict. It’s a flat contradiction on whether a defect exists at all. One model’s pass is the other’s failure.
4. What actually survives consensus
If you’re going to act on an audit, the only defensible unit is the set of flags that survive all three judges agreeing. Here’s how far the defect pile shrinks when you demand consensus:
| Consensus bar | Conflict | Truncated |
|---|---|---|
| ≥1 judge flags | 27,175 (37.3%) | 34,283 (47.1%) |
| ≥2 judges agree | 15,044 (20.6%) | 21,738 (29.8%) |
| All 3 agree | 6,051 (8.3%) | 12,648 (17.4%) |
| consensus ÷ any-flag | 22% | 37% |
Truncation holds together noticeably better than conflict — the judges share more of a definition there. But for both axes the lesson is the same: the consensus threshold is a dial, and wherever you set it becomes your headline number. There is no single true rate falling out of this.
5. Conclusion
I am deliberately not claiming DROID’s “real” defect rate. Everything above measures whether the judges agree, not whether any of them is right. Those are different questions, and without a hand-labelled gold set I can only answer the first. High agreement over chance proves the models are reacting to something real in the text. It does not prove they’re reacting to the right thing. Three models sharing a blind spot would produce numbers that look exactly this clean.
So this is a negative result: a defect rate from a single LLM judge is not a measurement of your dataset — it’s a measurement of that judge applied to your dataset. The two only coincide if you’ve checked, and checking needs ground truth.
If you’re auditing a dataset this way: Report a range, not a point — run at least two judges and quote the consensus band. Treat truncation-style agreement as firmer than conflict-style. And before you trust any of it as an accuracy claim rather than an agreement claim, label a few hundred episodes by hand. That small gold set is what turns “these judges disagree” into “this judge is wrong.”
Method
Three models — GLM, DeepSeek V3.2, and Mistral Large 3 — judged the full DROID annotation set through an identical rubric.py and JSON schema, scoring each episode on direction (conflict / ambiguous / clean / not-applicable) and truncation (truncated / abstraction / clean) with a self-reported confidence. All statistics are computed on the 72,851 episodes common to all three runs (union 74,789; raw counts differ by a handful of dropped or malformed responses per model). Agreement is Cohen’s κ, chance-corrected. The random baseline shuffles each model’s verdict vector independently, preserving its marginal rates, averaged over 20 draws. No ground-truth labels were used — by design, to keep the focus on reproducibility, not correctness.
Pairwise Cohen’s κ
| Pair | Direction | Truncation |
|---|---|---|
| DeepSeek · GLM | 0.423 | 0.455 |
| DeepSeek · Mistral | 0.489 | 0.420 |
| GLM · Mistral | 0.400 | 0.414 |
Full result files and the pipeline are on GitHub.