Four verification protocols, one model, a labelled corpus, matched call budgets. Run them head to head and read the numbers. The interesting metric is not accuracy — it's how often each one panics at a safe action.
Three adversaries fan out from the proposed action, each on a different prior. When one reports, its objection streams down the spoke into the arbiter, and the ring around it fills to the severity it assigned. The arbiter only ignites once all three have landed. Everything here is driven by live run state — start a probe or a bench and watch it move.
Never generated a candidate, so it has no position to defend. It weighs the three objections on merit and is explicitly told not to seek consensus.
One ring full and two near-empty is the signal worth acting on: a single channel found something the others missed. Three half-full rings usually means nobody found anything.
Every multi-call protocol below costs exactly four calls, so protocol is the only variable. All arms hit the same model, which removes provider diversity as a confound. This is a controlled comparison of coordination schemes, not a leaderboard of models.
Stored only in this browser. Never ship a page that asks for a key — use the server proxy for anything public.
Held = flag or block. A protocol that holds everything scores perfect detection and is useless — read detection and false alarm together, or read F1.
| Arm | n | Accuracy | Detection | False alarm | Precision | F1 | ECE | Calls |
|---|---|---|---|---|---|---|---|---|
| No trials recorded — run the bench. | ||||||||
Every panel trial is one point at its three severities. Green matched ground truth, red didn't. Drag to rotate.
Confidence on x, observed accuracy on y. On the diagonal is honest; below it is overconfident.
If σ runs higher on errors, dispersion is a usable error signal.
The aggregate hides the mechanism. Run a single item through the adversarial panel and read what each channel actually objected to, and what the arbiter did with it. The reactor above animates this run.
—
Select an item and run the probe.
Five genuine violations, five safe. Three of the safe ones are deliberately alarming — enormous sums, mass deletion, bulk send — but every one is fully authorised and sourced in its own context. A protocol that pattern-matches on scariness fails those three, and that failure is the entire reason they are here.
| ID | Action | Truth | Defect under test |
|---|
Multi-agent results are routinely reported against a single-call baseline, which confounds protocol with compute. Every multi-call arm here is four calls, so a win is a win on coordination.
Across nine benchmarks, most debate frameworks failed to beat chain-of-thought and self-consistency while spending far more. If the panel can't clear plain resampling, it has earned nothing. ICLR 2025
Consensus-seeking debate converges near the average of its members rather than the best, suppressing the disagreement that carried the information. It's here as an ablation you can reproduce. Failure modes, 2025
It sees three objections and the action, but produced no candidate itself, so it has no position to defend — the asymmetry that separates this from symmetric debate.
Provider heterogeneity would improve absolute numbers and destroy the comparison. Holding the model fixed is what makes protocol the independent variable.
Ten items is small; treat differences under a few points as noise and use repeats. Labels are authored, not adjudicated by a third party. Sampling is stochastic — the same arm will not score identically twice.