Quorum

# Things to try

• Press 100 trials a few times. The dot wanders, then settles near the diamond at 90.1%, and the band around it tightens as the tally grows.
• Drop the accuracy to 0.45. Nine reviewers are now right only 37.9% of the time, worse than any one of them alone. A majority amplifies whichever side of a coin flip its members are on.
• Put accuracy back to 0.70 and raise the correlation to 0.3, then 0.6. The panel slides from 90.1% to 84.1%, then to 78.0%. Show the trick and you can see why: shared draws pile probability onto all right and all wrong, and no number of reviewers moves those.

# How it works

Each reviewer is right with probability p, and the panel takes the majority. If their mistakes are independent, the chance that more than half are right is a binomial tail. At 0.70 it climbs from 70.0% with one reviewer to 90.1% with nine. This is Condorcet’s jury theorem, and it cuts both ways: below 0.5, every reviewer you add drags the panel further towards the wrong answer.

Independence is the expensive assumption. Reviewers who read the same evidence, or run on the same model, tend to make the same mistakes. The figure models that bluntly. With a probability equal to the correlation, c, the whole panel copies one draw; otherwise each reviewer judges alone. So the panel is right with probability c·p + (1 − c)·B(n, p), where B(n, p) is the independent majority.

Correlation caps what headcount can buy. However large the panel, it never beats c·p + (1 − c), so at an accuracy of 0.70 any correlation of 1/6 or more (0.17 on the slider) puts 95% out of reach. Even independent reviewers need 17 to get there: fifteen reach 94.999%, just short.

The simulated rate carries a 95% Wilson interval, which stays sensible near 0% and 100%, where the textbook normal interval breaks down.

prime-pstack, one of my public repositories, includes multi-model review panels. This toy is about the arithmetic behind panels in general, not about how prime-pstack works.

# Keyboard