Your eval score is a distribution

Run the same evaluation twice and you will often get two different scores. Neither is wrong. An eval samples questions from everything you could ask, and answers from everything the model could say. The single number on the leaderboard hides both.

The hiding matters most when numbers are close. A one-point gain between two checkpoints reads like progress. Whether it is depends on how far the score would move if nothing about the model had changed at all. The four figures below work through that spread. Three are small simulations and one computes its curves exactly; all run in the page, so you can push on them.

# One number

A benchmark score is an average. On each question the model is right or wrong, and the score is the fraction it gets right. Two separate kinds of chance feed into that fraction.

The first is which questions were asked. A benchmark is a sample from a much larger space of questions you care about: other bugs to fix, other maths problems in the same style. Swap in a different set of questions and the score shifts, even for a model that always answers the same way.

The second is which answers came back. Sampled at a non-zero temperature, a model can answer a question correctly on one run and fail it on the next. Greedy decoding narrows this but may not remove it, since batched inference on accelerators is not guaranteed to be bit-for-bit reproducible. Measure it rather than assume it away.

The two sources call for different fixes. More samples per question shrink the second. Only more questions shrink the first.

# Run it again

Start with the answers, because you can see their effect by doing nothing more than running the eval again. Fig. 1 holds its 50 questions fixed. Each question has its own probability of being answered correctly, and every run flips one weighted coin per question.

Fifty fixed questions, one weighted coin each per run. The strip shows the latest run question by question; the dot plot keeps every run’s score.

Press Re-run a few times and watch the strip. Most cells stay put. The ones that flip sit in the middle, where the model’s chance on that question is nearer a coin toss than a certainty. Each question adds p(1 − p) to the variance of the total, a term that peaks at p = 0.5 and vanishes at 0 and 1.

Mixed difficulty gives less run-to-run spread than every question being equally hard, because easy questions almost always pass and hard ones almost always fail.

Add those terms, take the square root and divide by the number of questions, and you have the expected run-to-run standard deviation of the score. For the questions in Fig. 1 it is 5.4 points. Switch to Uniform, where every question has the same 0.72 chance, and it rises to 6.3. The average difficulty barely moves. What changes is that every question is now a little uncertain, where before most were settled.

So a single run of a 50-question eval can land several points from its own long-run average. Averaging several answers per question shrinks that by the square root of their number: with four answers to each question, the 5.4 points become 2.7. If the budget allows, do that and report the mean with its spread. If it doesn’t, say that the number came from one run.

# How many questions

Now hold the answers still and change the questions. Suppose the model’s true accuracy, over every question you might have asked, is p. A benchmark of n questions samples that space, so its count of right answers is binomial and the score’s standard error is the square root of p(1 − p) / n.

With 200 questions and a true score of 72%, the 95% interval runs from 65.4 to 77.8%.

The band is the 95% interval around the true score. Each line below is one simulated benchmark of the same size with its own interval; lines that miss the true score end in crosses.

Drag either number in the paragraph above the figure, or use its sliders. Then draw a few sets of benchmarks and count the crosses. Roughly one line in twenty misses. That is what 95% means here: the procedure catches the true score 19 times in 20 across repeated benchmarks. It is a statement about the method, not a probability attached to any one interval after the fact.

Wilson rather than the normal approximation: it stays inside 0–100% and behaves near the ends.

The table shows how slowly certainty arrives. Halving the interval takes four times the questions: going from 50 questions to 200 halves it, and so does going from 500 to 2,000.

Even 2,000 questions leave about two points either side at this accuracy. A single score from a benchmark that size cannot place the model more precisely than that, whatever the number of decimal places on the leaderboard.

One detail is reassuring. If each question is answered once, this interval already contains the coin flips from Fig. 1. For a question picked at random, a single right-or-wrong answer has variance p(1 − p) whatever mix of easy and hard questions produced it. Re-running a fixed set, as in Fig. 1, measures only the answer noise. A fresh set of questions measures both.

The table assumes the questions are independent of each other. Many benchmarks group them: five questions about one passage, or a dozen tasks built on one code base. Questions in a group tend to succeed or fail together, so the benchmark holds less information than its count suggests, and the honest interval is wider than the table’s.

# At least once or every time

So far each question got one answer. Many evaluations give the model several attempts, and there are two natural ways to summarise k of them. They answer different questions.

The first asks whether the model succeeds at least once. If each attempt succeeds independently with probability p, that chance is 1 − (1 − p)^k, known as pass@k. It fits tasks where something cheap can pick out the good attempt, such as code that must pass a test suite. Chen et al. (2021) used it to evaluate code models and gave the unbiased estimator still in use.

The second asks whether the model succeeds every time: p^k, written pass^k. Yao et al. (2024) use it in tau-bench to measure whether an agent completes the same task reliably across repeated trials, which is closer to what someone depending on that agent will see.

Both summaries of k attempts, for k from 1 to 20. Drag along the plot to move k.

The curves part quickly. Take a model that succeeds with probability 0.90 per attempt and give it 8 attempts. The chance that at least one succeeds is 100.0% to one decimal place; the chance that all of them do is 43.0%. An agent that runs the same workflow every day lives on the second number.

The same arithmetic governs one long task. If each step of an eight-step task succeeds 90% of the time and the steps fail independently, the whole task finishes cleanly only 43.0% of the time. Reliability per step has to be very high before a long chain of steps looks reliable.

In practice p is unknown. Chen et al. draw n answers per problem, with n at least k, count the c that are correct, and estimate pass@k as 1 − C(n − c, k) / C(n, k): the chance that k answers picked from the n without replacement include a correct one, where C(n, k) counts the ways to choose k things from n. The same reasoning gives C(c, k) / C(n, k) for pass^k.

Plugging c/n straight into the formulas is tempting and biased. Show the trick in Fig. 3 to see by how much. With 18 of 20 samples correct, (0.9)⁸ gives 43.0%, but the unbiased estimate is 34.7%. Raising a noisy estimate to the eighth power magnifies its errors, and lopsidedly, so the plug-in overstates consistency.

# Two models

Most of the time the point of an eval is a comparison. Is B better than A? Suppose both answer the same 300 questions. A gets 210 right (70.0%), B gets 223 (74.3%), and B − A is +4.33 points.

The common move is to give each score its own interval and look for overlap. A little better is to treat the two scores as independent samples and put an interval on the difference. That interval is ±7.16 points, comfortably including zero.

The scores are not independent. Both models saw the same questions, and a question that is hard for one tends to be hard for the other. Look at the questions one by one. On 237 the models agree, and agreement says nothing about which is better. The comparison rests entirely on the 63 where they disagree: 25 only A gets right, and 38 only B does.

Taking the per-question difference d = B − A, which is −1, 0 or +1, as the data gives the paired interval: ±5.17 points, using the sample standard deviation of d with n − 1 in the denominator. Same answers, an interval 28% narrower.

Even so, the paired interval crosses zero, running from −0.84 to +9.50. McNemar’s test, which looks only at the disagreements, agrees: with the usual continuity correction, chi-squared 2.29, p = 0.13. That is still a useful result: these 300 questions cannot separate the two models yet. If the rates of disagreement held steady, roughly 430 questions would let the paired interval clear zero, and only just.

Fig. 4 simulates the same set-up, two models with true accuracies of 70% and 74% on 300 shared questions, but draws its own sample, so its counts differ from the worked example’s.

Three hundred shared questions, each split into A’s answer (left) and B’s (right). Only the outlined cells, where the models disagree, inform B − A.

In the figure, shared difficulty sets how strongly the two models’ successes go together. Raise it and the disagreements thin out: the paired interval narrows while the unpaired one barely moves, because the unpaired calculation never looks at which questions were shared. Lower it to zero and the two intervals nearly coincide. Pairing buys precision in proportion to how much the two models’ mistakes overlap.

# What to report

A score worth comparing arrives with enough context to rebuild its error bar. In a write-up or a model card, that means:

• the number of questions, and how they were chosen;
• the number of samples per question, and how they were combined;
• decoding settings: temperature, top-p, maximum length and the prompt template;
• the interval and the method behind it, such as Wilson, a bootstrap or a standard error;
• for any comparison, the paired difference with its interval, not two separate scores;
• seeds, so someone else can run it again.

If you can, publish the per-question results as well. They let anyone run a paired comparison against their own model without repeating your eval.

Reporting all of this costs little; Miller (2024) covers the same ground in more depth, including clustered standard errors for questions that arrive in related groups.

Next time two numbers sit side by side on a leaderboard, ask for the paired difference. If nobody has computed it, the comparison has not been made yet.