What an AI Benchmark Score Does and Does Not Tell You

A benchmark score is a compact number: the share of a fixed set of questions that a model answered the way the test’s authors marked as correct. That is useful, but narrower than it looks, and research has documented several ways the number can drift from the ability it is meant to stand for.

What a benchmark measures

Hendrycks and colleagues built MMLU, a widely adopted test, from 57 tasks, covering subjects such as elementary mathematics, US history, computer science and law. Students collected its 15,908 multiple-choice questions from freely available online sources, including exam practice questions.

Their own results show why a single average needs care. In the few-shot setting, the largest GPT-3 model reached 43.9% accuracy, against a random-chance level of 25%. Its performance was lopsided: almost 70% on its best subject and near-random on several others. In zero-shot tests, its confidence was a poor guide to its accuracy, with the two differing by up to 24% on some subjects.

Where a score can mislead

The answer key can be wrong. Northcutt, Athalye and Mueller examined the test sets of 10 commonly used computer vision, language and audio datasets and estimated an average of at least 3.3% label errors, including at least 6% of the ImageNet validation set. On ImageNet with corrected labels, ResNet-18 outperforms ResNet-50 if the share of originally mislabeled test examples increases by just 6%. For MMLU itself, Gema and colleagues had 14 experts re-annotate 5,700 questions across all 57 subjects, estimated that 6.49% of MMLU questions contain errors, and found that re-scoring leading models on the corrected questions changed their ranking.

The model may have seen the test. Sainz and colleagues note that because models are trained on text crawled from the internet, it is very hard to know whether a given benchmark was in their training data, especially for closed models with little documentation. A model trained on a benchmark’s test split will have its performance on that benchmark overestimated, which they argue can lead to wrong scientific conclusions being published. The extent of the problem is unknown, they add, because it is not straightforward to measure.

The layout of the prompt matters. Sclar and colleagues changed only meaning-preserving formatting, such as separators, spacing and capitalization, and found accuracy differences of up to 76 points for one open model, LLaMA-2-13B, in few-shot settings, and about 10 points on average across more than 50 tasks and several models. HELM reported a similar effect from how multiple-choice questions are posed: one 175-billion-parameter model scored 79.1% on the HellaSwag benchmark when each answer choice was presented in its own prompt with no worked examples, and 30.2%, almost random, when the choices appeared together, exam-style, in a prompt with five examples. The Cognitive AI lab page walks through the prompt-format result in an explainer.

Benchmarks wear out. Kiela and colleagues observed that models now routinely surpass estimates of human performance on new benchmarks within a few years, yet still fail on simple challenge examples; GLUE saturated within a year. The HELM authors cite Strathern’s Law, “When a measure becomes a target, it ceases to be a good measure,” because benchmarks give researchers an incentive to build models that improve on them.

Other ways to measure

HELM measures seven things (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency) rather than accuracy alone; before HELM, the models it studied had on average been evaluated on just 17.9% of its core scenarios. The natural language processing program page summarizes that work. Dynabench collects test data dynamically: annotators try to write examples that a current model gets wrong but another person would not. Chatbot Arena shows a visitor answers from two anonymous models and asks which is better; it had gathered over 240,000 votes by January 2024. Its authors expect their users to be mostly AI hobbyists and researchers, and note that prompts from its chat interface may not reflect use in production or specialized domains.

NIST’s AI Risk Management Framework warns that measurement approaches “can be oversimplified, gamed, lack critical nuance, become relied upon in unexpected ways, or fail to account for differences in affected groups and contexts,” and that measurements made in a laboratory or controlled environment may differ from risks that emerge in real-world settings.

INSTAR’s research interest

INSTAR’s interest is in the questions a careful reader can ask before trusting a score. What exactly was tested, and who checked the answer key? Could the model have seen the questions before? Would an equivalent prompt, laid out differently, change the result? Is the score reported as a range across plausible prompt formats, as Sclar and colleagues suggest, or as a single figure?

These questions apply to any benchmark claim, including claims on this site. The page on how this site checks a fact describes the checks applied to the sources cited here.