BenchMIRT splits a single benchmark score into the capabilities actually driving it

A benchmark score is normally read as one number standing for one ability. BenchMIRT, introduced in a Hugging Face blog post by Kyle Wiggers, works one level down: it looks at how models perform on each individual question and estimates which underlying capabilities are most closely associated with getting that question right. The post frames the problem with BBQ, a benchmark meant to test reliance on social stereotypes, whose Uber question about a grandson and grandfather also requires tracking who is who and reasoning from the evidence given.

The method takes cues from Item Response Theory, the psychometric technique built on the idea that not every question tells you the same amount about a test taker — some are harder, some distinguish stronger performers better. Earlier work, including AI2's Fluid Benchmarking, applied single-dimensional IRT to individual benchmarks. BenchMIRT extends that to multidimensional IRT, so multiple capabilities can contribute to the same questions, and applies IRT at both the model level (a model's strength on the capabilities reflected across the selected benchmarks) and the question level (difficulty, and how well a question distinguishes stronger from weaker models).

It was trained on benchmarking results from 100 LLMs across 16 benchmarks and more than 34,000 questions. Six of those benchmarks measure general reasoning, including MMLU-Pro, GPQA, MATH and BBH; the other ten come from the Olmo 3 safety suite, including HarmBench, StrongReject, WildJailbreak, BBQ, WMDP and XSTest. The post states that BenchMIRT was not told which benchmarks measured which capabilities, and that it independently recovered two dominant dimensions — safety and general reasoning — with the same two emerging when the analysis was repeated from scratch.

The findings that matter for anyone quoting a benchmark number: BBQ, commonly grouped with safety benchmarks, aligned much more strongly with general reasoning, so a low score may partly reflect difficulty understanding or reasoning through the questions rather than safety behaviour. WMDP, which tests dangerous dual-use knowledge, also tracked general reasoning more than safety — and because the benchmark counts refusing or failing to supply that knowledge as the desired response, stronger general reasoning was associated with lower WMDP scores. HarmBench mixed signals within one benchmark: its standard and contextual prompts aligned more closely with safety, its copyright prompts with general reasoning.

The post is explicit that these findings do not necessarily mean the benchmarks are flawed or incomplete, only that a single score can combine several signals. Treat the result as method-plus-one-lab's-analysis: the benchmark selection and the repeat run are AI2's own, and no external replication appears in the evidence here.