Your Benchmark Score Is Two Scores in a Trench Coat. BenchMIRT Just Proved It.

The Allen Institute just ran an audit of AI benchmarks that should make every model-buying decision you have made this year deeply uncomfortable. It took 100 LLMs, ran them across 16 benchmarks, and scored more than 34,000 questions. The result is blunt: the two things everyone measures separately, safety and general reasoning, are effectively the only two dimensions that matter, and most single benchmark scores quietly blend them together.

Here is the part that matters. BenchMIRT found that split on its own. Nobody told it which benchmark was supposed to measure which ability. It looked at the raw question-level data and independently recovered two dominant dimensions, safety and general reasoning, and those same two dimensions came back every time the analysis was rerun.

If you have ever picked a model because it climbed a leaderboard, this is the article about what that number actually contains. It is rarely one thing. It is almost always two scores wearing a single number as a disguise.

We covered why your safety benchmark is probably lying to you back in Note #149, "Your Safety Benchmark Is Lying to You." BenchMIRT is the next layer down. It is not just proof that benchmarks can be gamed. It is proof that the scores themselves conflate two capabilities you need to reason about separately, and it gives you the tool to pull them apart.

THE TWO-DIMENSION RULE

State this rule plainly, because you should be able to quote it in your next model selection meeting. Every headline benchmark score is a weighted blend of at least two capabilities: safety and general reasoning. Most teams read it as one number describing one skill. It is not. Treat any single benchmark score as a cocktail until you can prove otherwise.

For the newbies in the room, here is MIRT in one sentence. Item Response Theory is the statistical method standardized tests have used for decades to figure out what each individual question actually measures. Multidimensional Item Response Theory, the MIRT that BenchMIRT sits on, does that for many abilities at once. BenchMIRT applies it to LLM benchmarks prompt by prompt instead of averaging the whole suite into a flat number.

Here is what that buys you. When a vendor leans across the table and says "state of the art," your first question should be: on which dimension, measured how? If they cannot answer that, they do not know what their own benchmark measures. You do now.

WHAT BENCHMIRT ACTUALLY FOUND

The concrete findings are worth walking through, because they are not comforting.

The first is the recovery of the two dimensions. Safety and general reasoning emerged independently from the data and reproduced across repeated analyses. That means the two dominant things a model's benchmark score encodes are exactly the two things procurement teams most often conflate.

The second finding is uncomfortable if you trust benchmark names as labels. Take BBQ, the bias-evaluation benchmark that everyone files under safety. BenchMIRT found it aligns much more strongly with general reasoning than with safety. A low BBQ score can mean the model reason through a tricky question badly, not that it behaved unsafely. The file the benchmark lives under is not what it measures.

Then there is HarmBench, which tests whether models comply with harmful requests. Its standard harmful-content questions aligned with safety, as you would expect. But its copyright questions, things like reproducing song lyrics, split off into a different dimension entirely. One benchmark, multiple signals, averaged into a single score.

Here is the lesson, and it is worth underlining. Benchmark names are marketing labels, not measurement guarantees. The folder the evaluation sits in is not a description of what it tests. When you report a benchmark score as evidence, you are reporting a blend you have not audited.

WHY 10 PERCENT OF THE QUESTIONS IS ENOUGH

Now the punchline for anyone who actually builds an eval suite. BenchMIRT found that keeping only 10 percent of the questions across those 16 benchmarks largely preserved the picture of which models were stronger on safety or reasoning. On top of that, it could predict how a model would answer a held-out question correctly 79 percent of the time. The simpler baseline that assumed a model performs about as well as its overall score said hit 70 percent.

Let that sink in, because it is a direct cost-saving rule you can take to your budget meeting. Your eval suite is probably ten times bigger than it needs to be and half as informative as it should be. Every low-information question you run is inference spend, wall-clock time, and maintenance you do not need, and it is diluting the signal from the questions that actually discriminate between models.

The citable rule is this: prune to the informative questions, not the famous ones. If you are keeping a benchmark question out of habit just because the benchmark has a recognizable name, you are paying for noise. BenchMIRT gives you a repeatable way to find the questions that actually separate strong models from weak ones and keep only those.

THE LEADERBOARD ECONOMY DOESN'T WANT THIS

Here is the thing nobody on a leaderboard team will tell you. The industry's answer to "what does this score mean" is increasingly to change the score.

Look at what happened this week. Artificial Analysis overhauled its entire Intelligence Index to version 4.2 the moment people started questioning how GPT-6 Astra scored. It dropped GPQA Diamond because models had saturated it, and it doubled the private-test-set weighting to 40 percent. That is a methodology change, sure, and much of it is defensible. But notice where it lands: the score changed right after the market started distrusting the score. That is the pattern.

Gemini 3.8 Flash is another case. It sits at the top of the DeepSWE leaderboard for complex software-engineering problem solving, and it is cheap to run. Yet on OSWorld-2.0, the agentic computer-use test, it still trails Claude Opus, the market leader. Two leaderboards, two different stories about the exact same model. DeepSWE says workhorse. OSWorld says not yet. Both are true because they measure different capabilities, and neither of them is "the model."

The uncomfortable truth is that benchmarks have become competitive weapons in a release war. The vendor that controls the score benefits from a single confusing number. BenchMIRT is the tool that asks the question the scoreboards cannot survive: what, exactly, is in that number?

RUN THE CONFLATION AUDIT ON YOUR OWN STACK

You do not need to wait for the research community to fix this. You can run the Conflation Audit on every benchmark and eval suite your team touches. Here is the three-question rubric.

1. Does this suite claim to measure one thing? Assume it measures at least two until you prove otherwise. A benchmark named for one ability is not evidence that it measures only that ability.
2. If a model's score jumped after a release, which dimension moved, safety or reasoning? Can you tell from the published per-item breakdown? If you cannot answer it, treat the jump as unverified, no matter how loud the announcement was.
3. Could 10 percent of the questions produce the same final ranking? If yes, your eval budget is misallocated. You are paying for redundancy instead of discrimination.

That audit costs you an afternoon. It will kill more wrong model-purchase decisions than any vendor call you have ever sat through, and it is the kind of engineering discipline that separates teams that pick models on evidence from teams that pick them on marketing. Evaluation is the part of AI engineering most teams fake, and it is the part that decides whether the rest of your stack works.

WHAT TO DO TODAY

1. Pull the eval suite your team actually uses and label each benchmark with its dominant dimension, reasoning or safety, using published item analyses where they exist.
2. Flag any benchmark whose name and measured dimension disagree. BenchMIRT showed BBQ is the cautionary tale, and you almost certainly have a few of your own.
3. Read the Allen Institute's BenchMIRT writeup on the Hugging Face blog before your next model selection meeting. It is short, it is the primary source, and it will change how you read every score in that room.
4. Cut a test copy of your eval suite down to the most informative 10 percent and check whether your model ranking changes. If it does not, shrink the suite. The saved spend goes back into your engineering budget, where it belongs.
5. Stop repeating single leaderboard numbers in internal docs. Quote dimension-level scores, or quote nothing. A number you cannot decompose is a number you should not be defending.

THE UNCOMFORTABLE QUESTION

Your team made its last model decision off a single number you now know blends two different capabilities. Nobody audited it. So here is the real question. Who is the person at your company whose job it is to know what that number measured?

If the answer is nobody, why does that number keep appearing in your slide decks?

You can't do anything in AI without solid engineering, and evaluation is the engineering discipline most teams fake. BenchMIRT just handed you the audit. The only thing left is whether you run it before you sign the next purchase order, or after.