A model can climb 45 places on a public leaderboard without its provider touching the weights, retraining anything, or shipping a single improvement to the model itself. That is the top-line finding of "Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation," posted to arXiv on November 10, 2026 by Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell and Dirk Hovy. The authors worked on public LMArena data, a snapshot of 55 models and roughly 100,000 votes, and they did not have to break into anything to do it. Every attack they describe runs through the front door of the arena, using the prompt box, the vote buttons, and the model submission form.
Each of the three attacks can move a model 15 to 20 places on its own. The combination the authors actually ran, nomination followed by prompting, moved models up to 45 places, which is more than either attack delivers alone and more than adding the two would suggest. That is not a rounding difference between two capable models. That is the entire middle of the ladder.
So here is the part that should bother you. If your procurement process reads a public arena ranking and signs a contract off it, you are not buying the best model. You are buying the best-ranked model, and those are different things, because one of them is a measurement and the other is a measurement plus whatever a motivated competitor decided to spend. We have written about this failure one layer down, in "The Verification Gap: Why Your Agent Pipeline Is Flying Without Instruments," where the pipeline looked healthy because nothing measured it. Model selection is the same disease at the top of the stack. You picked a number instead of running a test.
THE THREE ATTACKS, NAMED

Strategic voting is the brute force version. A provider casts votes that improve its own model's standing, including votes on battles the model is not even in, because in a Bradley-Terry rating every comparison shifts the shared scale. The greedy strategy moved models up to 20 places using a cap of 10,000 manipulated votes, about 10 percent of the dataset. Twenty-three of the 55 models, 42 percent of them, moved more than 10 places, and eight models landed in the top 5. A cheaper heuristic still moved models up to 15 places, put 15 models into the top 5, and needed no per-vote Elo calculation at all.
The cost of the small version is almost insulting. A 2 to 3 place improvement required under 2 percent of total votes, meaning about 2,000 votes out of 100,000. Two thousand votes is not an intelligence operation. It is a Tuesday.
Voting requires knowing which model is which inside an anonymized battle, and the paper found that easier than it should be. Likelihood-based attribution identified open-weight models with more than 80 percent precision, and similarity scoring on API-only models performed above chance at 40 to 50 percent. Anonymity is a curtain, not a wall.
Strategic prompting is subtler and it does not require a single dishonest vote. The prompts are real, the voters' preferences are real, and the distribution is skewed. By adding prompts in languages where a target model happens to be strong, the authors moved models up to 15 places, with as few as 500 added prompts. reka-core-20240501 went from 33rd to 18th on Hungarian prompts. Llama-3.1-8b-instruct went from 38th to 23rd on Croatian. Phi-3-medium-4k-instruct went from 46th to 32nd on Scots. Language is only one axis, and the paper says so plainly. Repeat the experiment with categories the arena already labels, complexity, creativity, domain knowledge and math, and the boost reached 14 places.
Strategic nomination is the one that requires no votes and no prompts, only the ability to submit models. A provider posts copies of a model that is favorable to its target's relative standing, and the target rises because the comparison graph changed around it. Ten copies of the most favorable model moved one entry from 19th to 4th, a 15 place jump, and the paper reports that nomination effects are typically more moderate at 1 to 7 places. The planning cost is trivial. Computing the optimal nomination target across all 55 models in the dataset took less than 5 minutes on one CPU.
None of this needs insider access. All three attacks work against any preference-based evaluation that aggregates pairwise comparisons with Bradley-Terry and allows open prompting, open voting or open submission. That is the standard recipe, not an edge case.
WHY BRADLEY-TERRY BREAKS IN THE OPEN
The rating math is simple enough to hold in your head, and the simplicity is the problem. Bradley-Terry models the probability that model A beats model B as a logistic function of the difference in their latent strength scores, so P(A beats B) equals one over one plus e to the negative of theta A minus theta B. Fit those scores by maximum likelihood over every observed pair and you get a ranking. Elo is the same idea in online form.
Two properties fall out of that objective, and both are exploitable. First, every comparison is one unit of evidence, and repeated comparisons of the same pair keep adding leverage with no diminishing return and no per-voter budget. Vote ten thousand times on the same matchup and the fit treats it as ten thousand independent observations of the world, because nothing in the objective knows who voted or why.
Second, rank is a non-linear function of score. The paper's regressions put this bluntly. For strategic voting and strategic nomination, the strongest predictor of how easily a model moves is what the authors call Elo density, the number of models sitting within 50 Elo points above it. For strategic prompting, the top features are the number of battles in the model's best language and the Elo gap to the model one place above it. Where the ladder is crowded, and where a model has a deep well of battles in one language, a tiny score change buys several places of rank. That is the same trap your own team falls into when it reads a leaderboard top ten and treats the gaps as meaningful.
There is a deeper structural point in the paper that most coverage will skip. The Bradley-Terry objective is mathematically equivalent to a Borda count, the scoring rule that gives n minus r points to the model in rank position r. That means the vulnerabilities are not new discoveries in machine learning. They are old results in social choice theory, and the authors cite the real-world version. Kiribati uses classic Borda in some elections and documented strategic voting and nomination have been effective there. Nauru uses the harmonic, or Dowdall, variant, where a rank r position earns 1 over r points, and the same strategies are far less effective. When a voting system's own history tells you how it gets gamed, a leaderboard that adopts the system inherits the history.
THE 45-PLACE PROBLEM IN PRACTICE
Picture the decision most platform teams are making right now. Two frontier models, both credible, both within a couple of points on the public ladder, one of them cheaper. You are choosing the default model for a coding agent that will open pull requests and a support agent that will talk to customers. The tie gets broken by rank position, because rank is the only number in the room that looks objective.
Now run the paper's numbers against that decision. A model sitting 51st today can be a top 6 entry in the combined attack scenario, or a model sitting 40th can be a top 15 entry, without anything about either model changing. The ordering you consulted last month is a snapshot of an adversarial process, taken at one moment, with no error bar attached. Your selection memo states a rank as a hard fact about a variable that the paper's own experiments move by as much as 45 places.
My read, and this is opinion rather than a finding from the paper, is that rank is now the least stable number in any model procurement document. Cost per million tokens is contractual. Latency is measurable. Context length is a spec. Rank is a score that a competitor can move with a few thousand votes and a handful of Hungarian prompts, and yet it is the number that gets quoted in the slide deck because it is the only one that lets you avoid the work of measuring your own workload. "SWE-Bench Is Dead: Build Your Own Agent Evaluation Stack" made the same argument about task benchmarks, and the lesson transfers upward. A public score is someone else's test on someone else's distribution.
THE INTERNAL EVAL PROTOCOL: PRIVATE-5
The paper's defense section is worth reading in full, because it tells you what a leaderboard operator should do and, by subtraction, what you cannot rely on. Bot detection exists today for voting. The authors say they know of no protection at all against strategic nomination or strategic prompting. So if your model selection depends on a public board, you are depending on a control nobody has shipped.
You do not need their permission to fix your own process. Five moves, and the whole thing is a week of engineering:
- Private held-out prompts, never submitted anywhere public. If your eval set is public, it is a target. Prompt leakage is how strategic prompting gets a roadmap, and a private set is the only version of this test that stays yours.
- Blind routing. Evaluators never see which model produced which output. Strip the labels, randomize the order, log the mapping outside the scoring sheet. Every scoring pass in this protocol assumes the person or judge doing the scoring cannot tell whose answer they are reading.
- No vendor rank enters a procurement decision. Ranks are marketing. Write it as policy, not preference, because the day a rank supports the decision someone already wanted is the day the policy matters.
- A weighted private harness over 200 or more task-representative prompts. Weight categories evenly rather than letting volume decide, which is the same intuition as the paper's prompt stratification defense, applied to your own data. If 70 percent of your prompts are code review, your score is a code review score wearing a general capability label.
- Re-run quarterly, and treat a model that only wins public arenas while losing privately as a no. That last clause is the whole protocol compressed into a sentence. When the private harness and the public board disagree, the private harness is the one that runs your workload.
The engineering shape matters as much as the five rules. Keep the harness deterministic, version the prompt set, store raw outputs with a run identifier, and score offline from the stored outputs so a scoring bug can be fixed without re-running inference. Gate on private win rate against your incumbent, not on absolute score, because absolute scores drift with every harness change and win rate against the model you are replacing is the decision you are actually making.
WHAT PUBLIC LEADERBOARDS ARE STILL GOOD FOR
Be fair to the boards, because the paper is fair to them. It shows manipulation is cheap. It does not show that every ranking is fake, and the authors point out that the live LMArena text leaderboard carries more than 5 million votes, against the roughly 100,000 in their dataset, and that the manipulations needed scale roughly linearly with dataset size. Larger arenas are harder to move. They are not exponentially harder, which is the sentence that matters for anyone who assumed volume was protection.
What a public board still gives you is a directional signal, a regression smoke test, and a cheap sanity check. If a model's public scores collapse after a release, that is a real signal worth investigating. If a new entrant jumps 40 places in a week, that is worth a raised eyebrow. Use the board the way you use a weather radar, as an input to a decision you are going to make with instruments anyway.
And read the board's own defenses before you lean on it. Bot detection for voting exists today, at least in some form. Prompt clustering, verified voters, a harmonic aggregation function and a block on bulk submission of near-identical models are the paper's proposals, not shipped controls, and the harmonic variant has not been adopted by any leaderboard the authors name. Where a board has none of those, you now know exactly what its ranking is worth.
One more limit belongs here, in the authors' own words. They did not attack the live rankings, for ethical reasons, so every number in the paper comes from a historical snapshot they could analyze offline. That makes this a demonstration of exposure rather than a record of an actual incident. It also means nobody can tell you how much of it has already happened on the board you used last quarter.
WHAT TO DO TODAY
Pull your last model selection memo and highlight every place a public rank appears. That list is your exposure.
Stand up a 50 prompt private eval this week, even a rough one. Fifty prompts you wrote beats a thousand you borrowed.
Blind the outputs before scoring. Randomize the labels and keep the mapping out of the scoring sheet.
Write the rule into your procurement policy in one line. No public leaderboard number enters a purchase decision.
Re-baseline after every major model release, and store the runs so the next comparison is against your own history instead of someone else's chart.
THE UNCOMFORTABLE QUESTION
The attacks in this paper cost a few thousand votes and a few minutes of CPU time, and the largest one moved a model 45 places on a board that teams cite in contract discussions. So the question is not whether leaderboards are trustworthy in the abstract. The question is what your last model decision actually measured, and whether the answer survives a competitor deciding to spend an afternoon proving it wrong.
Get More Articles Like This
Getting your AI agent setup right is just the start. I'm documenting every mistake, fix, and lesson learned as I build PhantomByte.
Subscribe to receive updates when we publish new content. No spam, just real lessons from the trenches.
Build Real AI Infrastructure
PhantomByte teaches you to build real AI infrastructure yourself: local AI stacks, autonomous agents, multi-agent orchestration, web scraping, and custom tools. Step-by-step PDF tutorials you download, follow, and deploy. No subscriptions. No fluff. Just skills that ship.
