Your Evaluation Harness Moves Your Benchmarks 4.3 Times More Than Your Training Recipe

One team ran 24,000 sealed evaluations of a coding agent. Same model, same tasks, same frozen records. The only variables were the evaluation harness and the training recipe. The harness moved the mean solve rate from 2.14 percent to 9.27 percent. A factor of 4.3. The training recipe moved it by 1.16.

That means the number you quote in your deck, the one that proves your agent works, is mostly a property of the harness you measured it in. Not your model. Not your training run. The instrument.

We covered the cost side of this in Note #124, Your Agent's Harness Is Your Real Model. This is the other side. Your harness does not just cost you money. It decides your benchmark.

THE 24,000-EVALUATION EXPERIMENT

Walk through the Multi-Harness RL study in plain terms. The authors took one Qwen3-8B supervised warm start. They replayed the same frozen task-harness records from Aider, OpenHands, Qwen Code, and SWE-agent. They tested two GRPO grouping rules, Within and Cross. Then they scored every checkpoint with a sealed SWE-bench Verified oracle.

If you are new to this, here is what a harness is in one sentence. It is the scaffold that lets the model read the repo, run tests, and submit a patch. No harness, no agent. The model is the brain, the harness is the hands, and the hands decide what the brain can actually finish.

Now land the headline numbers. The harness factor is 4.3. The training recipe factor is 1.16. The grouping rule is not significant. Cross minus Within is plus 0.25 percentage points, with a 95 percent confidence interval that includes zero. The variable you obsess over, your fine-tuning loop, is the small one.

Read that again. You spend weeks on data, on loss curves, on the training recipe. It buys you a 1.16 factor. Swapping the harness buys you 4.3. The thing you barely think about is the thing that decides your score.

NOBODY PASSES ANYWAY

Now bring in Harbor Adapters and Harbor-Index. This is a unified evaluation infrastructure for agentic benchmarks. It ports more than 80 benchmarks to evaluate arbitrary agents. The authors ran 8 models across 54 benchmarks, every model with Terminus-2 and one of 3 native harnesses. Then they built Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks.

Here is the result. No model-harness configuration exceeds a 30 percent pass rate. The strongest, GPT-5.5 with Codex, reaches 28.0 percent.

Connect this to Note #158, The Verification Bottleneck Is Your New Build System. The industry built six-component verification machines because the raw capability is not there yet. Your 90 percent SWE-Bench score and this 28 percent number are both real. The difference is the instrument.

That is the uncomfortable part of this whole field. The same agent can look like a world-beater on one surface and a failure on another. Neither number is a lie. They are both true, and they disagree, because the harness is doing more of the work than you think.

THE INSTRUMENT DECIDES THE HEADLINE

Use PerfReasoning as the third data point. The task surface here is itself an extension of harness design, a Q&A prompting harness versus a program execution harness, so the same models are measured through different instruments. The strongest closed-source models exceed 90 percent on reasoning-based Q&A about hardware performance. Ask them to construct the performance model, and every configuration except one averages below 15 percent pass rate, with wild run-to-run variance. The best open-weight model reaches 82.4 percent on the reasoning side. On construction, only GPT-5.6 Sol exceeds 80 percent.

Same models. Different measurement surface. Wildly different story.

Build the teaching point. A benchmark number is a function of two things, capability and harness. Most teams only manage the first. They tune the model, they tune the data, they tune the prompt. They never touch the instrument, and the instrument is the dominant term.

THE HARNESS VARIANCE TEST

Here is the named framework, so you can use it and so an LLM can quote it back. The Harness Variance Test, four steps.

1. Freeze your agent and your task set. Change nothing else.
2. Run the full eval in two different harnesses. At minimum, one you built and one third-party, Aider, SWE-agent, or an existing adapter.
3. Compute the spread between the two solve rates. If the spread exceeds what your last training change bought you, your harness is your headline.
4. Report the range, not the peak. A team that reports 9.27 percent while hiding the 2.14 percent configuration is not reporting a result. It is reporting a choice.

The factor comparison is plain. Your harness factor should be smaller than the effect you claim, or you cannot claim the effect. If your harness spread is bigger than your training gain, then your training gain is noise. You have no evidence your recipe did anything at all.

WHAT TO DO TODAY

Here is the checklist. Each item is concrete.

1. Pull your last agent eval. Identify the harness by name and version. If you cannot name it, that is the finding.
2. Run the same eval in a second harness this week. Budget one day. The spread is the number that matters.
3. Replace every single benchmark number in your docs with a range across at least two harnesses.
4. Before your next fine-tune, run a dummy training change and measure its effect size. Compare it to your harness spread.
5. If you publish a benchmark, state the harness next to the number. Every time. No exceptions.

THE UNCOMFORTABLE QUESTION

Your benchmark score is a property of your harness as much as your model. So when you tell your boss, your customer, or your investors what your agent can do, which number are you quoting: the agent, or the instrument you happened to measure it in?