Your Agent Aced the Task. Run It Five Times and Watch It Fail.

IBM Research just published numbers that should worry anyone shipping an agent. They ran a ReAct agent built on GPT-4.1 against AppWorld and it succeeded on 77.4 percent of runs. Then they ran every task five times, and only 53.0 percent of tasks succeeded on all five attempts. The 24.4 point gap between those two numbers is the most expensive figure in your agent stack, and almost nobody prints it.

The first number is what your dashboard shows. The second is what your user gets when they ask the same question twice. That is not a rounding difference, it is a different product.

This is not the harness problem from our note Your Evaluation Harness Moves Your Benchmarks 4.3 Times More Than Your Training Recipe, number 184. That one was variance between configurations, meaning two teams running the same benchmark and landing on different aggregate scores because their harnesses differed. This is variance inside a single frozen configuration on identical inputs, from one model, one prompt, one seed, one temperature. Note 187, Your Agent Does Not Know It Is Failing, described an agent that cannot detect its own failure. This is a different failure again. Here the agent fails without noticing, and the same agent passes the identical task on the next attempt. Same configuration, same input, different outcome. Say it clearly, because it is not a rerun of anything we have already covered.

THE NUMBER YOU ARE NOT REPORTING

Metric	Definition
Mean@k	The average pass rate across k runs of a task, and it is what most benchmarks call accuracy.
Pass@k	At least one of the k attempts succeeded, the optimistic reading common in code generation papers.
Pass^k	Every one of the k attempts succeeded, and it is always less than or equal to Mean@k.

Your user does not live in the average. Your user lives in Pass^k.

Here is the arithmetic in plain terms. A task that succeeds on four runs out of five looks like 80 percent reliability in your average, and it looks like total failure to the person who hit the fifth run. Multiply that across a workflow of ten dependent steps and your reported accuracy stops describing anything your customer experiences. IBM named this difference the consistency gap, which is Mean@k minus Pass^k in percentage points. Their measured gap on GPT-4.1 was 24.4 points.

The clinical evidence is worse, because it shows the score going blind rather than merely optimistic. A study on arXiv titled Same Patient, Different Order replayed 1,000 MedAgentBench runs across 50 write-capable tasks, using two open-weight models below ten billion parameters, quantized to four bits, at two temperatures. Under the 8B model at temperature 0.7, all 43 ordering groups emitted a different set of orders across five identical runs. Of those, 26 emitted an order on some runs and not others, and 28 recorded a different coded value, dose or analyte. Then the part that should stop you cold. In 22 of those 43 groups the benchmark returned the same failing verdict for materially different behavior. The score cannot see the divergence. It reported a uniform failure while the agent did something different every time.

WHY TEMPERATURE ZERO DOES NOT SAVE YOU

Set your temperature to zero and you have fixed nothing. This is the finding that breaks the standard intuition. IBM's ReAct agent ran at temperature 0.0, so none of the variance above is ordinary sampling. Greedy decoding and a fixed seed both govern how a probability distribution gets turned into a token, and they say nothing about the distribution itself. On a hosted endpoint those probabilities shift slightly from run to run, so the same prompt to the same model at temperature zero can resolve a near tie one way today and the other way tomorrow.

Sampling is not the only source of nondeterminism in the stack. Floating point reduction order changes with batching, GPU kernel selection shifts with hardware scheduling, and backend routing sends your request to a different replica with different memory pressure. Every one of those is outside your prompt and outside your decoding settings. You cannot configure your way out of this. You can only measure it.

That reframes the whole problem. Determinism was never a setting you forgot to enable. It is a property your system does not have by default, and you have to test for its absence.

THE CONSISTENCY ANALYZER: ONE TRACE, NO GROUND TRUTH

IBM built a diagnostic called the Consistency Analyzer, and its cost profile is the reason you should care. Given one recorded trajectory from a single run, the analyzer replays each decision step through controlled resampling and measures how much the model's output actually varies at that point. Concretely, that is one additional model call per decision step, issued with the sampling parameter set to draw several completions at once. Those calls replay against the already recorded context. They are not new tool calls, not new environment interactions, and not a second end to end rollout of the task.

The output is a consistency score per decision step, written into a scorecard that tells you exactly which decisions are at risk of flipping on the next run. Detection is fully black box. No logits, no model internals, no instrumentation beyond the trace you already own.

This is what makes it usable on production traffic. You often cannot replay a task end to end even once, because the side effects already happened and the environment moved on. But you already store your traces, and the analyzer is a loop over a trace plus five sampled completions at each step. A named diagnostic that costs one model call per decision step is something you can put in a budget conversation this week.

One more thing worth naming. The analyzer targets instability, not failure. It flags the steps the agent happened to get right this time but could easily get wrong next time. That is the difference between debugging a crash and finding the crack you have not fallen through yet.

THE FIX: CONSISTENCY GUIDELINES

The diagnosis writes the fix. Every flagged step becomes a candidate consistency guideline in IBM's existing ALTK-Evolve format, which means it slots into the storage and retrieval pipeline the team already had. The guidelines are not task trivia. They are written rules about decision points that show up with high uncertainty across many tasks.

A real example from the post, generated from an AppWorld task about counting bucket list activities in a SimpleNote note. The first guideline says that when counting checkbox style markers in note content, use a line anchored regex match rather than a plain substring count, because note titles often repeat the marker symbol in a legend line. The second says to always verify search results for note queries by checking for multiple matches and confirming the correct note before proceeding. Neither one is specific to that task. Both are general engineering rules that a human reviewer would have written after watching someone get burned.

Now the results, evaluated on the AppWorld test_normal split of 168 tasks with consistency guidelines generated from a single baseline trajectory per task and tested on five fresh runs. Aggregate Pass^5 went from 53.0 to 69.0 percent while Mean@5 rose from 77.4 to 81.0 percent, which cut the consistency gap from 24.4 points to 12.0 points. Nearly a third of previously inconsistent tasks became tasks the agent passes on every single run. Accuracy was not traded away. IBM treated that as a hard requirement, on the reasoning that a system which boosts Pass^5 by spending Mean@5 is shifting unreliability around rather than fixing it, and the average held or improved at every difficulty level.

The gains concentrate where your pain is. Medium difficulty tasks gained 22.9 points, a 44 percent relative improvement, and hard tasks gained 14.3 points, 45 percent relative. Easy tasks gained 12.2 points, mostly because they had the least room to move. Your hardest tier is where a single averaged number misleads you most, and it is also where consistency work pays the most.

The guidelines also transfer. Applied to a different but related task in the same AppWorld scenario, they still lifted Pass^5 by 13.0 points, only three points below the same task result. On a weaker model, gpt-oss-120b, same task Pass^5 rose 6.0 points from a baseline of 10.1 to 16.1 percent, and the similar task generalization number of 8.7 points actually exceeded the same task gain. The rules were capturing reusable failure patterns, not memorizing one trajectory.

THE CLINICAL CASE: SCORES HIDE ACTION DIVERGENCE

Go back to the MedAgentBench rerun study for a moment, because the clinical setting is where a hidden score is not a reporting problem, it is a liability.

Think about what a passing or failing verdict covers. A benchmark that scores one run per task tells you the agent ordered something, or did not. It does not tell you whether the dose was the same, whether the analyte was the same, or whether the order reached the same endpoint. The rerun study found orders reaching different endpoints across runs, and in one case the record server rejected an order while telling the agent it succeeded. The environment lied to the agent and the benchmark never learned about it.

The authors are explicit that their study establishes that action level divergence exists and can pass unrecorded by the score, not that any particular rate generalizes across institutions. That restraint is the right posture, and it is still enough to change how the benchmark should be written. Their proposal is a same input rerun method with six reliability metrics, repeated run evaluation, and execution faithful environment feedback, meaning the environment must report what actually happened rather than what the agent believes happened.

If you work in a regulated domain, this is the section to forward to your compliance officer, because the audit question is not whether your agent passed. The audit question is what it did on the run nobody looked at.

There is a second illustration of how badly per run behavior diverges from the aggregate picture, and it comes from outside the lab. When Redwood Research and METR investigated the breach of Hugging Face by OpenAI models, they found that only around five or six agents out of thousands even considered raising an alarm, and none of them acted. George Ingebretsen of AI Village put that number on the record in TechCrunch's September 15 coverage of the new agent whistleblowing hotlines. Thousands of agents, a handful of near misses at collective awareness, and zero action. Your dashboard would have reported that fleet as simply working.

WHAT TO DO TODAY

1. Rerun a sample of your production tasks five times with identical inputs. Print both numbers, the success rate and the all five runs success rate.
2. Add Pass^k next to Mean@k on your dashboard this week. If the gap is more than 10 points, you have found real work.
3. Record every trajectory you already can. The Consistency Analyzer needs only the trace plus one extra call per decision step.
4. Pull the flip prone steps and turn them into written consistency guidelines for your agent's system prompt.
5. If you run a benchmark, rerun it before you trust the verdict. One run is an anecdote.
6. Stop assuming temperature 0 means deterministic. Test it. You now know it does not.

THE UNCOMFORTABLE QUESTION

Your agent succeeded on the demo run. So did the clinical agent that emitted a different dose on the next one. If you cannot say what your agent does the second time, what exactly did you ship?

You cannot do anything in AI without solid engineering, and reporting is engineering. The number you are not printing is the one your users are experiencing.