Your Agent Queue Is a 3.5x Speedup You Already Own

Under load, one scheduling change cut the P95 of agent workflow flow time by up to 3.50x. No new model. No new hardware. Just a different answer to a question almost every runtime answers the same lazy way.

The result comes from arXiv 2609.10964, "Decoupling Readiness from Release for Tail-Aware Scheduling of Agentic LLM Workflows," submitted September 10, 2026 by Bochao Feng and six other authors. They evaluated on real agent execution traces from software engineering tasks, across multiple LLMs and multiple workflow arrival rates. Under light load their scheduler matches eager release exactly. Under contention it leaves it in the dirt.

So here is the thesis, blunt as I can make it. Your agent runtime releases a turn the instant it is ready, and that eager habit is exactly what destroys your tail latency. In Note #177, "Your Agent Dies at the Timeout," we fixed agents that die while waiting for the outside world. Now we fix the agents that crowd each other at the door.

THE EAGER RELEASE TRAP

Readiness is when the model says the turn can run. Release is when you actually let it. Most runtimes collapse those into one event: ready means run, right now, immediately.

That feels generous. It is actually a forfeiture. The moment you submit a released turn, it is in flight, and the workflow-level policy can no longer reorder it. The order is committed. You spent your one scheduling decision at the exact moment you had the least information about how crowded the machine was about to get.

Contention is simple to define for anyone new to this. Contention is when you have more ready work than your runtime can execute right now.

Under contention, eager release does something specific and ugly. It accumulates released-but-unfinished work, a growing pile of turns that all got submitted and now all compete for the same compute. Tail latency climbs, and the fix is no longer available, because reordering requires the option of not having released yet.

Every agent framework that demos beautifully solo does this. Solo is exactly where eager release is correct, because with one agent and an idle machine there is nothing to reorder. That is why the flaw survives so long. Your benchmark never sees it.

The stakes are not academic. OpenAI published an internal research report on September 6, 2026 stating that its research organization was logging 3.1 agent-workdays for every human workday by mid-August, and that the median researcher measured by agent usage was spending more than $600 per day on inference at API prices, with the 90th percentile exceeding $7,000 per day. The New Stack covered those numbers on September 11, 2026. At that burn rate, tail latency is a payroll line, not a metric on a dashboard nobody opens.

THE READINESS-RELEASE SPLIT

Name the move, because you are going to use it in a design review. It is the Readiness-Release Split.

Decouple readiness from release. A scheduler that holds both decisions gets to answer two questions instead of one. Which ready turn should I release next? And how much released-but-unfinished work should I tolerate right now? Most runtimes answer the first question with "whoever got ready first" and never ask the second one at all.

The paper's objective is mean-Conditional Value-at-Risk. For anyone new, here is Conditional Value-at-Risk in two sentences. Instead of optimizing the average outcome, you optimize the worst slice of outcomes, the tail. That is the part your users actually feel, so that is the part you should be optimizing.

For the professionals, the detail matters. The method uses a mean-CVaR objective that captures the evolving tail risk of unfinished workflows, incorporates online estimates of turn work when prioritizing ready turns, and adapts the released-work budget to observed queue pressure. Online estimates beat static guesses because turn work is not constant, and a scheduler tuned to a stale workload assumption is just a slower version of the bug you are trying to fix.

One opinion, labeled as such: the split is the reusable idea here, more than any specific CVaR implementation. Once you accept that readiness and release are separate decisions, you can plug in your own risk measure and still capture most of the gain.

WHY THE TAIL IS THE ONLY NUMBER THAT MATTERS

Averages lie in agent systems. If nine workflows finish fast and one hangs, your mean looks healthy and the user of workflow ten is writing your support ticket. The average is a comfort blanket. P95 flow time is the honest metric, because P95 is where your worst-served users live.

The paper's result is an asymmetry, and the asymmetry is the argument. Under light load, the tail-aware scheduler performs comparably to eager release. You pay nothing when the machine is idle. Under contention, it substantially reduces the P95 of workflow flow time, up to 3.50x. Free when idle, decisive when crowded.

The industry mood is already pointed this way, which is why this lands now. Ramp's September 2026 AI Index, as reported by The Decoder on September 10, 2026, found that the top 1 percent of US AI spenders cut median per-employee AI spending by 9.7 percent in August to $7,205, and that the effective price per million tokens had fallen 41 percent since its March 2026 peak to $0.68, as buyers traded down toward cheaper inference.

Read that alongside DeepSeek's V4.1-Flash, released September 10, 2026 and built specifically to cut the memory footprint of agent deployments, and you see the shape of the market. Vendors are attacking agent cost at the model layer and the memory layer. Nobody is selling you the scheduler layer, because the scheduler is yours. It is the one component on the efficiency ledger that no vendor can fix for you, and it is the one almost nobody is tuning.

THE RELEASED-WORK BUDGET IS YOUR NEW TUNING KNOB

Here is the practical lever. The released-work budget caps how much released-but-unfinished work may exist at once.

Set it too low and you starve the machine. Ready turns sit waiting while compute idles, and your throughput suffers for a latency win you did not need. That starvation is a deliberate trade-off rather than a free win: a strict cap prioritizes tail-latency guarantees over maximum parallel throughput, so you accept lower overall token throughput in exchange for a steadier P95. Set it too high and you have rebuilt eager release with extra steps and a config file.

The tuning sequence is boring on purpose.

1. Start from eager, meaning effectively unbounded.
2. Measure P95 flow time during your worst production hour.
3. Drop the budget. Measure again.
4. Keep dropping until P95 stops improving, then stop.

This knob belongs next to the constraints in Note #174, "Your Agent Runs on a CPU," where we built the Agent Compute Balance Sheet. GPU, CPU, memory bandwidth, context, and now released-work budget. It is a resource you are already spending. The only question is whether you are accounting for it or bleeding it invisibly.

WHAT TO DO TODAY

1. Pull P95, not mean, of workflow flow time from your agent runtime's logs. If you only track the mean, you are blind to the actual problem.
2. Count how many released-but-unfinished turns your runtime tolerates right now. If the answer is unbounded, you have found the bug.
3. Instrument per-turn work estimates. The scheduler's adaptation needs them, and online estimates beat static guesses.
4. Prototype a released-work cap in front of your existing queue. One configuration flag, measured against the same traces.
5. Rerun your worst production hour against the capped scheduler. Compare P95 flow time before you believe anyone, including me.

THE UNCOMFORTABLE QUESTION

Your model vendor is shipping models to cut agent memory costs. Your chip vendor is shipping CPUs for agents. How much did you spend this month on the one component neither of them will ever fix for you, the scheduler you wrote yourself?

Your tail latency is not a hardware problem, it is a decision you make thousands of times an hour and have never once measured.