Your GPU Scheduler Is Leaving 33 Points of Utilization on the Table

A Hugging Face post from Dharma-AI just showed that changing the order of GPU allocation decisions, on identical hardware, raised utilization by up to 33 percentage points and priority-weighted output by up to 105%. No new GPUs. No new racks. No new power contracts. The only thing that changed was the scheduler.

You already know utilization is the moat. PhantomByte Note #147 walked through Amazon's $173 billion infrastructure spend and its first period of negative free cash flow. The thesis was simple: the AI infrastructure boom is no longer a model story, it is a utilization story. This post tells you the specific engineering lever inside that thesis. It is not the model. It is not the GPU. It is the allocator.

The Wall Street Journal just reported that Big Tech's true AI spending is $3 trillion higher than disclosed. Nine major companies carry roughly $3 trillion in off-balance-sheet commitments tied mostly to AI infrastructure, against about $600 billion in reported capex. That is the size of the bet your scheduler is quietly undermining.

The Problem: Two Shapes That Do Not Fit

The core conflict in the Dharma-AI post is a shape problem. Batch-like workloads, training, batch inference, and quantization, need contiguous uninterrupted GPU blocks. Once a training job starts, it holds a block of GPUs without interruption until it finishes. Real-time inference is the opposite. It is elastic and demand-driven, growing and shrinking as traffic does.

These two allocation shapes are incompatible. You cannot serve both from the same pool without one starving the other. A training job needs a solid block. Real-time inference needs capacity that appears and disappears every timestep. Put them in one queue and they fight over the same hardware.

The standard answer is a FIFO scheduler with a fixed reservation for real-time inference. The post shows this leaves significant GPU capacity stranded under contention. The reservation protects latency but wastes capacity. An application that needs six GPUs at midday and two at 4am holds all six for the whole day, and the four idle GPUs are unavailable to any batch job for twenty-four hours. They are not being used, and they are not free either.

The batch queue waits for gaps that never come because the reservation fragments the available space. In the two scenarios where reservation dominates, the baseline sits near half the cluster: 51.6% in the mixed control and 53.6% in the training-heavy case. Roughly half a pool, with much of the idle half reserved rather than free.

Teach yourself to see your cluster as two overlapping demand patterns, not one queue. The failure is not that you lack GPUs. The failure is that your allocator treats all work as the same shape.

The Fix: Constraint-Aware Allocation

A constraint-aware allocator does something different. Instead of FIFO, it models the constraints of each workload type and allocates in an order that minimizes fragmentation. Batch work gets contiguous blocks. Real-time inference gets elastic capacity from the gaps.

	FIFO Allocation	Constraint-Aware Allocation
Ordering	First come, first served	Priority across the whole horizon
Batch work	Waits for gaps that never come	Gets contiguous blocks in the troughs
Real-time inference	Fixed reservation holds GPUs all day	Elastic capacity from the gaps
Idle capacity	Stranded and reserved, not free	Reclaimed and placed in priority order
Latency	Protected by holding hardware hostage	Enforced inside the same optimization

The allocator treats real-time demand as a curve rather than a ceiling. It allocates against demand at each timestep, with batch-like work occupying the troughs. Batch-like jobs are placed by priority across the whole horizon instead of in the order they arrived. The penalty for unmet real-time demand is priced five to ten times above the reward for batch work, so latency obligations are enforced inside the same optimization that places batch work, rather than by a separate autoscaler competing for the same GPUs. When a sudden traffic surge hits, the allocator re-prices the horizon in the same pass and pulls capacity out of the lowest-priority batch work first, which is what keeps p99 latency inside its SLA guardrail without pre-reserving GPUs that sit idle the rest of the day.

The seven-scenario benchmark on identical hardware produced the headline numbers: up to 33 percentage points of utilization recovery and up to 105% more priority-weighted output. Across the five scenarios built for genuine contention, utilization moved from a 52 to 85% band to a 72 to 88% band. Priority-weighted value rose between 24.6% and 105.1%, averaging 52%. Every scenario, both metrics, no tradeoff to explain away.

The strongest single case was a training-heavy workload on 8 GPUs. Utilization went from 53.6% to 87.0%, and value more than doubled, up 105%. Thirty-three points of a fixed, already-depreciating asset, recovered by reclaiming reserved standby capacity and placing the rest in priority order.

That is not a marginal optimization. That is recovering a third of your cluster for free.

Here is the decision rule. If your scheduler does not model allocation shape, you are paying for GPUs you cannot use. The lever is the order of decisions, not the size of the pool.

Why the Industry Buys Instead of Scheduling

The capital is flowing. Nvidia is investing $1.5 billion in SB Energy and providing up to $105 billion in credit for OpenAI's Ports-Pike data center near Cincinnati. Databricks just raised $5 billion at a $190 billion valuation. Groq raised $350 million as it pivots from chips to a neocloud. Stripe is acquiring OpenRouter for more than $7 billion. The instinct is to buy more.

The Dharma-AI data says the first lever is not buying. It is scheduling. You can recover 33 points of utilization with zero capital expenditure. That is the engineering authority angle: you cannot do anything in AI without solid engineering, and the engineering most operators skip is the allocator.

The buy-more reflex is understandable. When a cluster is slow, the instinct is to add capacity. But the allocator post makes the counterintuitive point that order is not a tiebreaker applied after the capacity question is settled. Order is a capacity decision. FIFO commits capacity to whichever job asked first, and a job that arrives later and needs a specific shape may find nothing left that fits, so it goes unscheduled and the GPU-hours it would have consumed go unclaimed. You are not short on GPUs. You are short on a scheduler that knows what shape the remaining work needs.

There is a second connection worth making. PhantomByte Note #151 argued that energy per request is the new AI cost moat, and that token-based billing misprices inference by nearly 50%. Recovered utilization is recovered energy. Every idle GPU that your scheduler could have filled is wasted joules. The utilization lever and the energy lever are the same lever.

The Scheduling Audit: How to Tell If You Are Leaking

Here is a concrete diagnostic framework. Three checks.

1. Measure your real-time inference reservation as a percentage of total cluster capacity. If it is fixed and above 30%, you are likely stranding capacity during batch contention. The reservation is protecting latency by holding GPUs hostage all day for a peak that lasts a couple of hours.
2. Track fragmentation. What percentage of your GPU memory is in blocks too small to serve a batch job? If it is above 15%, your allocator is creating gaps it cannot fill. A batch job needs a contiguous block of a given size, and if the free pool is chopped into pieces too small to hold it, the job goes unscheduled and the GPU-hours go unclaimed.
3. Compare priority-weighted output under FIFO versus constraint-aware allocation on the same workload mix. If the delta is above 20%, the scheduler is your bottleneck, not the hardware. Remember the scale test: 30 jobs across 64 GPUs, where FIFO and the allocator produced identical utilization at 44.9% and identical throughput, but the allocator delivered 15.9% more priority-weighted value. Every dashboard read the same. The cluster produced materially different output.

This is the teaching layer. You get a rubric you can run on your own cluster today.

What to Do Today

1. Pull your scheduler's allocation logs for the last 7 days. Identify how much GPU capacity was reserved for real-time inference and how much was actually used during peak batch windows.
2. Measure fragmentation. Calculate the percentage of GPU memory sitting in blocks too small for your smallest batch workload.
3. Benchmark a constraint-aware allocator against your current FIFO policy on the same workload mix. The Dharma-AI post provides the methodology.
4. If the utilization delta is above 10 percentage points, prioritize scheduler engineering over any new GPU purchase. The ROI is immediate and capital-free.
5. Connect your utilization metrics to your energy-per-request metrics. Recovered capacity is recovered power. See PhantomByte Note #151 for the energy accounting framework.

The Uncomfortable Question

You have a cluster running at 60% utilization and a purchase order for 40% more GPUs. Which one is the problem?