Prefill and Decode Are Two Different Machines. You Are Buying Them as One.

A French startup just squeezed 3,000 tokens per second out of standard datacenter GPUs. Not purpose-built inference chips. Not Cerebras wafer-scale silicon. The same AMD MI300X and NVIDIA H200 cards sitting in racks right now. The bottleneck was never the hardware. It was the assumption that prefill and decode are the same job.

OpenAI rolled out Ultrafast mode for GPT-5.6 Sol, hitting 750 output tokens per second. That is 14x the standard speed, delivered through its Cerebras partnership. They framed it as "more useful work per second," not a bigger model. The framing is the story. The race is no longer about whose model is biggest. It is about who separates the two phases of inference and allocates compute where it actually matters.

You are paying for prefill and decode as if they are one machine. They are not. Prefill is compute-bound, parallel, and gorges on FLOPs. Decode is memory-bandwidth-bound, sequential, and starves for the one resource your GPU wastes. Treating them as one job is why your inference bill is higher than it needs to be and your latency is worse than the benchmarks promised.

THE TWO MACHINES

Break down what prefill and decode actually do, and you will see two fundamentally different compute problems.

Prefill: the model processes your entire prompt in parallel. Every token is computed at once. This phase is compute-bound. It eats FLOPs. Your GPU is busy and happy. This is where the model "reads" your input and writes the KV cache.

Decode: the model generates one token at a time, each dependent on the last. This phase is memory-bandwidth-bound. Your GPU's compute cores sit mostly idle while the memory controller shuttles the KV cache back and forth for every single token. The GPU is bored. The bandwidth is the bottleneck.

Kog's CEO Gael Delalleau, a former offensive cybersecurity expert, says newer GPUs have memory bandwidth that is being left unused. That unused bandwidth is the gap between what your GPU could do during decode and what it actually does. Kog hit 3,000 tokens per second by exploiting that gap through software optimization, not new silicon. The company promises up to 30x faster LLM inference on hardware enterprises already own, and it expects to demonstrate a major model at 10x speed by September.

Here is the mental model: if you profile your inference pipeline, you will see two distinct phases with two distinct bottlenecks. Buying more compute fixes prefill. It does not fix decode. Buying more memory bandwidth fixes decode. It does not fix prefill. You need both, allocated independently.

This is why the naive answer to slow inference is wrong. Throwing more GPUs at a decode-bound workload does not help, because the new cards hit the same memory bandwidth wall. The utilization number is what matters, not the raw spec. A GPU that is 90 percent utilized on FLOPs during prefill and 30 percent utilized on bandwidth during decode is not a slow GPU. It is a misallocated one.

The industry is starting to price this correctly. The shift is from "which model is best" to "how do we extract more useful work per unit of compute." The architecture of inference, not the choice of model, is becoming the primary lever for both speed and cost.

THE DUAL-FLOW ARCHITECTURE

The architecture-level response to this problem just landed on arXiv. The Dual-Flow Transformer paper proposes decoupling the primary prefill path from additional decode computation.

The primary flow is a complete causal language model that processes the prompt and writes the KV cache. An auxiliary flow activates only from the final prompt position onward to add continuation-prediction computation. The two flows share major attention, MLP, and output matrices, but use separate token embeddings and lightweight coupling.

In plain terms: instead of one model doing both jobs with the same resource allocation, you split the work. The primary flow handles prefill. The auxiliary flow adds compute only where decode needs it. The shared matrices keep the model coherent. The separate embeddings let each phase optimize for its own bottleneck.

In MoE (Mixture of Experts) models, the separation goes further. Primary and auxiliary expert fan-outs become independent controls. You can allocate different experts to prefill and decode, tuning prompt cost, continuation cost, and predictive quality as three separate knobs instead of one. The paper demonstrates lower validation loss across architectures and data configurations.

What this means practically: the architecture itself is becoming phase-aware. You will see this pattern in production inference engines within 12 months. Understanding it now lets you evaluate inference platforms on a dimension most operators do not know exists.

The engineering detail matters here, so let me be precise about what the paper actually does. The auxiliary flow is omitted during prompt processing. It activates only from the final prompt position onward, adding continuation-prediction computation without writing persistent state or influencing the primary flow. That last part is the key. The auxiliary flow does not touch the KV cache. It does not add memory pressure to the phase that is already memory-bound. It adds compute to the phase that is compute-starved, without bloating the cache that decode has to shuttle around.

The weight sharing is what keeps this practical. Because the two flows share the major attention, MLP, and output matrices, you are not doubling the model. You are reusing the loaded weights and the cached keys and values during grouped execution. The result is a model that behaves like two machines but costs like one, which is exactly the point.

SPEED IS THE NEW PRODUCT

Connect the engineering to the market shift, and the pattern is unmistakable.

PYMNTS reports that speed has become the core product differentiator as OpenAI and Google race to sell faster AI. Competition is shifting from model quality to latency and throughput. OpenAI's Ultrafast at 750 tokens per second is not a model improvement. It is an inference architecture improvement. The model is the same GPT-5.6 Sol. What changed is how the compute is delivered.

This is the pattern: the labs are competing on inference engineering, not model size. Kog promises 30x faster inference through software. OpenAI promises 14x through a Cerebras partnership. Google ships Gemini 3.7 Flash three weeks after 3.6 Flash with improved agentic performance. The model gets incrementally better. The inference gets dramatically faster. The market is pricing speed, not quality.

Here is the decision rule: when evaluating an inference platform, ask which phase it optimizes. If it only speeds up prefill, your long-generation workloads will not benefit. If it only speeds up decode, your short-prompt high-throughput workloads will not benefit. The best platforms optimize both phases independently.

THE PHASE-SPECIFIC AUDIT

Run this framework on your own inference stack. Five steps.

1. Step 1: Profile your inference. Separate prefill time from decode time. Most inference engines expose this in their metrics. If yours does not, you are flying blind.
2. Step 2: Identify the bottleneck per phase. Prefill bottleneck is compute (FLOPs). Decode bottleneck is memory bandwidth (bytes per second). Your GPU spec sheet tells you both numbers.
3. Step 3: Calculate utilization. During prefill, what percentage of your GPU's FLOPs are used? During decode, what percentage of your memory bandwidth is used? If decode utilization is under 50 percent, you have the same gap Kog exploited.
4. Step 4: Evaluate phase-specific optimizations. Can you batch prefill requests separately from decode? Can you use a smaller model for prefill and a larger one for decode, or vice versa? Can you allocate different expert routing for each phase in MoE models?
5. Step 5: Project the cost impact. Kog's 30x speedup on standard GPUs means the same workload on the same hardware. No new procurement. The savings are in the architecture, not the invoice.

A note on the numbers before you run this. The 3,000 tokens per second Kog demonstrated was on a purpose-built 2-billion-parameter model called Laneformer 2B, which it has now open-sourced. That is not the same as a frontier model at 3,000 tokens per second. The claim to watch is the 10x on a major model by September. Treat the headline number as proof of the mechanism, not as a promise about your workload. The mechanism is what matters: software can recover the bandwidth your decode phase is wasting, on hardware you already own.

This extends the argument from Note #160 on harness efficiency, which cuts costs 40 percent on average. Harness efficiency decides how efficiently you use your model. Phase-specific compute decides how efficiently you use your hardware. Both compound. And it goes past the KV cache fixes in AgentKVShift: Fix Prefill Latency. Those are necessary but insufficient. The next lever is decoupling the two compute phases at the architecture level.

WHAT TO DO TODAY

1. Profile your top inference workload. Separate prefill latency from decode latency. If your engine does not expose this, switch to one that does.
2. Check your GPU's memory bandwidth utilization during decode. If it is under 50 percent, you are leaving performance on the table that software optimization can recover.
3. Read the Dual-Flow Transformer paper. Even if you never implement it, the mental model of two separate compute phases will change how you evaluate inference platforms.
4. When comparing inference providers, ask which phase they optimize. "Faster" is not an answer. "We cut prefill time by 3x and decode time by 10x" is.
5. Audit your batching strategy. Are you batching prefill and decode together? Separating them can improve throughput without new hardware.

THE UNCOMFORTABLE QUESTION

You bought GPUs for their FLOPs. Your decode phase does not use them. You are paying for a supercomputer and running a memory bus. How long are you going to let that slide?