Stop Tuning Weights: How Session-Affinity Routing Cut Time to First Token 85 Percent

NVIDIA says that on 16 B200 GPUs, its Dynamo inference software delivered eight times as many concurrent users and an 85 percent drop in time to first token. Same models. Same weights. Same hardware.

The entire gain came from the routing layer. Agents resend the same system prompt, tool descriptions, and conversation history on every turn, so Dynamo caches those computations and routes each request to the GPU that already holds that session's cache. That is not a model improvement. That is a scheduling decision, and it is the kind of win your own stack can copy this week.

We covered prompt cache leaks on October 10 in "Asana Cut Its Agent's Cost 76x Without Touching the Model." This is the other half of the same bill: what happens to a session before the first token ever leaves the chip.

THE WASTE YOU ALREADY HAVE

Agents are the worst workload a naive serving stack can receive. Every turn resends the system prompt, the tool descriptions, and the full conversation history, so the stack recomputes context it computed one second ago. Then a load balancer scatters the next turn to a different replica, and the whole prefill runs again from zero. You did not change the prompt. You did not change the question. You still paid for the entire history a second time.

Look at the model NVIDIA shipped to sit on top of this stack. Nemotron 3.5 Super VL is an open 120 billion parameter vision-language model, a mixture of experts where only 12 billion parameters are active per token, with a context window up to one million tokens. Read the split carefully. The model is cheap per token precisely because it activates a tenth of itself. The context is what costs, and the context is the part your agent rebuilds on every request.

Here is your situation in one sentence. If your agent re-serializes its world on every turn and your load balancer routes by load instead of by session, you pay the same prefill tax dozens of times per conversation. Not once. Dozens.

The reason is structural. Prefill reads the whole input and dominates time to first token, the number a user actually feels. Decode is the cheap, steady part. An agent turn carries a large prefix that almost never changes and a small suffix that does. The prefix belongs in cache. A stateless scheduler treats it like fresh text on every call and bills you for it every time.

THE CACHE-AFFINITY RULE

State the rule plainly, because it is the whole article. A request is cheap when it lands where its cache already lives, and expensive everywhere else. The routing layer, not the weights, decides where it lands. Call it the cache-affinity rule, and use it as the first question in every serving review you run.

Three levers implement the rule.

1. Session-affinity routing, which is what Dynamo does. Route by session identity so the GPU holding that session's key-value cache keeps receiving that session's traffic. The cache is not merely reused, it is never thrown away by a scheduler that only sees load.
2. Cache reuse across capabilities, which is what IBM's Granite Switch 4.1 does. This is a different class of win from Dynamo's, and the distinction matters. Dynamo is a routing fix: same model, same weights, same GPUs, with a scheduler that stops throwing your cache away. Granite Switch is a composition fix: it packs several capabilities into the VRAM of one base model. Routing decides where a request lands. Composition decides how much capability fits on the machine it lands on. The 8B and 30B models compose a chat base model, a general-purpose Decision adapter, a Guardian safety adapter, and Granite Speech 5.0 Turbo CTC, which IBM calls the fastest ASR model on the OpenASR leaderboard. All of it runs in roughly the VRAM of the base model alone, so one endpoint carries the work of several. The mechanism is that the Switch adapters are cache-efficient Activated LoRAs, so the KV cache created by a base generation is reused for adapter-based decisions instead of being rebuilt. IBM credits its own recent vLLM contributions for low-rank adapter inference speed and a co-located ASR encoder. The demo is concrete: an 8B model transcribes audio, writes summaries, screens for harmful content, and tests 15 user-defined flags on a single endpoint in real time.
3. Trained-in compression, which is the NVFP4 move. NVIDIA trained the 4-bit build of Super VL to reproduce the uncompressed model's outputs rather than quantizing it after the fact. That is the difference between compression as a capability and compression as a loss applied to a finished product. On Blackwell cards the model runs entirely in 4-bit, while older cards store weights at 4 bits and compute at 16 bits, and sensitive parts keep higher precision.

Each lever attacks the same enemy: recomputing what you already computed.

THE FOUR METRICS BEFORE YOU BELIEVE A ROUTING CLAIM

The 85 percent number is a claim about a specific configuration, not a law of your cluster. NVIDIA measured it serving 50 tokens per second per user on 16 B200 GPUs with cache-aware routing, session affinity, and KV cache offloading. Change your traffic and the number moves. Instrument four things before you believe any routing claim, including this one.

1. Time to first token under your real session mix, not a synthetic benchmark. The gain scales with how much repeated context your traffic actually carries. A single-shot chatbot sees almost none of it, which is why vendor demos and your dashboard rarely agree.
2. Cache hit rate per session. A router that misses the cache delivers zero of the savings. This is the number that tells you whether the routing is working at all, and it is the first number most teams have never logged.
3. Concurrent sessions per GPU before quality of service breaks. That is where the eight times concurrency claim lives or dies, and it is the first thing to degrade when the cache working set outgrows the card.
4. Memory headroom. KV cache reuse only pays if the cache fits. A session-affinity router on a card too small to hold the working set is a queue with extra steps, and the latency you tried to cut comes back as a wait.

Keep one axis separate while you measure. Super VL also ships a multi-token prediction head that proposes tokens the main model verifies in a single forward pass, which NVIDIA says speeds generation by up to 2.3 times. That is a decode win, not a prefill win. Do not let a faster decode number stand in for routing savings. They come off two different ledgers, and only one of them touches the prefill tax you are trying to kill.

WHAT THE STACK DECIDES, NOT THE MODEL

Zoom out. The TensorFeed report for this week calls the serving layer the place where capability is won or lost, and the 85 percent number is the proof. Nothing about the model changed to produce it.

The decision rule for you is short. Before you buy a bigger model, check whether your router is throwing away your cache. The bigger model pays full prefill price on every turn. The routed cheaper model pays it once and reuses it. At scale the router wins, and it wins by more than a parameter bump usually does, because parameters buy quality while routing buys cost.

We made the same point from the harness side in "Your Agent Burns Half Its Tokens Before the Model Even Runs," a note on NVIDIA's SoL-Pi work. There the waste lived in the control layer before inference ever started. Here it lives in the scheduler that picks the replica. Either way the lesson holds: the cost of an agent is decided at every layer except the one you benchmark. You have been tuning the weights of a system whose cost is decided by the router.

WHAT TO DO TODAY

1. Log your per-session cache hit rate for one day. If you cannot measure it, that is the first finding.
2. Check whether your load balancer routes by session identity or by load. If it routes by load, your cache is being evicted on purpose.
3. Compute your average resent-context size per agent turn. That number is your routing upside, and it is usually larger than any quantization gain on the table.
4. Test one session-affinity configuration against your current setup and compare time to first token on real traffic, not synthetic prompts.
5. If you run adapters for safety or decisions on top of one base model, check whether they rebuild the KV cache. If they do, you are paying for the same context twice.

THE UNCOMFORTABLE QUESTION

You can name the model you serve down to the parameter count. Can you name the policy that decides which GPU a returning session lands on? If the answer is no, why did you spend more time choosing weights than choosing the router that decides what they cost?