The scariest bug in your inference stack is the one that never crashes anything.

No stack trace. No OOM kill. No failed health check. Your server keeps returning 200s, your dashboards stay green, and the tokens that come back are quietly wrong, or worse, quietly someone else's.

Jonguk Cheong, Myungbin Hwang, Eulgyu Kim and Byoungyoung Lee at Seoul National University posted a paper to arXiv on November 4. They went through the patch histories of vLLM and SGLang, pulled out 29 real-world KV cache safety bugs, and reproduced every one end to end. They built a checker for the bug class, ran it against current code during live serving, and it found 8 more violations nobody had reported. No false positives across every workload they ran.

We have spent multiple Notes on the KV cache as a cost object, including "Your Agent's KV Cache Dies at Every Turn Boundary" and the session-affinity routing piece. This is the other side of that object. It is not just your biggest cost line. It is a custom GPU allocator running inside your process, and allocators have bug classes.

THE ALLOCATOR YOU DID NOT SIGN UP FOR

PagedAttention is the memory manager nearly every serious serving stack uses. vLLM has it, SGLang has it, TensorRT-LLM has it. The design is one large cudaMalloc region partitioned into fixed-size physical blocks. Each request gets a block table, built on the CPU, mapping its logical block positions onto those physical blocks. GPU kernels read that table at runtime to find where a token's keys and values actually live.

That is an allocator. It has an allocate path, a free path, a free list, reference counts for shared prefix blocks, and a translation table. Different internals across vLLM and SGLang, same abstraction, same exposure.

Which means the classic allocator bug families arrived with it. Double-free. Use-after-free. Out-of-bounds access. Uninitialized read. Permission violation, meaning one request writes into a block it only holds read access to.

The paper's Case 1 should make an operator sit up. Two requests share the prefix "It is", so both map logical block zero onto one physical block, reference count two, read-only. A scheduler bug moves the second request's write position back two entries into that shared block. The cache write kernel translates the stale position through the request's own block table, gets a valid address, and overwrites "is" with "job". No crash. Every later request reusing that prefix reads "job".

Case 2 is worse for anyone who sells isolation. A finished request's block gets freed while the prefix cache still holds a reference. The pool hands that block to a new request, which writes its KV into it. A third request arrives, matches the cached prefix, and its attention kernel reads the second request's keys and values in place of the cached ones. One user's tokens, inside another user's forward pass.

WHY YOUR SANITIZER IS BLIND

Here is the part that should end the argument about tooling. Your sanitizer is not failing because it is badly made. It is answering a different question than the one you are asking.

Compute Sanitizer's memcheck and initcheck work at the CUDA allocation level. They see one giant allocation, the whole KV cache buffer, and reason about addresses inside it. When a kernel follows a stale block table entry, the address is still inside that allocation, in range and the right size, so it passes. The tool has no concept of a block, a request or a block table.

initcheck has the same problem on the time axis. Once any earlier request wrote to a block, the memory counts as initialized forever, so a new request's uninitialized read looks like live data.

ASan and its relatives carry the second problem. They buy detection with redzones, shadow memory and quarantine buffers, all of which assume you can afford to waste memory. KV cache memory is the scarcest thing in the cluster, because it bounds your batch size and your context length. The paper says the quiet part: reserve too much and the serving system shrinks its KV cache, serves fewer concurrent requests, or dies with a CUDA out-of-memory error before your workload runs. NVIDIA's own documentation notes that memcheck padding raises device memory pressure and can cause additional out-of-memory failures.

So the tooling is wrong on the abstraction and wrong on the budget at the same time. It cannot see the bug, and if you forced it to, you would pay for the visibility in throughput.

THE SHADOW BLOCK CHECK

The fix is a named pattern, and you should learn the name because it is the thing to demand from your serving framework. Call it the Shadow Block Check.

Infographic titled See What Sanitizers Miss, Five Checks One Safer KV Cache, showing a GPU KV cache memory block at the center with permission tracking and initialization tracking, five numbered checks arranged around it for allocator event, CPU block table construction, CPU-to-GPU translation, GPU block table lookup and KV entry access, an unauthorized write blocked by a green shield, and an arc labeled Traditional Sanitizer Scan showing a PASS badge
Five checks around one block, under a sanitizer arc that only ever prints PASS.

The mechanism is smaller than it sounds. For every physical KV block, keep one compact metadata record with two fields. A per-request permission field records which requests hold the block and in what mode, shared read or private write. A per-entry initialization record says whether each KV entry has been written during the block's current lifetime. Because the serving system never reuses a request index, the permission field also encodes the block's lifetime, which lets a checker tell a stale block from a live one.

Five checks run along the workflow, each where the facts needed to catch a violation first exist. The allocator event, so a double-free or bad free is rejected at the free call. The CPU block table construction, so a request is never mapped to blocks it was not granted. The CPU-to-GPU translation, so what the kernel reads matches what the CPU intended. The GPU-side block table lookup inside the kernel, so a stale row is caught at the read. And the KV entry access itself, so a write into a read-only shared block, or a read of an unwritten entry, is caught at the instruction.

Compact is the design constraint, and it is what makes this shippable. On H100, the shadow for Llama 3.2 1B with a 1,048,576-token KV pool is 17.8 MiB against a 32 GiB cache, 0.11 percent. On Qwen 2.5 14B it is 9.1 MiB against 24 GiB, 0.05 percent. The worst case in the paper's table is 1.5 percent of model weights, two orders of magnitude below Compute Sanitizer on the same workloads. Nothing gets padded, quarantined or delayed, which is why the check does not become the thing that forces you into a smaller batch.

THE PRICE AND THE DECISION RULE

Coverage first. All 29 historical bugs from the two patch histories were reproduced and caught. Eight new violations turned up in current vLLM and SGLang execution paths, each reported upstream with a fix pull request. Zero false positives.

Cost second. On Qwen 2.5 14B with all checks enabled, the geomean TPOT (time per output token) overhead is 3.0 percent on vLLM and 7.4 percent on SGLang.

Now the decision rule, and here is where I separate fact from my opinion. The fact is that the paper positions KVGuard as a development-phase tool, in the same class as ASan: continuous integration, pre-release kernel validation, post-incident debugging. The authors do not argue for always-on production. The read that follows is mine.

If you run multi-tenant traffic and request isolation is a promise you make to customers, a single-digit latency tax in your validation environment is the cheapest evidence you will ever buy. You cannot put a percentage next to a cross-tenant leak, or a number on a customer finding another customer's tokens in their completion. You model the three to seven percent and take the trade.

If you serve single-tenant batch workloads, or one model for one team behind one internal gateway, the calculus changes. The isolation promise is smaller, the blast radius is an internal answer you can live with, and the sanitizer belongs in CI and post-incident work rather than the request path.

Both readings are live. What is not live is the third option, doing nothing and assuming the stack is covered because Compute Sanitizer came back clean.

Treat this as a dependency decision, not a rewrite. Shadow checking is a pattern you demand from whatever framework you run, adopt where it ships, and press the vendor where it does not. Porting is small: the checks attach at the block manager, the block table, and the KV access kernels.

WHAT TO DO TODAY

  1. Grep your serving stack's version and release notes against the vLLM and SGLang patch histories for KV cache safety fixes. If you have never inventoried them, you are carrying some unknown subset of the 29.
  2. Ask your inference vendor one question: what detects a use-after-free in the KV block table today? "The sanitizer" is not an answer. Sanitizers see a single CUDA allocation, not blocks, requests or block tables.
  3. Size the tax. Model what 3 to 7 percent TPOT overhead costs you in a validation environment against one cross-tenant data leak incident.
  4. Pin arXiv 2611.06138 in your security review channel and assign one owner to track whether vLLM or SGLang adopts the shadow block pattern. An unowned paper is a bookmarked PDF nobody reads.

THE UNCOMFORTABLE QUESTION

You audit the code that loads the weights. You audit the tokenizer, the chat template, the safety filter and the system prompt. You do not audit the allocator that holds every user's tokens mid-request.

Why does the last one get a pass?

Enjoyed this article?

☕ Buy Me a Coffee

Support PhantomByte and keep the content coming!

Build Real AI Infrastructure

PhantomByte teaches you to build real AI infrastructure yourself: local AI stacks, autonomous agents, multi-agent orchestration, web scraping, and custom tools. Step-by-step PDF tutorials you download, follow, and deploy. No subscriptions. No fluff. Just skills that ship.