Your Agent's KV Cache Dies at Every Turn Boundary

GitHub Copilot just gave the world its first production-scale look at what agentic coding actually does to inference infrastructure. The numbers came from arXiv paper 2608.00101, published August 2026. The study sampled traces from June 2026: 3.2 million users, 13 million sessions, 761 million LLM calls, and 95 trillion tokens. That is not a benchmark. That is the real workload.

Inside that workload is a number that should change how you architect agent serving. KV cache hit rates average 90% within a single agent turn. They collapse to 55% across turn boundaries. That is not a 35% drop. That is your inference cost structure flipping from efficient to wasteful every time the user sends a new message or the agent starts a new loop iteration.

Note #143 (July 29) showed that KV cache was eating your inference budget within a turn. This is the next layer. The cache does not just cost you inside a turn. It dies between turns. And nothing in current LLM serving systems is designed to survive that.

The 90-to-55 Collapse: KV Cache Invalidation Metrics

The paper's core finding, stated plainly: KV cache hit rates average 90% within a turn and fall to 55% across turn boundaries. This means nearly half your cached computation is invalidated every time the conversation moves to a new turn.

The cause is structural. Agent sessions consist of "sparse user-initiated turns that unfold into autonomous agent loops of LLM calls almost always coupled with tool execution." Each tool call injects new tokens. Each context compaction restructures the prompt. Each model switch throws away the entire cache. The paper notes cache is "drastically invalidated" after model switches or context compaction.

If you are running a production agent, your cost model assumes the 90% hit rate. Your actual workload is operating closer to 55% across the session lifetime. The gap between what you think you are paying and what you are actually paying is the turn boundary tax.

Why Your Harness Is the Cache Killer: Agent Orchestration and Cache Invalidation

For clarity, the harness is the orchestration layer managing tools, memory, and model routing, also called an orchestrator or framework. Note #124 (July 10) proved orchestration design beats model selection by 10x in token cost. The harness is where cache lives or dies. Here is why.

Every time your agent compresses tool output (Note #107, June 23), the prompt changes. The cache breaks. Every time your agent switches from a reasoning model to a fast model for a tool call, the cache is gone. Every time your agent restructures memory from a chat log into structured format (Note #127, July 13), the prefix the cache was built on no longer matches.

The harness does not just orchestrate the agent. It determines whether your inference layer gets to reuse anything. Note #138 (July 24) showed that the runtime layer is where cache invalidation actually lives. Most harnesses are designed for correctness, not cache continuity. They break the cache to get the right answer, then pay full price to recompute what they just threw away.

Cache invalidation is not a model problem. It is an orchestration problem. The harness decides what stays cached and what gets rebuilt. If your harness does not know it is breaking the cache, it is breaking the cache.

The Idle-Time Opportunity: Reclaiming Lost Agent Compute

The Copilot trace study found something else. The authors designed a lightweight idle-time predictor that captures 86-90% of total idle time across agent sessions. That idle time is not waste. It is the window where resources can be proactively orchestrated.

If you know when your agent is going to idle, you know when to precompute the next turn's cache. You know when to warm the prefix. You know when to swap models without losing the cache window. The idle-time predictor turns the gap between turns into a preparation window, not a dead zone.

The turn boundary is where the cache dies, but it is also where you have time to rebuild it. The 86-90% idle-time capture rate means you can predict the gap and fill it with cache warming, not just wait for the user to type again.

The number to remember: 86-90%. The idle-time predictor captures 86-90% of total idle time across agent sessions. This is the single biggest immediate opportunity for engineers to reclaim lost compute. If you can predict the idle gap, you can fill it with cache warming.

The Memory Connection: Agent State Decay and KV Cache Invalidation

Meta AI recently published research on "behavioral state decay" in agents. Agents forget constraints, repeat failed commands, and rediscover errors they already diagnosed. Meta's proposed solution is a memory-coach agent that tracks state and decides when to remind the primary agent.

The connection to cache is direct. Behavioral state decay is a memory problem. Cache invalidation is an inference problem. They are the same failure at different layers. The agent forgets what it was doing (memory layer). The inference system forgets what it computed (cache layer). Both happen at turn boundaries.

Note #117 (July 3) proved persistent memory beats context windows. Note #143 (July 29) showed KV cache was the inference budget problem. This article connects them: the same turn boundary that breaks your agent's memory also breaks your inference cache. Fixing one without fixing the other leaves you with an agent that remembers what to do but pays full price to recompute how to do it.

The Decision Framework: Architecture Choices for Cache Continuity

Three architectural choices, based on the data:

1. Turn-aware caching: Design your harness to preserve cacheable prefixes across turn boundaries. Keep the system prompt, conversation history, and tool schemas stable across turns. Only invalidate what actually changed. The 90% within-turn hit rate is achievable across turns if the harness is built for it.
2. Idle-time orchestration: Use the 86-90% idle-time prediction to pre-warm caches for the next turn. If you know the agent will idle for 2 seconds, spend those 2 seconds computing the prefix for the likely next tool call. The idle window is free compute time you are currently wasting.
3. Model-switch cache preservation: If your harness routes between models (reasoning model for planning, fast model for execution), cache the shared prefix before the switch. The paper shows model switches cause "drastic" invalidation. A prefix cache that survives model switches turns your routing strategy from a cache-killer into a cache-preserver.

What To Do Today: Actions for Cross-Turn Cache Preservation

1. Audit your agent harness for turn-boundary cache behavior. Measure the actual KV cache hit rate across turns, not just within turns. If you do not know the number, you are paying the 55% rate without knowing it.
2. Identify which harness events break the cache. Tool output compression, model switches, context compaction, and memory restructuring are the four killers listed in the paper. Find which ones your harness triggers and how often.
3. Stabilize cacheable prefixes across turns. Keep the system prompt, tool schemas, and early conversation history identical across turns. Any change to these prefixes kills the cache.
4. Implement idle-time prediction. The paper's predictor captures 86-90% of idle time with a lightweight model. Use that window to pre-warm the next turn's cache instead of waiting idle.
5. Route models without destroying cache. If you switch between a reasoning model and a fast model, cache the shared prefix first. The switch should preserve what did not change, not rebuild everything from scratch.
6. Connect your memory layer to your cache layer. If your agent uses structured memory (Note #127), make sure the memory retrieval does not restructure the prompt in a way that invalidates the cache. Memory and cache are the same problem at different layers.

The Uncomfortable Question: What Is Your Cross-Turn Cache Hit Rate?

Your agent makes 761 million LLM calls across 13 million sessions. You optimized the model. You optimized the prompt. You optimized the tools. But you never looked at what happens to your cache when the turn ends. How much of your inference bill is paying to recompute tokens you already processed? If you do not know the number, that is the answer.