Your Decode Phase Is Memory-Bound. Speculative Decoding Is the Fix.

Liquid AI just published numbers that should make you rethink your inference stack. Their LFM2.5-DSpark checkpoints deliver up to 3.18x throughput on GPU and 2.87x on-device. Function-calling latency for the 2.6B model dropped 57%. The output quality did not change. The model weights did not change. The only thing that changed was how tokens get generated.

The technique is called speculative decoding. A small, fast draft model proposes several tokens ahead. A large verifier model checks them all in one parallel pass instead of generating tokens one at a time. If the draft model guessed right, you just decoded multiple tokens for the cost of one forward pass. If it guessed wrong, you fall back to the big model’s output and lose nothing.

This matters because your decode phase is memory-bound. We covered this in Note #161: prefill and decode are two different machines. Prefill is compute-heavy. Decode is memory-bandwidth-limited. Every single token your model generates requires loading the entire model weights from memory. The GPU sits idle waiting for memory most of the time during decode. Speculative decoding fixes that by sharing the weight-loading cost across multiple tokens at once. The payoff is largest at low concurrency, where a batch size of one makes decode almost fully memory-bound, whereas at batch sizes of 16 or higher, parallel requests already saturate memory bandwidth and push the workload closer to compute-bound territory, where draft verification yields diminishing returns.

Why Your Decode Phase Is the Bottleneck

The memory-bound decode problem is easy to describe in concrete terms. During autoregressive generation, each token requires a full forward pass through the model. That means reading every weight from high-bandwidth memory into the compute units. For a 7B model, that is roughly 14 GB of memory traffic per token. The GPU does a tiny amount of math and then waits for the next batch of weights to arrive. That is why utilization tanks during decode. The GPU is not thinking hard. It is waiting on memory.

This is the same problem we flagged in Note #143, Your Agent’s KV Cache Is Eating Your Inference Budget. The KV cache problem and the decode-phase memory problem are the same underlying issue. Memory bandwidth, not compute, is the binding constraint on inference speed. The industry has been treating inference as a compute problem. It is a memory problem.

The DSpark numbers make the point concrete. A 3.18x throughput jump means the GPU was spending most of its time idle during standard decode. Speculative decoding fills that idle time with useful work. The 57% function-call latency cut matters even more, because agent pipelines make many small model calls. Latency compounds. A 57% cut per call across a 10-step agent pipeline is not 57% faster. It is exponentially faster.

How Draft-Verify Works

Here is the mechanism, plain enough that you could quote it back to anyone.

Speculative decoding uses a small draft model to propose tokens that a larger verifier model accepts or rejects in parallel, turning sequential autoregressive decode into batched verification. It breaks down into four steps.

First, a small draft model generates N candidate tokens. These are cheap to produce because the draft model is small and fast. In DSpark’s case, the draft is a lightweight checkpoint.

Second, the large target model verifies all N tokens in a single forward pass. This is the key insight. Verifying N tokens in parallel costs roughly the same as generating one token the standard way, because both require one full weight load from memory.

Third, the accepted tokens, where the draft model guessed the same thing the target model would have produced, are committed. Rejected tokens are discarded, and the target model’s output for the first rejected position is used instead.

Fourth, you decoded N tokens for the memory cost of one forward pass. The only overhead is the draft model’s compute, which is negligible because the draft model is small.

The DSpark implementation keeps quality identical because the target model still makes the final decision on every token. The draft model only proposes. The target model verifies. No approximation. No quality loss. Just better use of idle memory bandwidth.

The specifics are worth noting. DSpark ships checkpoints for three LFM2.5 models: 1.2B-Instruct, 2.6B, and 8B-A1B. It is open-sourced with day-one support for llama.cpp and SGLang. That means you can try this today, not in six months.

Why This Beats Bigger Models

Note #82, The Parallel Brain, argued that smarter inference beats bigger models. Speculative decoding is the concrete implementation of that thesis. You are not making the model bigger. You are making the decode phase more efficient by using idle memory bandwidth that was already there.

The economics make the same argument. The widely discussed essay on what happens when the cost of intelligence drops 100x argues that when inference costs collapse, value shifts to distribution, data, and orchestration. Speculative decoding is one of the mechanisms driving that collapse.

Nvidia’s AVO scoring 100% on the ARC-AGI-3 interactive reasoning benchmark signals that the frontier is moving toward interactive, tool-using reasoning, not static language prediction. Agents that call tools make many small model calls. Latency per call is the user-visible bottleneck. A 57% latency cut on function calls is not a nice-to-have. It is the difference between an agent that feels instant and one that feels broken.

DeepSeek’s experimental v4-flash-vision model reinforces the same pattern. Efficient, open-weight models are adding capability without scaling parameters. The efficient-architecture playbook keeps expanding, and speculative decoding is a key play in it.

The Draft-Model Choice

The draft model is the critical design decision. It needs to be fast enough that the overhead of generating draft tokens is less than the time saved by batched verification. And it needs to be aligned enough with the target model that acceptance rates are high. Low acceptance rates mean the draft model wastes compute proposing tokens the target model rejects.

The DSpark approach uses checkpoint-based draft models from the same model family, which gives natural alignment. Other approaches train a dedicated draft model. The tradeoff is straightforward. A dedicated draft model can be optimized for acceptance rate but adds training cost. A checkpoint from the same family is free but may have lower acceptance on out-of-distribution inputs.

Here is practical guidance. Start with a checkpoint from the same family, like DSpark. Measure acceptance rate on your actual workload. If it is above 60%, you are winning. If it drops below 40%, your draft model and target model disagree too often and you are wasting compute.

What This Means for Your Agent Stack

Tie this back to the harness thesis from Note #124. Your agent’s harness is your real model. Speculative decoding is a harness-level optimization, not a model-level one. You do not change your model. You change how tokens get generated. That is exactly what “the harness is the real model” means in practice.

For agent pipelines that make many small calls, tool use, routing decisions, and function calls, the 57% function-call latency cut is the number that matters. Agents do not make one big call. They make dozens of small ones. Latency compounds across the pipeline.

Because DSpark is open-sourced in llama.cpp and SGLang, this is not a cloud-only optimization. You can run speculative decoding on your own hardware, on the edge, on a Jetson Orin Nano rack, the setup from Note #113. The same memory-bound decode problem that hurts cloud inference hurts edge inference more, because edge devices have less memory bandwidth to spare.

What to Do Today

1. One. Pull the LFM2.5-DSpark checkpoints from Hugging Face. They are open-source with day-one llama.cpp and SGLang support.
2. Two. Benchmark your current decode throughput. Measure tokens per second on your actual production workload, not a synthetic benchmark.
3. Three. Enable speculative decoding and re-measure. The DSpark path adds minimal memory overhead. Compare throughput and latency, especially on function-calling paths.
4. Four. Check your acceptance rate. If the draft model and target model agree less than 60% of the time on your workload, try a different draft model size.
5. Five. Profile your decode phase for memory bandwidth utilization. If your GPU sits below 50% utilization during decode, you are memory-bound and speculative decoding will help. If utilization is already high, your bottleneck is elsewhere.
6. Six. Roll speculative decoding into your agent pipeline’s function-call path first. That is where the 57% latency cut lands hardest.

The Uncomfortable Question

Your GPU spends most of its time during decode waiting for memory, not thinking. You bought it to think. How long are you going to let it sit idle?