Your Frontier Model Is Wasting Tokens on Plumbing

NVIDIA just shipped a 30-billion-parameter model with only 3 billion parameters active per token, and it was not built for reasoning. It was built for the boring half of agent work: tool calls, result validation, and subagent delegation. The plumbing that makes up most of an agent's runtime.

The model is called Nemotron 3.5 Lightning, and it comes with a routing library called NeMo Switchyard that decides which model in your system should handle each task. The small execution model handles the high-volume plumbing. A frontier model like Nemotron 3 Ultra handles the planning. NVIDIA claims up to four times the output speed of similarly sized models for the execution layer.

If you are running always-on agents and paying frontier-model rates for every tool call, you are burning money on plumbing. We covered harness efficiency in Note #124 (Your Agent's Harness Is Your Real Model) and model routing in Note #141 (Your Router Is the Moat, Not Your Model). This is the next step: a dedicated execution model that splits the agent stack into a thinking layer and a doing layer. The economics of agents are now decided in that split, not in the model you pick.

The Execution Layer Is Most of Your Runtime

Let's be precise about what the execution layer actually is. A long-running agent spends most of its wall-clock time not reasoning. It is making tool calls, validating results, delegating to subagents, formatting outputs, and retrying failed operations. NVIDIA's own engineers state it plainly: long-running AI agents spend most of their time on high-volume execution, and the interesting cognitive work of deciding what to do is a small fraction of the wall-clock. The rest is plumbing.

This is high-volume, low-complexity work. It does not need a frontier model.

The problem is that most agent architectures send everything through the same model. A tool call that takes 50 tokens to describe and 20 tokens to execute goes through a model that charges frontier rates per token. Multiply that by thousands of calls per hour, across dozens of agents, and you are paying premium prices for plumbing.

NVIDIA's argument is that this is a design error, not an unavoidable cost. The execution layer and the reasoning layer have different cost profiles, different latency requirements, and different complexity demands. Running them through the same model optimizes for neither.

Teach yourself the split now, because it is the frame for everything that follows. The reasoning layer handles planning, analysis, and complex decisions. It needs capability. The execution layer handles tool calls, validation, delegation, and formatting. It needs speed and cost efficiency. These are two different jobs with two different price points, and most of you are hiring one senior architect to do both.

Nemotron 3.5 Lightning: The Numbers

Here are the numbers that matter. Nemotron 3.5 Lightning is a 30-billion-parameter mixture-of-experts model with just 3 billion active parameters per token. That means only 10 percent of the model is firing on any given token, which keeps inference cost and latency low. NVIDIA claims up to four times the output speed of similarly sized models.

The mixture-of-experts design is the reason. A router sends each token to just a few of the model's many experts, so only a fraction of the parameters run per token. As NVIDIA puts it, this provides the capacity of a larger dense model at the compute cost of a small one. Thirty billion parameters of knowledge, three billion parameters of arithmetic per token. The model also ships with speculative decoding, so it can draft several tokens ahead and verify them in one pass, plus NVFP4 and BF16 checkpoints for different serving scenarios.

The detail that matters most for agents is the harness-optimized training. The model was trained against the popular agent frameworks it will actually run inside, so its tool calls land in the right format on the first attempt. Every malformed call is a wasted round trip. At the volumes an always-on agent generates, format accuracy is a latency feature. If the tool call works the first time, you skip the retry cycle.

The openness is unusual in degree. Nemotron 3.5 Lightning ships with weights, training data, and recipes under the permissive OpenMDW-1.1 license, plus an open agentic reinforcement learning dataset called Nemotron-RL Agentic Terminal Pivot. The open-source release matters because it means you can run it locally, fine-tune it, and avoid sending execution-layer traffic to a hosted API. NVIDIA positions it as deployable anywhere from a desktop DGX Spark to a data center, and it runs on the standard tooling: LM Studio, llama.cpp, Ollama, and Unsloth.

Now the honest caveat, and you should hold it hard. NVIDIA benchmarks NVIDIA. The "up to four times" figure comes with no independent replication and the usual latitude that "up to" provides. And the harness-optimized training is a double-edged property. A model tuned to the agent frameworks popular in August 2026 will see its specific edge decay as those frameworks change. The general capability is real. The specific advantage is dated the day it ships. The architecture pattern, the split into thinking and doing, is what outlasts the model.

NeMo Switchyard: The Routing Layer

The model is only half the story. The other half is NeMo Switchyard, the routing library that decides which model in your system should handle each task. It pairs the small execution model with frontier models for planning. The router is the component that makes the two-tier architecture work.

NVIDIA states the architecture plainly: frontier reasoning models such as Nemotron 3 Ultra handle orchestration and complex planning, while smaller, more efficient models handle the high-volume execution layer. Switchyard can expose Nemotron 3.5 Lightning as a routing target alongside your open and closed models, so every request lands on the most capable and efficient model that can handle it. Plans route up to the frontier. Execution routes down to Lightning. Your tokens are spent efficiently instead of spent everywhere.

This is distinct from the general model-routing articles we have covered. Note #141 covered inference cost optimization via routing, meaning smaller models, quantization, and workload-specific training. This is different. This is a shipped product that explicitly splits the agent stack into execution and reasoning, with a dedicated model for each. Model routing has been an optimization people talked about for two years. Shipping it as a supported library from the company that sells the hardware turns it into a default assumption.

The routing decision is the cost lever. If the router sends a tool call to the execution model instead of the frontier model, you pay 3-billion-parameter rates instead of frontier rates. Over thousands of calls, that is the difference between a sustainable agent operation and a token-spend crisis. The router decides whether your execution layer runs on the cheap model or the expensive one, and that decision, repeated a million times a day, is your bill.

The Harness Is the Multiplier

The model and the router are one piece of the cost story. The harness is the multiplier on top of both.

Writer launched Palmyra X6 on August 13, built as a post-training variation on Z.ai's open-source GLM-5.2, and it found something important. Writer's research showed that small changes in harness efficiency were often a more reliable way to reduce costs than model choice, cutting costs by an average of 40 percent across the models it tested. Combined with its harness upgrades, Writer estimates customers could cut costs by up to 50 percent for basic tasks.

CEO May Habib put the point directly: the harness is the one component whose efficiency multiplies across every model an organization runs, present and future. That is the multiplier. Model A versus model B is a one-time choice. The harness is the thing that decides how efficiently every model, present and future, spends its tokens.

DeepSeek is moving on the same layer. It open-sourced its Deepseek Harness v0.1 under the MIT license, built on the newly released Cordis plugin system where every feature is a swappable plugin, from tools and sandboxes to sessions and the UI. A continuous session log tracks every prompt, every tool call, and every result. The harness is now an open-source component you can inspect, modify, and optimize.

Watch the convergence, because it is the whole thesis of this article. NVIDIA ships a dedicated execution model with a routing library. Writer proves harness efficiency beats model choice for cost. DeepSeek open-sources its harness. The execution and orchestration layer is where the economics are now decided. The model is a commodity input. The architecture around it is the moat.

The WSJ made the same point about the infrastructure layer in a piece on August 16. Even freely available models still need compute, orchestration, and deployment tooling to run in production. Open-weight adoption may actually expand the addressable market by lowering the barrier to entry. The model gets cheaper. The plumbing around it becomes the business.

The Cache-Hit Cost Trap

Now add the cost pressure that makes this urgent, not optional.

DeepSeek raised its cache-hit prices sharply, and the new rates take effect August 16. Cache hits go from $0.003625 to $0.022 off-peak and $0.044 at peak, shrinking the cache discount from about one-hundred-twentieth of regular input price to about one-thirtieth. For agents that repeatedly read the same files, this is the most expensive part of the change.

Think about what that does to an agent. The cache-hit price increase matters because agents are the heaviest users of cached context. An agent that reads the same file on every turn was paying nearly nothing for the cache hit. Now it pays six to twelve times more. That makes the execution-layer split even more urgent. If you can route the file-reading and tool-call plumbing to a cheaper model, you avoid paying premium cache-hit rates for low-complexity work.

Here is the cost arithmetic you should run tonight. If your agent makes 1,000 tool calls per hour, and each call costs 0.01 cents on the execution model versus 0.10 cents on the frontier model, the execution layer saves you 90 cents per hour. Over a month of always-on operation, that is hundreds of dollars per agent. Scale to dozens of agents and the savings stop being a rounding error and become the difference between a project that ships and a project that gets cancelled for its burn rate. This is not hypothetical. The harness research measured a 40 percent average cost cut from orchestration changes alone. The execution model is the same lever applied to the model layer.

What to Do Today

1. Audit your agent's token spend by layer. How many tokens go to reasoning versus tool calls, validation, and formatting? If the majority is execution, you are overpaying. This is a two-line logging change and it tells you whether you have a problem.
2. Split your agent stack into a reasoning layer and an execution layer. Route planning and complex decisions to your frontier model. Route tool calls, validation, and delegation to a smaller, cheaper model. The split is the architecture.
3. Evaluate Nemotron 3.5 Lightning or a similar small MoE model for your execution layer. The 3 billion active parameters keep cost and latency low. Run it locally if you can, since the weights and recipes are open.
4. Implement a router. NeMo Switchyard is one option. Build your own if you need custom routing logic. The router is the component that makes the two-tier architecture work, and the routing decision is the cost lever.
5. Read DeepSeek's open-sourced Harness v0.1. The Cordis plugin system shows how a production agent harness is structured when every feature is a swappable plugin. You can learn more from a real harness you can read than from a marketing page.
6. Measure harness efficiency, not just model cost. Writer's research shows harness changes cut costs 40 percent across models. Your harness is the multiplier. If you are only tracking model choice, you are measuring the wrong variable.
7. Recalculate your cache-hit economics. If your provider raised cache prices, the execution-layer split is no longer optional. It is a cost survival move.

The Uncomfortable Question

If your agent spends 80 percent of its runtime on tool calls and formatting, why are you paying frontier-model rates for 80 percent of your tokens? You would not hire a senior architect to unclog the office toilet. Stop routing your plumbing through your most expensive model. The model that thinks is not the model that should do.