Voice Agent Latency Is Not the Model. Voxtral 3 and Eleven Realtime Brought the Round Trip In House

Two voice agent stacks shipped within hours of each other this week and both make the same bet. Mistral released Voxtral 3, speech to text and text to speech models wrapped in a voice agent SDK that streams listening, reasoning, tool calls and speaking through one pipeline across more than 50 languages. ElevenLabs released Eleven Realtime, two speech to speech models built for live conversation, with v1 Max claiming 98.8 percent on Big Bench Audio at 0.57 seconds to first speech.

One day, two companies, one conclusion. Voice agents stopped being an integration project and became a product you buy or host. The category did not get incrementally better this week. It consolidated.

Here is the part almost nobody wrote down. The text round trip did not get deleted. It got internalized. ElevenLabs' own documentation says Eleven Realtime models are not fused, that they use a cascaded architecture where each component is a distinct model. Mistral does not even pretend, because Voxtral 3 is not a speech to speech model. It is a transcription model, a synthesis model and an SDK, sold as one thing and tuned as one thing. Neither company removed the text step. Both removed the vendor boundary around it, and that boundary is where your latency lives.

WHERE THE LATENCY ACTUALLY GOES

Build the classic voice agent and you are assembling three vendors in a trench coat. A transcription model turns audio into text, a language model writes a reply, and a voice model speaks it. Three companies, three APIs, three auth tokens, one bill.

Every turn walks the same path. Audio streams to the transcriber, which returns text or partials you debounce. Voice activity detection decides the caller finished, and it is wrong at exactly the wrong moment. Your orchestrator packages the transcript with history and posts it to the model provider. That call queues, and the first token arrives after a time to first token that is never in the brochure. You buffer until you have a sentence, because half a clause sounds broken. Then it goes to the voice vendor, another hop, another queue. The audio finally starts.

Four boundaries, minimum, and every one is a place where a packet waits. None of them are the model thinking.

So name the thing. The Voice Agent Latency Budget has three stages. Listening is the time from the caller stopping to a usable transcript plus an end of turn decision. Deciding is time to first token plus however long you wait to assemble something worth saying. Speaking is time to first audible syllable. The text round trip sits between listening and deciding and again between deciding and speaking, and on a three vendor stack it is the dominant term in that budget, not the model.

That is why a 0.57 second number is plausible at all. When a vendor quotes latency, your first question is not how fast. It is which stages it measures and where the clock starts.

Pixels were the middleman on screens and text is the middleman in audio. Your Agent Does Not Need Eyes made that case for screens earlier this month. The difference is that a screenshot can be removed outright, while the text hop in a voice agent can only be moved inside a vendor you now rent, and Your Token Budget Is Coming is the same argument about a different line item.

WHAT SHIPPED THIS WEEK

Voxtral 3 is the larger release in surface area. Voxtral Small Transcribe 3 handles diarization, context biasing and end of turn detection across 50 languages, including 24 official EU ones, and Mistral claims it beats Whisper, Scribe v2 and Gemini 3.5 Transcribe. A realtime variant transcribes in under 500 milliseconds. Voxtral Mini TTS 3 covers 35 plus languages and 50 plus dialects at $13 per million characters. The SDK puts tool calls inside a live conversation, so the agent can book a slot mid call.

Voxtral 3 is available through Mistral's EU hosted API, or for self deployment on your own infrastructure. An enterprise that cannot legally send call audio to a US cloud can now run the whole voice stack inside its own walls, under its own retention policy. That is who Mistral is aiming at.

Eleven Realtime is the sharper engineering claim. Two models, v1 Mini for cheap conversation and v1 Max for multi step work, both covering 90 plus languages and able to use 17,000 plus voices. Artificial Analysis ran it, which matters. Max took 98.8 percent on Big Bench Audio and 87.1 percent on tool tasks. Mini took 75.9 and 60.3. On conversational dynamics the order flips, with Mini at 95.8 and Max at 94.1, the honest kind of result vendors bury. Max starts speaking in 0.57 seconds on average, Mini in 0.62.

ElevenLabs says no model on the leaderboard is both faster and more accurate, and that framing is precise. Max is not first on accuracy. It sits third, behind StepAudio 3 Realtime and Qwen Audio 3.0 Realtime Plus. The claim is about the Pareto frontier, and it held on launch day. Say it as the best model in the world and you are repeating marketing.

Then read the compatibility page, because it tells you what the product is. Eleven Realtime uses a cascaded architecture, each component a distinct model, optimized to work together. Recognition, voice activity detection, turn taking, the language model and the voice model are all still in there. They live in one runtime and one latency budget instead of three.

IBM closes the picture from the open side. Granite Speech 5.0 shipped as three Apache 2.0 models trained on 390,000 hours of public audio. The 2B model transcribes six languages and translates to and from English, takes keyword lists up to 2,000 terms so product names and acronyms survive, and returns capitalization with F1 between 89.7 and 99.5 on IBM's test sets. A Turbo CTC variant covers edge throughput and a 2B Plus model covers speaker attribution. The boring layer went open weight too, and that is the concrete answer to two of the three questions in the rubric below. Apache 2.0 weights are what let you take the host path without asking a vendor's permission, and what give the assemble path real switching leverage, because the model you depend on can be replaced with another checkpoint instead of renegotiated with a sales team. Open weights are how lock-in dies when you are the one holding the stack.

WHY THE CATEGORY CLOSED IN ONE DAY

The interesting fact is not either release. It is the simultaneity. Two competitors shipped the same architectural answer within hours.

When a market converges in one news cycle, the problem was solved a while ago and everyone was waiting on the packaging. The three vendor stack existed because each piece was hard. Transcription needed a specialist, synthesis a different one, orchestration a third, and the seams were the product.

The pieces stopped being hard. Transcription is close to solved for clean audio, synthesis got expressive enough to pass, and turn detection became engineering instead of research. Once the pieces are commodities the only value left is in the seams, and the fastest way to own a seam is to absorb both sides of it. Neither company invented an architecture. Both collapsed the contract count from three to one.

That is what already hit web hosting, then vector databases, then agent harnesses. You used to assemble a stack. Then somebody sold you the stack with a nicer control panel, and the work moved from your team to their roadmap. Voice agents just had their day.

The consumer edge is already moving on top of it. Yelp's Check for Me sends a voice agent to call a restaurant, ask about wait times and text you the answer, with party size and a callback number as the only inputs. An agent that phones a business has to survive hold music, a distracted host and a human who interrupts. Sub second conversation is not vanity there. It is the difference between a product and a demo.

THE BUY, HOST, ASSEMBLE RUBRIC

Three questions, in this order.

First, can your call audio legally leave your building or your country? If not, you host. Voxtral 3 self deployed, or Granite Speech 5.0 under Apache 2.0, and you build the glue. This question outranks everything else, because a latency win you cannot legally deploy is not a win.

Second, is voice your product, or a feature of something else? If voice is the product, buy the integrated stack and spend engineers above the audio layer. If voice is a feature on a product that already makes money, assemble from the open pieces so you keep switching leverage when the frontier moves again.

Third, do you need 50 languages or five? Multilanguage depth is the strongest card on the buy side. Voxtral covers 50 plus in transcription, Eleven Realtime covers 90 plus, Granite covers six well and translates a few more. If your roadmap is one country, paying for ninety languages is a tax. If it is the EU, that is the only reason to pick any of these.

Now the caveats, because this is not a vendor brief. Every benchmark here was either paid for by the vendor or run on the vendor's own test sets, and none of it has independent replication. Artificial Analysis ran the ElevenLabs numbers, which beats self reporting. Keyword biasing and speaker attribution stay separate purchases at most shops. And self hosting is an operations commitment, not a download. You are on the hook for GPU capacity, autoscaling under call spikes, upgrades that change voice character, and a pager at 2am.

WHAT TO DO TODAY

1. Draw your current voice pipeline and count the network hops per turn.
2. Test one integrated model against your three vendor pipeline on the same recorded conversation, and measure time to first audible syllable, not time to first token.
3. If you handle regulated audio, price a self hosted Voxtral 3 pilot before you sign your next transcription contract, and put the sovereignty requirement where procurement can see it.
4. Pull the Apache 2.0 Granite Speech models and test them on your own vocabulary and accents.
5. Write your latency budget down per stage, listening, deciding, speaking. You cannot cut what you never counted.

THE UNCOMFORTABLE QUESTION

The text round trip is still there. It now lives inside one vendor's runtime and one contract instead of three, which is a different problem rather than a smaller one, because the latency you can measure is the latency you can fix and you just lost the ability to measure the middle. That loss is operational, not academic. On a three vendor pipeline an incident trace gives you a waterfall with the stall sitting in a hop you own, so you can point at the queue and fix it. Once the pipeline is internalized you get a completed conversation and a total duration, and the queueing that used to show up in your own graphs happens in a runtime you cannot instrument. The first honest answer about where 900 milliseconds went arrives in a support ticket, days later, after the customer already left. So what is your voice strategy besides we will add it later? If your answer starts with wiring three vendors together, you are rebuilding the pipeline this week's releases replaced.