The Latency Budget

7 min read

V2
Playbook · Voice & Realtime Agents

The latency budget: where the sub-second turn actually goes.

Add up the latency figures the voice vendors published through 2026 and a single turn comes to roughly 1.1 seconds before a packet has crossed the internet — which is how a stack assembled entirely from “fast” components ships at twice the conversational gap a human expects. Almost every number in this market measures one component on the vendor's own path; the turn your caller feels is a sum of seven lines, and the two largest are the two nobody publishes. This page is the accounting exercise that makes the missing lines visible.

Step 1Measure the turn. Nobody else does.

Human turn-taking in natural conversation runs a gap of roughly 200 ms, and people register discomfort well before a full second. The working target for a voice agent's response latency — caller stops speaking, first audio comes out — is therefore sub-second end to end, with the best native stacks pushing the model's own contribution into the low hundreds.

That is the metric. It is not the metric anyone sells against. The published figures fall into three kinds, and keeping them apart is the first thing that saves you a quarter:

  • Component latency on the vendor's path. A time-to-first-audio or an end-of-turn figure measured from inside the vendor's own network, excluding your transport. Honest, useful for ranking, and not a thing your caller can experience.
  • Time to first byte, which is not time to first audio. A server can return response headers in 50 ms while the first audio samples arrive 200 ms later. A TTFB quote systematically understates what the ear waits for, and the two are routinely reported under the same heading.
  • Model TTFT in isolation. The number model vendors publish. In a voice turn it is one line of seven, and usually not the biggest.

Response latency is end-of-user-speech to first-audio-out. It is not time-to-full-response, and it is not any component figure. Critically, it includes endpointing delay — the single largest slice in most deployments and the one no vendor has an incentive to report, because it is a property of your policy rather than their model.

Step 2The budget, line by line.

The turn is a sum. You cannot improve a total you have not decomposed, so decompose it and write today's plausible value next to each line.

# response_latency = sum of these, ms
network_in        # mic/carrier → your edge
endpoint_detect   # "is the user done?" — usually the biggest
stt_finalize      # cascade only; 0 for native S2S
model_ttft        # prompt → first output token/frame
tts_first_audio   # cascade only; first synthesized chunk
network_out       # your edge → speaker/carrier
playout_buffer    # jitter buffer before audio plays

For a cascade every line is live. For native speech-to-speech, stt_finalize and tts_first_audio collapse into model_ttft — the structural reason native stacks are faster is not a better model, it is two fewer serial hops. Each hop costs its own processing plus a network round trip you pay whether or not the hop does much work.

Two of these lines are invisible in every vendor comparison you will read. endpoint_detect belongs to your configuration. network_in, network_out and playout_buffer belong to the carrier, the caller's connection and your jitter policy. On a PSTN call across a continent these are not rounding errors, which is why vendors with points of presence near the carrier sell proximity as a product: one telephony provider claims that co-locating a speech model in its own POPs saves 100 to 300 ms against a cloud deployment. Take the figure as a vendor claim; take the direction as arithmetic.

Step 3Endpointing is the biggest line, and in 2026 it stopped being a timer.

The agent cannot start until it believes the caller has finished. Naive silence detection waits out a fixed timeout — say 700 ms of quiet — and that timeout is added directly to every single turn, whether the caller paused mid-thought or genuinely stopped. It is the only line in the budget you pay in full on the easy turns as well as the hard ones.

This is why the market moved. A model-based detector predicts that the utterance is complete rather than measuring how long it has been quiet, so it can fire in a few hundred milliseconds on a finished sentence and hold the floor through a long pause on an unfinished one. The 2026 architectural step was folding the prediction into recognition itself: Deepgram's Flux performs transcription and turn detection in one model, with no separate voice-activity-detection and endpointing stage, and Deepgram documents end-of-turn decisions at roughly 260 ms — its multilingual variant at under 400 ms across ten languages. Those are vendor figures on the vendor's path, per the caveat above, but the structural win is real and independent of the number: one fewer serial hop, and a decision that scales with content rather than with patience.

Tuning endpointing down to shave latency directly raises the rate at which you cut callers off mid-sentence. This dial trades response latency against interruption rate and has no free setting. Any latency number reported without the false-cutoff rate beside it is half a measurement, and the half that was omitted is the one that got worse. The mechanism is on end-of-turn detection; the correctness side is turn-taking and barge-in.

Step 4Add the published numbers up and you are already over.

Here is the exercise that reorders most roadmaps. Take the most favourable public figures for a cascade stack as they stood through 2026 and sum them honestly.

turn detection + transcription   ~400 ms   (one platform's P50 breakdown)
model first token                ~500 ms
TTS first audio                  ~200 ms   (independent runs: best ~100-200 ms)
                                 -------
subtotal, nobody's network       ~1,100 ms
+ network_in, network_out, playout_buffer
= comfortably past the 1,000 ms target, on components all marketed as fast

Two things follow. First, the sub-second turn is not reached by picking fast components; it is reached by removing hops and by cutting the endpointing line, which is the only one of the four big items that is yours to set. Second, the gap between vendor-quoted and experienced latency is structural, not incidental: one TTS vendor cites about 40 ms for its fastest model and another about 75 ms, while independent-leaning benchmarks of the same category land around 100 to 200 ms median time-to-first-audio — the difference being network and application latency that the vendor figure explicitly excludes and your caller unavoidably experiences.

So the planning rule is a crude multiplier rather than a sum of datasheet values: budget roughly twice any vendor-quoted component figure for the version your caller hears, then measure your own path and replace the multiplier with data. Instrumenting that measurement is the subject of measuring agent latency.

Step 5Streaming hides the latency you cannot remove. The tool call is the cliff you cannot hide.

Some latency is irreducible, and the move is to cover it rather than remove it. Start TTS on the first tokens instead of the full response; run recognition on partial transcripts so the model is already warm when the caller stops; acknowledge in 150 ms with “mm-hm” or “let me check that,” which reads as responsive even when the real answer is two seconds out; draft speculatively on the partial and keep the work if the final transcript matches.

None of that survives a tool call. The budget above describes a turn with no backend work in it; the moment the agent must hit a database or an API you have blown through the conversational gap, because a 400 ms API call on its own is the entire gap.

no-tool turn:   ~500-900ms   acceptable
tool turn:      ~500ms + API + 2nd model pass
                = often 2-4s   → DEAD AIR unless covered

The fix is conversational before it is technical: speak before and during the call (“one sec, pulling that up…”), fire independent tool calls in parallel, and never let the audio channel go silent while a backend works. Developed fully in voice tooling and state.

Step 6Two clocks run on one call, and optimising either can lose the other.

This is the part the single-number framing hides. Response latency is a within-turn number and it is a perception cost — the caller's read on whether the machine is alive. Call duration is an across-call number and it is a cash cost, because the carrier, the platform and the speech layers all meter on wall-clock time. They are different levers, and three common decisions move them in opposite directions.

  • A hold phrase covering a tool call improves the perception clock and lengthens the cash clock.
  • A confirmation turn lengthens both clocks, and is usually still the right call.
  • Dropping to a smaller model shortens both, and buys the saving with accuracy — which lands on a third clock, the escalation rate.

The confirmation case has a measured price now, which makes it the cleanest illustration available. On the τ-Entity spoken-entity benchmark, a scaffold prescribing spelling, read-back, correction and confirmation raises robust Pass3 by 14 to 31 points at a cost of 21 to 28 seconds per call. That is a catastrophe on the perception clock if you spend it in one block, an irrelevance on the cash clock at any realistic per-minute rate, and a large win on the escalation clock. The arithmetic is worked in the cost of a voice minute, and the evidence in the read-back worked, the repair did not.

Spend the seconds between turns, never inside one. Four short confirm-and-continue exchanges at 300 ms response latency feel fast and bill the same as one silent seven-second pause that feels broken. Optimise the gap to ruthlessness and let the call be as long as correctness requires.

Every millisecond you reclaim from endpointing or model size is a millisecond you spent on a higher interruption rate or a worse answer. There is no latency win that is free — only a latency cost you have decided is worth it, and the decision is only honest if you wrote down what the other clock did.