Turn-Taking & Barge-In
Turn-taking and barge-in: the hard part is not detecting the interruption, it is that your conversation state now disagrees with what the caller heard.
Every other component in a voice stack is a stream processor you can tune in isolation; the turn manager is the only one holding mutable state, and a barge-in invalidates it mid-write. Three positions diverge at that instant — what the model generated, what the synthesiser produced, and what actually reached the caller's ear — and if your history records the first, the agent will spend the rest of the call referring confidently to sentences nobody heard. That is why most "the agent is confusing" tickets are state bugs wearing a detection bug's clothes, and why the fix is an asymmetric truncation rule rather than a better threshold.
Step 1VAD is not endpointing, and conflating them is the root bug underneath the state bug.
Two distinct jobs, constantly merged into one config value:
- VAD (voice activity detection) answers "is there speech in this frame?" — a low-level, sub-100 ms signal. It cannot know whether a turn is over, because a thinking pause is silence too.
- Endpointing answers "is the caller done, such that I should respond now?" — a decision built on top of VAD plus, ideally, the words and their meaning.
Endpointing on raw silence is the canonical failure: a 600 ms threshold cuts off anyone who pauses to think ("my account number is… 4 4 7…"), and 1,500 ms makes the agent feel asleep. There is no fixed value that is both responsive and patient, because mid-turn pauses and turn-final gaps come from two overlapping distributions that also shift per speaker. End-of-turn detection has the full argument and the reason a threshold is the wrong knob; the operational consequence here is that a turn boundary is a prediction, which means it is sometimes wrong, which means your state model has to survive being wrong.
Step 2Use a model for the boundary, and treat its output as a probability rather than an event.
The 2026 answer is a model that predicts utterance completion and modulates the wait. Three shapes ship in production, and the difference between them is what they consume. OpenAI's Realtime API exposes a semantic_vad turn-detection mode with an eagerness setting alongside the older server_vad. LiveKit's turn detector runs on the audio directly, fusing acoustic and semantic cues rather than waiting for a transcript. Deepgram's Flux folds end-of-turn detection into recognition itself with configurable thresholds, documenting end-of-turn at roughly 260 ms and under 400 ms multilingual.
# turn/endpoint.py — the threshold is an output, not a setting def end_of_turn(partial, silence_ms): p = complete_utterance_prob(partial) # acoustic + semantic if p > 0.85: return silence_ms > 120 # sounds finished: fire fast if p < 0.30: return silence_ms > 1400 # "uh, my number is..." — wait return silence_ms > 600 # unsure: middle ground # what this buys you is not accuracy, it is a KNOB WITH A UNIT: # raise the p cut-offs -> lower false cutoffs, higher response latency # lower them -> snappier, and you clip people mid-account-number # Pick the operating point from your own false-cutoff budget, not from a default.
"What's the weather" with falling intonation ends a turn in about 120 ms; "my card number is" with a trailing rise should buy the caller over a second of thinking room. Same silence, opposite decision, because the words carry intent that silence cannot. Note what the model does not buy: it does not eliminate wrong boundaries, it moves them onto a curve you can choose a point on. The latency budget is where that point gets paid for.
Step 3Barge-in is non-negotiable, and the audio part is the easy half.
If the caller starts talking while the agent is speaking, the agent must stop within a couple of hundred milliseconds rather than finish its sentence. An agent that talks over an interruption is experienced as rude and then as broken, and people hang up. The mechanical sequence is well understood:
agent speaking ............ caller starts talking
|
+-- detect speech (VAD, <100ms)
+-- STOP agent audio immediately
+-- FLUSH unsent TTS / cancel the LLM stream
+-- RECONCILE state to what was actually heard <-- the hard one
+-- listen; the caller now has the floor
Four of those five lines are plumbing you write once. The fifth is the one that generates bug reports for the next two quarters, and the reason is that at the moment of the barge-in there are three different answers to "what did the agent say?", separated by real buffers:
- Generated. Every token the model produced. The LLM streams far ahead of playback, so this is routinely the whole reply.
- Synthesised. What TTS has turned into audio. Ahead of playback by whatever you buffer to avoid underruns.
- Emitted and heard. What left the speaker before the stop took effect — minus whatever is still sitting in the jitter buffer and the network, which the caller never got.
Step 4Reconcile to what was heard, and bias the estimate — the error costs are not symmetric.
Put numbers on the gap, because it is much larger than it feels. Conversational speech runs around 150 words per minute, or 2.5 words per second:
# how far "generated" runs ahead of "heard" at a barge-in speech_rate = 2.5 # words/second detect_stop = 400 # ms to notice and silence the output playout_buf = 200 # ms of audio already in flight, never heard tts_lookahead = 1500 # ms synthesised but not yet played uncertain_words = speech_rate * (detect_stop + playout_buf) / 1000 # = 1.5 unheard_words = speech_rate * tts_lookahead / 1000 # = 3.75 # and the LLM is further ahead still: if it streamed 10 sentences while # playback reached sentence 3, SEVEN sentences are in `generated` and were # never heard. Store `generated` as history and the agent now believes it # explained something it did not.
The reconciliation rule follows from an asymmetry. Record a word as heard when it was not, and the agent references material the caller has no memory of — which is not a small glitch, it is the agent appearing to hallucinate its own side of the call, and it poisons every later turn. Record a word as unheard when it was, and the agent repeats itself slightly. One of those is incomprehensible and the other is mildly redundant, so bias the estimator toward "not heard": truncate at the last playback position you have positive confirmation of, drop the partial word, and throw away everything synthesised-but-unplayed. Then mark the truncation in the history explicitly rather than silently, so the model knows the turn was cut:
# what goes in the transcript after a barge-in {"role": "assistant", "content": "Sure — your balance is $412.60 and the payment that", "truncated": True, # the model must see that it was cut "heard_ms": 2140} # confirmed playback, not synthesis # the `truncated` flag is load-bearing: without it the model reads a # mid-sentence stop as its own choice and will not offer to finish.
Two side effects to handle while you are in here. A tool call the model emitted in the discarded tail must not execute — cancel the stream before it reaches the tool runner, not after. And if the truncated turn contained a disclosure you are obliged to make, the obligation is unmet: disclosing the agent on a call and recording consent both have scripts that a barge-in can cut in half, and a compliance script needs re-delivery rather than a resumption.
Step 5Not every noise is a bid for the floor, and echo cancellation is a prerequisite rather than an audio nicety.
Real conversation is not strictly half-duplex. People say "mm-hm" and "right" while the other is talking — backchannels — and these are not barge-ins; treating every "uh-huh" as an interruption makes the agent stop constantly and is one of the most common causes of a stack that feels twitchy. A good turn manager distinguishes a continuer ("mm-hm", keep going) from a genuine bid for the floor ("wait — actually…", yield now), and the cheap first approximation is duration plus energy: sub-300 ms, low-energy, non-lexical events are continuers until proven otherwise.
Underneath that, the agent's own voice returns through the caller's microphone, especially on speakerphone. Without acoustic echo cancellation, VAD hears the agent as the caller, fires a false barge-in, and the agent interrupts itself into a stutter loop. A turn-taking bug report is very often an echo bug, and on a narrowband PSTN leg the acoustics are worse than your tests — audio quality and codecs covers what the channel does to the signal your detector sees. Check AEC before you touch a threshold.
Two tuning rules that save weeks. Barge-in sensitivity should depend on what the agent is currently saying: while reading back a long number or a legal disclosure, require more evidence before yielding, because a false cutoff there costs a restart of the whole read-back — see alphanumerics over voice. And never tune eagerness globally: a debt-collection IVR wants fast and decisive, a grief-support line wants slow and patient, and shipping one tuning for both is the only real mistake available.
Step 6Full-duplex removes the boundary and not the state problem. Measure both faces of one curve.
Native speech-to-speech models move toward genuine full-duplex: listening and speaking at once, the way people do, with no explicit endpointing step to tune. That is more natural and considerably harder to operate, and it is worth being clear about what it fixes. It dissolves STEP 1 and STEP 2 — there is no threshold, because there is no discrete boundary. It does not dissolve STEP 4: something still has to record what the caller actually heard, and now the truncation point is continuous rather than turn-aligned, so the bookkeeping gets harder rather than easier. Full-duplex speech covers the model side. Until the operational story matures, most production stacks run a disciplined half-duplex turn manager with fast barge-in and explicit backchannel handling, because a predictable agent beats an unpredictable one.
Whichever you run, instrument four numbers and read the first two as one curve:
- False-cutoff rate. Share of turns where the agent responded while the caller was still speaking. The number everyone forgets to measure, and the one that silence thresholds are optimised against by accident.
- Response latency at the chosen operating point. Paired with the line above, always. A latency figure without a false-cutoff figure is unfalsifiable, because you can always buy one with the other.
- False barge-in rate. Agent stops with no real bid for the floor. Usually echo or backchannels, and it has a different owner than the two above.
- Reconciliation defects. Turns where the agent referenced content the caller did not hear. Find them by replaying recordings against stored history and diffing — this is the measurement nobody has, and it is where the complaints come from.
Do it in this order. Separate VAD from endpointing in your code so the two have different owners, then replace your silence threshold with a completion model and expose the operating point rather than the millisecond value. Fix AEC before tuning anything. Build the reconciliation path next, not last: track generated, synthesised and confirmed-played positions separately, truncate to confirmed playback, bias toward "not heard", set an explicit truncated flag in the history, and cancel the LLM stream before any tool call in the discarded tail can run. Treat continuers as continuers by duration and energy. Make barge-in sensitivity depend on what the agent is saying, and tune eagerness per deployment, never globally. Then measure false cutoffs and latency as one paired number, false barge-ins separately, and reconciliation defects by replay — evaluating voice agents has the harness. If you keep one sentence: detection decides when the agent stops talking, and reconciliation decides whether the rest of the call makes sense.