Multi-Party Calls

Step 1Three shapes of second voice, and you designed for none of them.

Multi-party is not one scenario. It arrives in three, with different failure modes:

• The hand-off. The verified caller passes the phone or the handset to someone else — a spouse, a colleague, a carer, a parent. Identity changes; the agent's state, slots and authorisation do not.
• The join. A third party is conferenced in, or the caller was on speaker all along. Now two or more voices are in-band simultaneously, and the agent must decide which one it is answering.
• The bystander. Nobody joined the conversation, but a television, a colleague across the desk, or a child is in the audio. These are the ones that produce phantom turns and, occasionally, phantom instructions.

The first is the expensive one, because it looks like nothing. Audio quality does not change, the transcript stays clean, barge-in behaves, and every metric you watch stays green while the person now instructing your agent was never authenticated. The third is the one that will embarrass you publicly, and it is the voice channel's version of the untrusted-input problem: anything audible is in the prompt.

Step 2The published numbers: realtime agents are near chance at multi-party turn-taking.

This is not a long-tail concern you can defer on the grounds that the models will handle it. Three 2026 benchmarks measured it directly and agree on the shape.

• MP-Bench (arXiv 2609.13076, Findings of EMNLP 2026) evaluates voice agents as participants in multiparty conversation rather than as transcribers of it. Across twelve agents it reports realtime voice agents at or below 22% on multiparty comprehension — an earlier HTML version of the paper states 33% for the same claim — and performance near chance on multiparty turn-taking, with both realtime and non-realtime systems close to random on implicit and negative turn-taking (being addressed indirectly, or deliberately not being addressed).
• MultiPartyTurnBench (arXiv 2610.20172) benchmarks floor control, and reports learned systems reaching 82.5–92.6% end-of-turn recall at 4.3–11.8% false-positive rates — respectable — while interruption detection fails in both directions, missing takeovers or firing spuriously.
• MSI-Bench (arXiv 2609.24812) covers multi-speaker interaction across 1,152 test cases split evenly between Mandarin and English, which matters if you are shipping the bilingual case discussed in multilingual voice agents.

Read the three together and the engineering conclusion is unambiguous. The thing the models are worst at is not hearing two people — streaming diarization does that adequately — it is knowing which utterances are addressed to it. "Near chance on negative turn-taking" means an agent cannot reliably tell that two humans are talking to each other rather than to it. So do not design a multi-party flow that depends on the model inferring the addressee. Make addressing explicit, or constrain the call to one party.

Step 3Speaker labels are provisional, so never bind a permission to one.

The subtle defect is in the streaming-diarization contract itself. Providers that offer realtime diarization label speakers inline and revise earlier labels as more audio arrives — AssemblyAI documents inline labelling of up to ten speakers with exactly that revision behaviour, and Speechmatics ships realtime diarization in-engine with a LiveKit integration aimed at speaker-aware agents. Revision is the correct engineering choice for transcript accuracy and a disaster for authorisation.

The consequence: at the moment you act, the label is a guess that may be corrected afterwards. The utterance you attributed to the verified account holder can be re-attributed to the other person in the room two seconds later, after the tool call has run. Your trace will show the final labels and will not record that the decision was made under a different one.

# Wrong: a diarization label is not an identity.
if turn.speaker == "SPEAKER_0":          # may be revised later
    await transfer_funds(turn.args)

# Right: bind to a party you verified, and record the label you acted on.
party = session.parties.resolve(turn)        # label -> party, or UNKNOWN
if party.verified_for(action) and not party.stale(max_age="120s"):
    await act(turn, on_behalf_of=party.id,
              label_at_decision=turn.speaker, labels_final=False)
else:
    await say("I need to confirm who I'm speaking with before I do that.")
    await reverify(party)

Two rules fall out. A new or changed speaker label is an event, not a transcript attribute — it should hit the same code path as a transfer. And UNKNOWN must be a first-class party value that fails closed for anything consequential, because "I could not attribute this utterance" is the common case in a noisy room, not an exception.

Step 4Make authorisation a property of the utterance, with an explicit party scope.

The fix is the same one that works for agents generally: stop treating authorisation as session state. A call-level grant is ambient authority — everyone audible inherits it — and the voice channel makes the inheritance invisible because there is no UI to show who is holding the permission.

Model the call as a set of parties, each with its own verification state, and give every action a required scope:

• Read-only, non-personal (store hours, policy questions) — any party, no verification. Most of the call lives here.
• Read, personal (balance, order status, appointment details) — the verified account party only, re-asserted if the speaker label changed since the last verification.
• Write or irreversible (payments, address changes, cancellations) — the verified party, a fresh proof on this utterance, and an out-of-band confirmation per caller authentication's argument that the proof should not travel over the audio the attacker controls.
• Third-party authority (a spouse, a carer, a power of attorney) — a pre-existing delegation record, not something an agent grants mid-call. If there is no record, the correct answer is a human.

The practical test is one line of policy: a consequential action requires a party scope, and a party scope requires a verification newer than the last speaker-label change. That single rule closes the hand-off bypass, and it needs no model capability at all — which is the point, given the benchmark numbers in STEP 2. It is the voice instance of the task-scope discipline, and the delegation case belongs with the records described in delegated access and consent records.

Step 5Disclosure and consent are re-triggered by a party join, and your code probably does not know.

Two obligations you already implement for the start of the call reattach when the population of the call changes, and both are easy to miss because nothing in the audio signals a state transition.

Disclosure. A person who joins mid-call has not heard your opening disclosure that they are speaking to an AI system. The disclosure page already argues for a must_disclose predicate re-evaluated at transfer, party-join and resume rather than a line in a system prompt; multi-party is the case that makes the predicate earn its keep. Say it again, short, to the new party, and log it on final frame.

Recording consent. In all-party-consent jurisdictions, the consent you captured covers the parties who were on the call when you captured it. A new voice entering a recorded call is a new consent requirement, and the buffer-gating argument in recording, consent and redaction applies per party rather than per call. The implementation is unglamorous: a party-join event pauses the recording sinks until consent is re-captured, which means your recorder has to be gateable mid-stream.

Step 6Evaluate it, and know when to refuse the multi-party call outright.

Your golden set is almost certainly single-speaker, which means your evals cannot see any of this. Build the multi-party slice from recorded audio the way evaluating voice agents builds the rest: real calls where a hand-off, a conference join or a loud bystander actually happened, scored on three things that are not word error rate — did the agent detect the party change, did it withhold the consequential action, and did it re-disclose. Add an adversarial slice where a second voice asks for something the first was not entitled to; that is a prompt-injection test wearing a telephone.

And accept that for some flows the right answer is to decline. Refuse the multi-party path when the action is irreversible and the delegation record does not exist, when the jurisdiction requires all-party consent and you cannot re-capture it mid-call, or when the call is an emergency or crisis contact where the correct move is a blind transfer regardless of who is speaking. Declining is a designed branch: "I can help with that, but I need to speak with the account holder — shall I call back, or would you like me to put you through?" costs one turn and removes the whole class of bypass.