Streaming Tool Calls in Practice

The common delta shape (and where it stops being common).

All three vendors ship tool-call streaming under the same general pitch: the model emits an initial event that names the tool being called (with an id / call_id / block_id, an index or content-block index, and the function name), followed by a series of delta events that each carry a fragment of the JSON arguments string, followed by a completion event when the tool call is done. Concatenate the fragments in order, JSON-parse the result, dispatch. The K7 vendor matrix covers the non-streaming call shape; streaming reuses the same underlying event content but wraps it in per-vendor SSE or stream envelopes with different names.

Three axes diverge and cause most bugs. First, the granularity of the fragment: OpenAI's arguments deltas can carry one token at a time or a whole line depending on the model; Gemini's chunks tend to be larger and sometimes whole values; Anthropic's are typically small JSON-delta text pieces. Second, the identity of the call: OpenAI uses an index on the tool-call array plus a tool_call_id that appears in the first chunk and repeats; Anthropic uses a content-block index that is stable across the block's lifetime; Gemini uses a candidate/part-index pair. Third, parallel-call interleaving: all three can emit tokens from multiple parallel calls in one stream, but the ordering guarantees and the presence/absence of chunk-completion sentinels are not uniform. The accumulator has to key on the identity, not on positional assumptions, and any that doesn't will silently corrupt parallel calls.

OpenAI accumulation, and the GPT-4.1-nano duplicate-call bug.

OpenAI's Chat Completions streaming shape is stable and well-documented. Each stream chunk carries a choices[0].delta.tool_calls array; each element in that array has an index, and either the first-chunk fields (id, type, function.name) or the ongoing-fragment fields (function.arguments — a partial JSON string that concatenates over chunks). The accumulator keys on index, seeds the entry on first sighting, appends arguments on every chunk that has it, and closes when the outer finish_reason becomes "tool_calls".

# OpenAI Chat Completions — accumulator keyed on tool-call index
calls = {}
for chunk in stream:
    for tc in chunk.choices[0].delta.tool_calls or []:
        e = calls.setdefault(tc.index, {"id": None, "name": None, "args": ""})
        if tc.id: e["id"] = tc.id
        if tc.function and tc.function.name: e["name"] = tc.function.name
        if tc.function and tc.function.arguments:
            e["args"] += tc.function.arguments
final = [{"id": v["id"], "name": v["name"], "args": json.loads(v["args"])}
         for _, v in sorted(calls.items())]

The GPT-4.1-nano duplicate-call bug is worth flagging by name. On some prompts, the nano model emits a second tool-call block with the same index and id as the first, effectively repeating the whole call. The naive accumulator sees the repeat, appends its arguments onto the already-completed args string, and JSON-parse fails because the concatenated string is two JSON objects glued together. The defensive fix is idempotency: once a call's args parse cleanly, treat further chunks with the same id as a no-op. The tracker on the OpenAI SDK repo has a specific report for this pattern; the workaround is a couple of lines and prevents a silent corrupt-call bug you will spend an hour chasing otherwise.

The Responses API changes the shape a little but the accumulator pattern is the same — you key on item_id for a function_call item and append arguments deltas. The typed stream event names are different (response.function_call_arguments.delta), but the state machine is one for one.

Gemini: chunks are larger, aggregation is coarser.

Gemini's streaming emits GenerateContentResponse chunks, each carrying candidates[0].content.parts. Function-call parts appear as functionCall: {name, args} — and the args field, unlike OpenAI's incremental string, tends to arrive as a mostly-complete or whole JSON object in one chunk. The accumulator still has to be tolerant of receiving multiple chunks for the same call (Gemini's newer models occasionally split large arg objects across two chunks), but the common case is one function-call part per stream, arriving late, complete.

# Gemini — accumulate function-call parts, tolerant of coarser chunks
calls = []
for chunk in stream:
    for part in (chunk.candidates[0].content.parts or []):
        fc = getattr(part, "function_call", None)
        if fc is None: continue
        # If Gemini split args across chunks, merge into the last-open call
        if calls and calls[-1]["name"] == fc.name and not calls[-1]["closed"]:
            calls[-1]["args"].update(dict(fc.args))
        else:
            calls.append({"name": fc.name, "args": dict(fc.args), "closed": False})
# Close on stream end
for c in calls: c["closed"] = True

Two Gemini specifics matter. First, args arrives as a structured object, not a JSON string — you do not parse it, you consume it. That is nicer than OpenAI's shape but breaks any code that assumes "streaming means concatenating strings." Second, parallel calls in Gemini appear as multiple functionCall parts in the same chunk's parts array or across chunks; the accumulator has to walk the parts list per chunk, not assume one call per chunk.

Anthropic: content blocks, parallel-call interleaving.

Anthropic's streaming is event-typed: message_start, content_block_start, content_block_delta, content_block_stop, message_stop. A tool call arrives as a content_block_start with a tool_use block (carrying id and name), followed by a series of content_block_delta events with input_json_delta partial JSON strings, followed by a content_block_stop. Multiple parallel tool calls appear as multiple content blocks with different indices, and Anthropic can interleave deltas from different blocks in the same stream — the accumulator keys on the block index and does not assume a call is complete until its content_block_stop arrives.

# Anthropic — event-typed stream, index-keyed accumulator
blocks = {}
for event in stream:
    if event.type == "content_block_start" and event.content_block.type == "tool_use":
        blocks[event.index] = {"id": event.content_block.id,
                               "name": event.content_block.name,
                               "args": "", "done": False}
    elif event.type == "content_block_delta" and event.index in blocks:
        if getattr(event.delta, "type", None) == "input_json_delta":
            blocks[event.index]["args"] += event.delta.partial_json
    elif event.type == "content_block_stop" and event.index in blocks:
        blocks[event.index]["done"] = True
        blocks[event.index]["args"] = json.loads(blocks[event.index]["args"] or "{}")

The parallel-call trap is real. Two tool_use blocks running in parallel can interleave: the stream can carry a delta for block 1, then block 2, then block 1 again, and the accumulator that keys on "the last block" (rather than the block index in the event) will misroute the fragment. This is documented in the Anthropic SDK but not always mirrored in downstream framework glue; the tool-error-recovery essay treats the recovery patterns that clean up when the accumulator does misroute, but the cheaper fix is the index-keyed accumulator above.

One more Anthropic-specific note: the initial content_block_start for a tool_use includes a input: {} field that is usually empty — the real args come in the deltas. Some accumulators mistake the empty input for the final args and miss the deltas entirely, ending up with a tool call whose arguments are the empty object. The fix is to always concatenate deltas into a fresh string, ignoring the empty initial input.

A portable accumulator shape.

Three accumulators, one common state machine. Every one of them looks like: (1) on the first event that names a tool call, allocate a per-call state keyed on the call's identity (index, block index, or item id); (2) on every event that carries an argument fragment for a known call, append it; (3) on stream end (or per-call completion event), parse and validate the args, then hand the completed call list to the harness. A portable adapter that maps each vendor's stream into the same "start / append / close" event trio is small — a hundred or so lines per vendor — and lets the rest of your harness be vendor-agnostic.

Two things are worth wiring in from the start. First, per-call idempotency: track which call ids you have already dispatched, and drop repeats. This defeats the OpenAI GPT-4.1-nano duplicate-call bug and any equivalent that shows up on the other vendors as their models evolve. Second, malformed-JSON tolerance: if the args string does not parse at close time, do not silently ship a broken tool call — surface the malformed args to the harness and let it emit a corrective tool result the model can retry against. The error-messages-as-prompts discipline treats what that corrective message should look like.

Read the five steps together and streaming tool calls stop being a per-vendor puzzle and start being an adapter-layer exercise. The shape is the same everywhere: identity-keyed accumulator, fragment append, complete-and-parse close. The bumps are all in the vendor-specific field names, the chunk granularity, and the specific bugs that show up on specific models. Get the adapter right once, keep the per-vendor bug notes in a comment next to the branches, and the streaming loop stops being where your harness breaks in production — it becomes the boring piece under all the interesting reasoning.