Modalities: text, vision, audio, code, and "multimodal."
"Multimodal" tells you what a model will accept, which is the least decision-relevant thing about it. Every non-text input gets converted into tokens somewhere, and the only question that changes your architecture is where — because that one boundary fixes both what information is destroyed on the way in and what you pay for it on every step of an agent loop. Choose the conversion point first; the spec-sheet checkbox follows from it.
Step 1Modality is a pair, and the pair is lopsided.
A modality is a kind of data — text, images, audio, video. Input and output modalities are separate claims, and almost every model described as multimodal is multimodal in and text out. That asymmetry is the single most common misreading of a model card: "supports audio" usually means it can listen, not that it can speak.
- Text in, text out. The baseline every large language model shares.
- Vision in. Images alongside text — screenshots, charts, scanned documents, photos — reasoned about and answered in text. Standard across frontier families as of November 2026.
- Audio in and out. The genuinely newer capability, and the one where "multimodal" earns the word: as of November 2026 OpenAI's Realtime API and Google's Gemini Live API both take speech in and emit speech out without a text round-trip in the middle.
- Video in. Frames, sometimes with audio. The most uneven support and by far the most tokens per second of source material.
- Images and audio out as artefacts. Generating pixels or waveforms as a deliverable is usually a separate specialised model you orchestrate, not a mode of your reasoning model.
- Code. Not a sensory modality at all, but worth tracking as its own capability axis because it has its own evaluation regime — see reading benchmarks.
Read a model card as two lists, never one. If your product needs the agent to say something aloud, the vision column is irrelevant and most of the "multimodal" shortlist does not qualify.
Step 2Someone converts every modality to tokens. The only choice is who.
Models consume tokens. Audio and images are no exception — they are tokenised, just not by a tokeniser you can read. So for each non-text modality you are picking one of two conversion points, and the industry's two architectures are exactly those two choices:
- You convert, early (the pipeline). Speech-to-text transcribes, a text model reasons, text-to-speech speaks. You own the intermediate representation: it is inspectable, loggable, redactable, and cheap, because text is the densest encoding of content there is. You also destroyed everything the words did not carry.
- The provider converts, inside (native). One model shares a representation space across modalities. Tone, hesitation, overlapping speech, the position of a number on a page — all of it survives into the reasoning. In exchange the intermediate representation is gone: you cannot read it, filter it, or store it as your record.
This is why the cascade-versus-native argument is never really about audio quality. It is about whether your guardrails, redaction and audit trail run on the model's actual input or on a reconstruction of it. The voice section works that through in detail at STT, TTS & speech-to-speech.
Step 3Each modality is metered in a different unit — and one of them is wall-clock.
Text bills per token of content: say less and pay less. The other modalities do not work that way, and this is the part that surprises teams after launch.
- Images bill per area. Token count scales with resolution through a tiling rule — Anthropic's computer-use tool documents it as
⌈width / 28⌉ × ⌈height / 28⌉visual tokens — so a screenshot's price is set by the frame you chose, not by how much of it mattered. A 1024×768 desktop capture and a full-frame 2576×1448 one differ by about 4.6× on every step. - Audio bills per second of elapsed time. Native speech models are priced per minute of audio in and out, at a rate equivalent to tens of tokens per second. The consequence is blunt: three seconds of silence costs the same as three seconds of speech. A caller thinking, a hold tone, a pause you inserted for naturalness — all billed, none informative.
- Video bills per second times per frame. It is the image rule multiplied by a frame rate, which is why video input is the one modality where a naive implementation can be an order of magnitude off budget.
Then the loop multiplies whichever unit you picked. An agent re-sends its transcript on every step, so a screenshot taken at step 3 is paid for again at steps 4 through N — the arithmetic, and the prompt-caching fix that flattens it, are in perception cost per step and agent cost control.
A per-minute audio price looks small next to a per-million-token price and is not comparable to it. Convert both to cost per completed task before you choose, or you will pick the modality that reads cheapest on the pricing page and bills most on the invoice.
Step 4Decide by what must survive the boundary.
The conversion point is a lossy compression step you get to specify. So name the one thing your task cannot afford to lose, and let that pick the architecture:
- An angry support caller. The signal is prosody, and transcription deletes it. "Fine." and "Fine!" transcribe identically. If escalation depends on hearing how it was said, native audio is doing work no pipeline reproduces.
- A scanned invoice. The signal is layout — which number sits in the total column. Flattening the page to text first destroys the geometry that answers the question, which is why vision-in beats OCR-then-text here, and why document parsing is its own discipline: document parsing for RAG.
- A compliance-recorded phone order. The signal is the record itself. Here the pipeline's inspectable transcript is the requirement, and native audio turns your evidence into a second ASR pass with its own errors.
- A GUI agent clicking. The signal is pixel position, and nothing survives a text summary — but cost is set by frame size, so buy detail with a zoom on the region you are about to click rather than a bigger screenshot. See GUI grounding and computer use.
Note that capability is uneven within a model: strong vision rarely implies strong audio, and benchmarks are published per modality. A single "multimodal" badge is an average over things you will not use equally.
Write down two columns — what goes in, what must come out — and eliminate every model that physically cannot do both. Then ask the one question the spec sheet never answers: what gets destroyed at the conversion point, and can I afford to lose it? Default to a pipeline, because text intermediates are cheap, inspectable and the substrate your guardrails already run on, and reach for native multimodality only where you can name the signal that transcription kills. Price the result per completed task, not per token, because the modalities are metered in units that do not compare.
Related: choosing a model for the rest of the shortlist, cost, quality & latency for the three-way trade-off, full-duplex speech for where audio-in-and-out is heading, and context windows for the ceiling every modality shares.