Embeddings: meaning as geometry.
An embedding is not a measurement of similarity; it is a trained opinion about what similarity means, and you freeze that opinion into every vector you store. The practical consequence surprises people: a question and its own answer can land far apart under a model that was taught to compare sentences, and close together under one taught to match queries to documents — which is why the major embedding APIs make you declare, on every call, whether the text you are sending is a query or a document. Get that one string wrong and nothing errors; your search just quietly gets worse.
Step 1The core move: meaning becomes coordinates.
A computer cannot compare the meaning of "dog" and "puppy" — they are just different character strings, no more related than "dog" and "xylophone." An embedding fixes this by mapping each item to a list of numbers — a vector — chosen so that things with similar meaning land at nearby points and unrelated things land far apart. "Meaning" becomes a location in a high-dimensional space; "similarity" becomes geometric distance, usually cosine similarity, the angle between two vectors.
The number list is not designed by hand. A neural network learns it from data, under pressure to make useful predictions, and the by-product is a space where geometry reflects meaning. A 2-D sketch makes the idea visible even though real embeddings have hundreds or thousands of dimensions:
cat · · kitten
· dog
· puppy
· car
· truck · van
near = similar meaning · far = unrelated · clusters = topics
Words about pets cluster together; words about vehicles form a separate cluster far away. Nobody placed them — training shaped the geometry. That one conversion, from the fuzzy human notion of "similar in meaning" to the precise machine notion of "near in space," is what makes semantic search, recommendation, deduplication and retrieval-augmented generation possible at scale, and it works the same way for images, audio and code.
Step 2"Similar" is a training decision, and that is why your API call has a role field.
Here is the part the geometry picture hides. Similar in what sense? A model trained to score sentence pairs learns that two texts are close when they say roughly the same thing in roughly the same way — same register, comparable length, overlapping surface form. A model trained for retrieval learns something different and deliberately lopsided: that a short interrogative fragment should land near a long declarative passage that answers it. Those are not two qualities of the same space; they are two different spaces, and the second is called asymmetric because the two sides of the comparison are not interchangeable.
Work one case through, because the failure is easy to mistake for a bad model. A user types how do I reset my password. Your index holds two candidates: a navigation stub titled "How to reset your password" that contains nothing but a link, and a 400-word article titled "Account recovery steps" that contains the actual procedure. Under a symmetric sentence-similarity space the stub wins easily — it is a near-paraphrase of the query, nine words long, same interrogative framing. Under an asymmetric retrieval space the article wins, because the model was trained specifically on question-to-passage pairs where the passage looks nothing like the question. Both systems returned their top result correctly. Only one of them answered the user, and the difference was a parameter, not a model tier.
Which is why retrieval-grade embedding APIs will not let you stay silent about it. As of November 2026 each major provider has its own spelling of the same declaration:
# the same sentence, embedded for two different jobs # Cohere embed-v4.0 co.embed(texts=[q], model="embed-v4.0", input_type="search_query") co.embed(texts=[d], model="embed-v4.0", input_type="search_document") # Gemini embedding (gemini-embedding-001 and earlier) embed_content(content=q, task_type="RETRIEVAL_QUERY") embed_content(content=d, task_type="RETRIEVAL_DOCUMENT") # open weights: the declaration is a literal text prefix embed("search_query: how do I reset my password") # nomic-embed-text embed("search_document: Account recovery steps ...") # and the failure mode: no exception, no warning, no log line. # a symmetric call on an asymmetric model returns a perfectly # well-formed vector that is pointing at the wrong neighbourhood.
Three things about that block are worth more than the syntax. First, the spelling keeps moving: Google's newer gemini-embedding-2 drops the task_type parameter and expects the task to be stated as a prefix in the prompt instead — so an SDK wrapper that hard-codes task_type is not forward-compatible with the next model from the same vendor. Second, the default is a trap rather than a neutral choice; practitioners report that an omitted task_type on Vertex has historically been treated as the query side, which is precisely the wrong side for the ten million documents you are indexing. Third, OpenAI's text-embedding-3-* family has no role field at all, which is not a gap — it is a different design, a symmetric space in which you are expected not to make the distinction. That is a real difference between products, not a missing feature, and it is invisible in every benchmark table that reports one score per model.
This is the cheapest retrieval bug there is to create and one of the most expensive to find, because every intermediate artefact looks healthy. The vector has the right dimension. The index builds. The nearest-neighbour search returns k results with plausible scores. Only end-to-end recall on real queries moves, and it moves by a few points — the size of a dozen other things. Assert the role string in a test; do not trust it to code review.
Step 3Directions carry meaning — the result that made this click.
In a well-trained embedding space, directions encode relationships, not just positions. The classic illustration:
vector("king") - vector("man") + vector("woman") ≈ vector("queen")
Read it as: start at king, remove the "maleness" direction, add the "femaleness" direction, and you arrive near queen. A consistent geometric offset corresponds to a real-world relationship, and similar offsets appear for capital-of-country or singular-to-plural. This emerged from training on plain text with no dictionary of relationships — good evidence that the geometry captures structure rather than mere topical proximity.
Treat it as intuition, not a guarantee. Real spaces are messier than the clean analogy, the arithmetic does not always land, and learned vectors inherit the biases of the training text. An embedding reflects its data, prejudices included — which matters directly when it ranks job applicants, support articles or product recommendations.
Step 4The stored vector is a schema commitment, so write down what produced it.
Because the opinion is baked in at index time, every vector in your database is a commitment to one model, at one version, at one width, under one role convention. None of those four can change unilaterally. Mixing vectors from two models in one index does not degrade gracefully — the two sets of coordinates are not comparable at all, and the symptom is a search that works for recently added content and fails for old content. So store the provenance beside the vectors, in the row, not in a wiki page:
- Model identity and version —
embed-v4.0, not "Cohere". A version bump is a new space. - Dimension and numeric precision — the width you asked for and the type you stored. Both are dials on modern models, and what they cost you in recall is the subject of vector compression and the memory tier.
- The role convention — which input type or prefix the document side was embedded with, so the query side can be made to match three years from now by someone who was not there.
- The chunking rule — the vector describes a span of text somebody chose; see chunking and vector search.
Treat a model change as a migration with a backfill, a dual-read window and a cutover, which is exactly how reindexing and embedding migrations describes it. The cost is not the API bill for re-embedding; it is that you cannot serve two spaces at once, so the index has to be rebuilt beside the live one.
Two limits keep expectations calibrated. Similarity is not relevance. Nearest-neighbour search finds text that is about the query; whether it answers the query is a separate judgement, which is the entire reason a second-stage reranker reads the candidate text properly and reorders it — see hybrid search and reranking. And embeddings compress. Collapsing a passage into a few hundred numbers loses nuance by construction, so two genuinely different sentences can land suspiciously close, and no amount of index tuning recovers a distinction the encoder threw away.
Finally, the connection back to language models: embeddings are not a gadget bolted onto an LLM, they are its native input. Text becomes tokens, tokens become integers, and the model's first layer turns each integer into a learned vector. Everything after that is arithmetic on those vectors. "Meaning as position in space" is not an analogy for how an LLM works — it is literally the first thing it does.
Before you compare models, do the two things that cost nothing. Build a fixed set of 100–200 real queries with known-good answers, and measure recall@10 on your own corpus — public leaderboards rank the average of other people's data, and your data is not the average. Then assert the role convention in a test: embed one query and one document, check the request bodies carry the query-side and document-side declarations respectively, and fail the build if they match. That second test takes ten minutes and catches the defect that is otherwise indistinguishable from "semantic search just isn't that good." For the product comparison itself, see OpenAI vs Cohere vs Voyage vs Qwen3 embeddings.