Vector compression and the memory tier

Start with the arithmetic, because it is the only part nobody disputes.

A vector index has two memory consumers: the vectors themselves and the graph that navigates them. The vectors dominate, and their size is exactly dimensions × bytes-per-component.

# bytes per vector, 1536 dimensions

float32    1536 x 4 = 6144 B      10M vectors -> ~61 GB
float16    1536 x 2 = 3072 B      10M vectors -> ~31 GB
int8       1536 x 1 = 1536 B      10M vectors -> ~15 GB
1-bit      1536 / 8 =  192 B      10M vectors -> ~1.9 GB

# plus the HNSW graph: roughly 2 x m x 4 B per node at the
# base layer, so ~128 B/vector at m=16 — small next to
# float32, and the DOMINANT term once you go 1-bit.

Two things fall out of that table immediately. First, the interesting threshold is not a compression ratio, it is whether the working set crosses from "fits in the RAM you are willing to pay for" to "does not" — and at 10M vectors of a modern 1536-dimension embedding, float32 does not fit on a commodity box while 1-bit fits with room to spare. Second, once you are at 1 bit the graph links cost more than the vectors, so further compression of the vectors buys nothing. That is the real ceiling, and it arrives sooner than people expect.

Which numbers apply to you depends on the embedding you chose, and dimension counts vary by an order of magnitude across models. Do this calculation before reading any vendor benchmark — most of the disagreement between published results is a dimension difference wearing a technique's clothes.

Binary quantization does not work without rescoring. That is the whole article.

One bit per dimension keeps the sign and throws away the magnitude, and the distance it computes is a Hamming distance over bit strings — which is why the speed claims are real, since that is a popcount instead of a dot product. What it also throws away is enough information to rank the top-k correctly.

Qdrant's own published measurements put 1-bit search without rescoring at 0.605 recall@10 against exact search. That is not a tuning problem; it is the technique. Turn rescoring on — retrieve more candidates by the cheap binary distance, then re-rank those candidates using the original vectors — and the same configuration reports 0.977 recall@10 at 2× oversampling and 0.988 at 4×, which is within a point of the float32 baseline at 1/32 the vector size. On 1536-dimension OpenAI embeddings the reported figure is about 0.98 recall@100 at 4× oversampling; on a 4096-dimension Cohere model, about 0.98 recall@50 at 2×.

# what rescoring actually does, per query, for k=10

1. search the 1-bit index for k * oversampling candidates
2. read the ORIGINAL vectors for those candidates
3. compute exact distances, sort, return k

# step 2 is the part the "32x smaller" headline omits.
# at 4x oversampling, k=10: 40 full-precision vector reads.

So the architecture has three copies to place, not one: the 1-bit vectors (hot, random access, every query), the graph (hot), and the full-precision vectors (read for oversampling × k candidates per query). Vendors default oversampling low and rescoring on for good reason, and Qdrant's documentation states the rule plainly: use binary quantization only with rescoring enabled.

Where the full vectors live is the decision you are actually making.

Now the tiering question has teeth, because step 2 of the rescore has to read the originals from somewhere, and there are only two somewheres.

• Full vectors in RAM. Then you did not save memory — you saved index memory, and what you bought is speed: the candidate scan is a popcount over 192-byte strings instead of a dot product over 6 KB. This is the configuration the dramatic throughput numbers come from, and it is the right one when your problem is QPS or p99 latency rather than the size of your instance.
• Full vectors on disk. Then the memory saving is real and large, and you have moved the cost to the storage layer: every query now issues oversampling × k random reads. Qdrant's documentation is explicit that rescoring against on-disk originals can significantly decrease search speed. On NVMe with a warm page cache this is often fine; on network storage, or with a working set that does not cache, it is where your tail latency goes to live.

This is why the same technique gets written up as "40× faster" in one post and "32× cheaper" in another. Those are two different deployments and you cannot have both at once: the speed claim assumes the originals are already in memory, and the memory claim assumes they are not. Decide which number you are buying before you turn the feature on, and size oversampling against that choice rather than against a recall table — Qdrant's measurements show oversampling past 4× adds almost nothing to recall while continuing to add reads.

The same framing settles the related question of whether to keep the graph on disk. At 1 bit per dimension the links are the larger term, so a memory-constrained design that quantizes aggressively and then leaves the HNSW graph in RAM is correctly ordered; one that pushes the graph to disk is trading the one structure whose access pattern is pure random hops. Choosing a vector database is in large part choosing whose defaults already encode this.

Dimensionality is the budget both techniques spend.

Matryoshka representation learning trains an embedding so that its leading dimensions are independently useful, which lets you truncate a 1536-dimension vector to 512 or 256 and keep most of the quality. The dimensions parameter on current embedding APIs is this, and it is genuinely free in a way quantization is not — you store fewer numbers and read fewer numbers, with no second copy and no rescore.

What is not obvious is that truncation and binarisation draw on the same reserve. Both work because a high-dimensional embedding is redundant; truncation spends that redundancy, and so does throwing away every component's magnitude. Spend it twice and the second technique has less to work with. The published figures show exactly this: at 512 dimensions, 1-bit accuracy without rescoring sits around 53–55%, recovering to 71–91% with rescoring — far below the near-baseline recall the same method reaches at 1536 and 4096 dimensions. Qdrant's own guidance says as much in careful language: models with lower dimensionality, or a different distribution of vector components, may need their own parameter search.

# ordering that works, cheapest decision first

1. pick dimensions      # MRL truncation; measure recall@k
2. pick precision       # int8 is the low-risk 4x
3. consider 1-bit       # only if dims are high AND you
                        # have decided where originals live
4. tune oversampling    # 2x, then 4x; stop there

# never change two of these between measurements.

Treat the widely-repeated claims that 256-dimension truncation retains 95–98% of full quality, or that truncation plus int8 costs under two points of recall, as hypotheses about someone else's corpus. They come from uncontrolled write-ups, the retained fraction is strongly dataset-dependent, and the whole point of MRL is that it is cheap to test on yours.

Measure recall against exact search, not against last week's index.

Almost every quantization regression ships because the team measured the wrong baseline. The number that matters is recall relative to exhaustive search over the same vectors, because that isolates what the approximation cost you from what your retrieval was never going to find anyway.

• Build the ground truth once, by brute force. Take 500–1000 real queries from your logs, compute exact top-k over the full-precision vectors offline, and store it. This is a few minutes of compute and it is reusable across every configuration you will try.
• Report recall@k at the k you actually consume. Recall@100 flatters quantization badly: the approximate candidate set almost always contains the right documents somewhere in a long tail. If the model sees five chunks, measure recall@5 — and note that this is the same reason a reranker's gain is bounded by your recall curve.
• Measure the end metric too, on a subset. Retrieval recall is a proxy. A 0.98-recall index that drops the one authoritative document for a class of queries can move answer accuracy more than the number suggests, which is what RAG evaluation exists to catch.
• Record p95 latency and read volume in the same run. If the originals are on disk, the recall table is only half the result; the other half is reads per query, and it is the half that regresses under load rather than under test.

One operational trap worth naming: quantization parameters are part of your index's identity. Changing oversampling is a query-time knob, but changing dimensions or precision means a rebuild, and a rebuild over tens of millions of vectors is a migration with a dual-write window — the same shape of problem as an embedding migration, and worth planning as one rather than discovering as one.

When to reach for which, and when to reach for none.

The honest answer for most systems is that they are nowhere near needing this. A million chunks at 768 dimensions in float32 is about 3 GB; that is a laptop, and spending a week on quantization to save it is a week not spent on the retrieval quality problem you actually have.

• Under ~5M vectors, change nothing but dimensions. Use the dimensions parameter if your provider offers it, keep float32, and move on. No second copy, no rescore, no new failure mode.
• 5M–50M vectors, or a memory bill you can see: int8. Scalar quantization is the boring 4× with the smallest recall risk and no hard dependency on rescoring. It is the default most teams should land on and the one least likely to produce an incident.
• Above that, or when QPS is the constraint: 1-bit with rescoring. Only with high-dimensional embeddings, only with oversampling at 2–4×, and only after you have written down where the full-precision vectors live and what a rescore costs there.
• On-device or single-binary deployments: dimensions first, then int8. The rescore's random reads are exactly what a constrained device handles worst, which is part of why local-first retrieval leans on smaller embeddings rather than aggressive quantization.