Local-First Retrieval

First decide whether you need an index at all.

The strongest evidence against reflexive indexing comes from the coding agents, because they are the highest-volume retrieval systems in production. Anthropic's Claude Code shipped an early version with a local vector database and removed it in favor of live ripgrep calls; Cursor, Codex, and Cline all lean on grep-shaped, scriptable retrieval for code. The reasons are structural rather than incidental:

• Precision. When you are looking for PaymentRetryPolicy, exact match is not merely adequate, it is correct. Semantic similarity introduces confident near-misses.
• Freshness. An index over files that are being edited is stale between the write and the re-index. Grep reads the file as it is now.
• No build step. Nothing to provision, nothing to maintain, nothing to invalidate when you change embedding models.
• Composability. The agent can chain a search into a narrower search, which is a plan; top-k similarity is a single shot.

But the trade is not one-sided, and a May 2026 arXiv paper from PwC — "Is Grep All You Need? How Agent Harnesses Reshape Agentic Search" — is the most useful measurement to date. Across 116 LongMemEval questions and several harness/model pairs, lexical search beat vector retrieval uniformly when results were injected inline into the context (93.1% versus 83.6% on one Claude Opus configuration, with the narrowest margin at 76.7% versus 75.0%). Yet when the same results were written to files for the agent to read rather than spliced into the prompt, the ordering reversed on half the configurations. The lesson is not "grep wins"; it is that retrieval quality is not a property of the retriever alone — the harness and the delivery mechanism move the result as much as the algorithm does.

The ingest half decides your ceiling.

Retrieval quality is capped by what got indexed, and on a local build there is no vendor quietly running a tuned layout parser for you. This is where most local knowledge bases lose most of their quality, invisibly:

• Parse with layout awareness, not text extraction. A naive PDF-to-text pass turns a two-column paper into interleaved nonsense and a table into a word salad that will never match any query. Structure-preserving parsers, and for hard documents a vision model that reads the page as an image, are the difference between a corpus that is searchable and one that merely exists.
• Chunk on structure, not on character count. Split at headings, sections, function boundaries, or turn boundaries. A fixed 512-token window cutting mid-sentence produces chunks whose embedding represents nothing in particular.
• Carry metadata from the start. Source path, title, heading trail, timestamp, author, tenant, permission scope. Retrofitting metadata means re-ingesting; and filtering by metadata is usually a bigger quality win than tuning the vector index.
• Prepend context to each chunk. Storing the document title and heading path at the top of the chunk text before embedding is a cheap, large improvement — an isolated paragraph is often uninterpretable without knowing what it is a paragraph of.
• Keep a pointer to the original. Retrieve a chunk, cite a file and a line range. Answers without a path back to the source are unverifiable, which defeats the point of grounding.

The document parsing and ingestion quality entry covers this half in depth. On a local build, budget more time for it than for anything else on this page.

Pick the embedding model, then size the machine.

This is the one component where local is not a compromise. Open-weight embedding models lead the public leaderboards outright — Qwen3-Embedding tops multilingual MTEB above the major hosted endpoints, and BGE-M3 is the MIT-licensed multilingual workhorse that produces dense, sparse, and multi-vector representations in a single pass, which gives you hybrid retrieval from one model. At the small end, EmbeddingGemma and nomic-embed-text run in well under a gigabyte, and all-MiniLM is a 46 MB fallback for hardware that has nothing to spare.

Three numbers size the build:

• Model memory. At 4-bit quantization, roughly 0.6 GB per billion parameters — so an 8B embedding model is about a 5 GB resident footprint, and a 0.6B one fits anywhere.
• Index memory. Raw float32 vectors cost dimensions × 4 bytes each. A million chunks at 1024 dimensions is about 4 GB before any index overhead. This is the number that decides whether your store can be memory-resident; scalar or binary quantization cuts it by 4× or 32× at some recall cost.
• Ingest time. Embedding is embarrassingly batchable and GPU-friendly; a first full index of a large corpus is an hours-long batch job, and every subsequent incremental update is seconds. Plan the first run, not the steady state.

Pick the store by architecture, not by benchmark.

On a single machine, the interesting distinction is not which product is fastest but what shape it is, because the shape decides your operational story. Four shapes cover the field:

• A library, no storage. FAISS: index structures (Flat, IVF, HNSW, PQ) and GPU search, with no persistence, no metadata, no collections. You write the serialization and the metadata join yourself. Right when search is the whole problem and you already have somewhere to put the data.
• A SQL extension. sqlite-vec: pure C, no dependencies, runs anywhere SQLite runs, including WASM in a browser. Vectors become vec0 virtual tables with typed metadata columns, partition keys for multi-tenancy, and auxiliary columns for payloads. Default search is brute-force KNN, which is exactly right up to the low hundreds of thousands of vectors and a poor fit far beyond it. Still pre-1.0.
• An embedded engine. Chroma: a Rust core since 1.0, log-structured — writes append to a WAL, background compaction materializes HNSW segments, reads query the segments. One import, a persistent directory, no server. The default in many framework tutorials, and the fastest path from nothing to working retrieval.
• A columnar format with search on top. LanceDB: data lives in the Lance columnar format on disk (or on object storage), queried in-process with IVF-PQ over memory-mapped files, plus full-text search, versioning, and zero-copy access from Arrow, Pandas, Polars, and DuckDB. Designed for corpora larger than RAM and for multimodal data.

The selection rule that actually holds: choose by where the data has to live and how much of it there is, not by a QPS chart. If your app already ships a SQLite file, put the vectors in it. If the corpus outgrows memory, you want disk-native columnar. If you are prototyping, take the shortest import. And if the answer is "we already run Postgres," the honest recommendation is pgvector and a link to choosing a vector database — a local knowledge base does not have to mean a new dependency.

Two retrievers and a reranker, not one vector search.

The most common local build is dense-only top-k, and it is the most common reason a local knowledge base underperforms a hosted one. Keyword and vector search fail on disjoint queries — BM25 misses paraphrase, dense retrieval misses exact identifiers, rare tokens, product codes, and error strings. Running both and fusing the rankings costs one extra index and recovers a large fraction of the gap.

The production shape on a single machine:

• Retrieve wide from two sources. BM25 (SQLite FTS5, Tantivy, or whatever your store ships) and dense vectors, roughly 50 candidates each.
• Fuse. Reciprocal rank fusion needs no score calibration between the two systems and is a dozen lines of code.
• Rerank. A local cross-encoder scores the fused candidates against the query properly, rather than by cosine proximity. This is the single highest-leverage quality upgrade available, and it is small enough to run on CPU.
• Cut hard. Pass the top 5–10 to the model. Context is not free, and precision at the top of the list is what changes the answer.

Then measure it, because none of the above is worth doing blind. Twenty to fifty question-and-expected-document pairs, drawn from real questions, scored as recall@k, is enough to tell you whether a change helped. Without it you will tune chunk size by vibes for a week. See hybrid search and reranking and evaluating RAG.

Handing it to the agent.

A knowledge base becomes agent infrastructure at the moment it is exposed as a tool, and the interface design matters as much as the retrieval quality:

• Expose search, not the database. A tool called search_docs(query, filters, k) is a contract you can evaluate and rate-limit. A tool that executes arbitrary SQL against the index is a much larger blast radius for very little extra capability.
• Return citations, always. Every result carries source path and location. This is what makes an answer checkable and what lets the agent read the full document when the snippet is not enough.
• Let the agent search more than once. Iterative retrieval — search, read, refine, search again — beats a single top-k shot on hard questions, which is the whole thesis of agentic retrieval. Give it a budget and a stopping rule so it does not loop.
• Consider file delivery over inline injection. The PwC result in Step 1 suggests writing results to a scratch file the agent reads can change which retriever wins. If you support both, it is a cheap thing to A/B.
• MCP is the portable wrapper. An MCP server puts the same knowledge base behind Claude Code, an IDE, and your own harness without three integrations. Chroma and Qdrant ship official servers; a thin custom server over your own search function is a small amount of code and keeps the tool surface exactly as narrow as you designed it.

Keeping it alive.

The unglamorous half. A knowledge base nobody maintains becomes a confident source of last year's answers, and the failure is silent because retrieval always returns something:

• Incremental updates keyed by content hash. Re-embed a document only when its hash changes. On local disk a file watcher makes updates near-instant, which is a genuine structural advantage over upload-based platforms.
• Deletion must actually delete. Renames and moves orphan chunks; deletes that only remove the row leave the vector in the index. Reconcile the index against the source tree on a schedule, or the corpus quietly accumulates ghosts.
• Watch for index drift. Store the ingest timestamp and the embedding model version per chunk. Any query path that mixes vectors from two models is broken, and you want that to be a detectable condition rather than a mystery.
• Track the questions that failed. Log queries that returned nothing useful, and read them weekly. This is the cheapest possible eval-set generator and the fastest route to knowing whether your problem is parsing, chunking, or ranking.

When to build none of this.

Everything above assumes you want a retrieval layer you control from parse to rank. Often you do not, and the self-hosted platform market is mature enough that assembling the pipeline yourself is the wrong default for a team that needs a working internal knowledge base this quarter rather than a tuned one next quarter. Broadly three tiers:

• Turnkey platforms with a UI and connectors. RAGFlow leads on deep document understanding for messy formats; AnythingLLM is the easiest MIT-licensed self-host with document management and team features; Onyx targets the enterprise case with connectors and permission-aware search; kotaemon and Khoj sit closer to the personal end. All run fully offline against local models.
• Frameworks you assemble. LlamaIndex and Haystack give you the pipeline components without the product, which is the right layer when your ingestion or ranking is genuinely unusual.
• Graph-shaped knowledge bases. LightRAG and Cognee build a knowledge graph alongside the vector index and run locally — worth it only if your unanswered questions are relational or whole-corpus, since the extraction pass costs an LLM call per chunk.

The decision is the same one you make about any infrastructure: build the layer where your requirements are genuinely specific, and take the finished product everywhere else. For most teams that means a platform for the general document corpus, and a small hand-built retriever — or plain grep — for the one corpus with unusual structure that the platform handles badly.

Related: local knowledge bases for the concept-level version, small and local models for the model-sizing question, and LanceDB vs Chroma vs sqlite-vec vs FAISS for the store comparison in detail.