Local Knowledge Bases

"Local" is three dials, not one switch.

People say "local RAG" and mean wildly different architectures. There are three places data can leave your machine, and each is a separate choice:

• Where the documents sit. On your disk, on your NAS, in your VPC — versus uploaded to a vendor's ingestion API. This is the dial most compliance requirements actually care about.
• Where the embeddings are computed. A local embedding model versus a hosted embeddings endpoint. Note that calling a hosted embedder means every chunk of every document is sent to a third party at index time — a "local vector database" full of vectors computed by someone else's API has already leaked the corpus.
• Where generation happens. A local small model versus a frontier API. Most production "local KB" setups are local on the first two dials and remote on the third, because the retrieved passages are a small, reviewable fraction of the corpus while the quality gap in generation is large.

The five stages, and what changes when you own them.

Every knowledge base — local or hosted — is the same pipeline. Owning it locally changes the failure mode at each stage, not the stage itself:

• Ingest and parse. Turn PDFs, wikis, tickets, and code into text. Locally this is where most quality is lost: a hosted platform ships a tuned layout parser, and your first pass with a naive PDF extractor will silently mangle tables into word salad.
• Chunk. Split into retrievable units. See chunking and vector search — the rules do not change locally, but nobody is tuning them for you.
• Embed. Convert chunks to vectors with a model you now pin, host, and eventually upgrade.
• Index. Store vectors plus metadata in something queryable. On one machine this is often a file, not a server — an embedded store like LanceDB, Chroma, or a SQLite extension, rather than a cluster.
• Retrieve. Given a question, return the passages worth putting in the context. Hybrid keyword-plus-vector retrieval with a reranker is the production default, and it is the stage where local setups most often under-invest.

One consequence deserves its own sentence, because it surprises people at exactly the wrong moment: changing the embedding model invalidates the entire index. Vectors from two different models are not comparable, so an upgrade means re-embedding every chunk you have ever stored. Budget for it as a recurring cost, not a one-off migration.

What local actually buys, and what it actually costs.

The case for local is stronger than the privacy slogan suggests, and the costs are more specific than "you have to run it yourself."

• Buys: data residency you can prove. Not a vendor's assurance about their sub-processors — a network boundary. For regulated corpora and for source code, this is frequently the whole argument.
• Buys: no per-query retrieval bill. Retrieval cost collapses to electricity. This matters more in agent loops, where one task can trigger a dozen searches, than in a chatbot that searches once.
• Buys: freshness without an upload round-trip. Re-indexing a changed file on local disk is milliseconds. It is also the only architecture that works offline or on an intermittently connected machine.
• Costs: you own recall. When the agent answers "I could not find it" and the document exists, nobody else is going to diagnose the chunking. You need an eval set — even twenty question-and-expected-document pairs — or you are flying blind.
• Costs: index maintenance is real work. Deletions, renames, stale entries, corrupted files, and the re-embed on model upgrade. A knowledge base that nobody maintains degrades into a confident source of last year's answers.
• Costs: hardware and a scale ceiling. An embedded store on a laptop is genuinely fine into the millions of vectors. It is not fine for a hundred concurrent users, and the migration from "a file" to "a service" is a rewrite of your retrieval layer.

Ask whether you need an index at all.

The most useful thing to know about local knowledge bases in 2026 is that the leading coding agents largely stopped building them. Claude Code, Cursor, and Codex retrieve over source trees with glob, grep, and read rather than a local vector index — exact matching, no index to build, nothing to keep in sync with files being edited, and no embedding step at all. When the corpus is on disk, is text, and the agent can afford several tool calls, letting the model search directly is often better and simpler than the pipeline above.

A rough dividing line:

• Skip the index when the corpus is local, structured, and small enough to navigate — code, config, a docs folder — and when exact identifiers matter more than paraphrase.
• Build the index when the corpus is large, prose-heavy, and queried by meaning rather than by string: support histories, research papers, policy documents, contracts, anything where the user's words will never match the document's words.
• Do both for most real systems — keyword search and vector search fail on different queries, which is exactly why hybrid retrieval is the default.

The local-first retrieval deep-dive walks the build end to end, including model choice and the hardware budget; choosing a vector database covers the store selection, and agent memory covers the adjacent question of what an agent should retain about itself rather than about your documents.