Small & local models.
The wrong question is whether a model you can run on your own machine matches a frontier model; it does not, and it will not. The right question is which jobs in your system never needed a frontier model — and the answer is more of them than you would guess, because embedding, reranking, routing, classification, and extraction are the high-volume steps, and they are exactly where a 0.5–8B model on your own hardware is both good enough and orders of magnitude cheaper.
What "small" means in bytes, not adjectives.
Parameter count is the headline; the number that decides whether it runs on your machine is the quantized file size. Weights are usually trained at 16 bits and then compressed for local serving — 8-bit, 4-bit, or lower — trading a little quality for a lot of memory.
- The common default, 4-bit (labels like
Q4_K_M), lands around 0.6 GB per billion parameters. An 8B model is roughly a 5 GB file; a 30B model is roughly 18 GB. - You need that much memory plus the KV cache, which grows with context length and with concurrency. Long contexts are frequently what pushes a model off a GPU, not the weights.
- Unified-memory machines (Apple silicon) blur the line usefully — system RAM is the budget, so a 32 GB laptop runs models a 12 GB discrete GPU cannot.
- Below roughly 4-bit, quality degradation stops being subtle. A heavily-quantized large model is not automatically better than a lightly-quantized smaller one; test on your task rather than assuming.
Two models of the same parameter count can differ enormously — a 2026-vintage 8B model outperforms a 2024-vintage 30B on most benchmarks. "Small" is a memory statement, not a capability statement, and the capability floor rises every few months.
The jobs small local models are genuinely good at.
Sort your pipeline by call volume and you will usually find the top entries are narrow, repetitive, and well-specified — which is the profile a small model handles well:
- Embedding. The clearest win, because the open models are not merely adequate here — they lead. Qwen3-Embedding tops the multilingual MTEB leaderboard above the major hosted endpoints, BGE-M3 is the MIT-licensed multilingual workhorse that emits dense, sparse, and multi-vector representations from one pass, and EmbeddingGemma and
nomic-embed-textrun comfortably in well under a gigabyte. For a local knowledge base, this is the component you should almost never pay an API for. - Reranking. A cross-encoder that scores fifty candidate passages against the query, called on every search. Small, local, and the single highest-leverage quality upgrade in most retrieval stacks.
- Routing and classification. Which tool, which index, which language, is this in scope, is this a refusal — hundreds of tokens in, a label out.
- Extraction and normalization. Pulling fields out of a document into a schema, deduplicating entities, cleaning text before it is indexed.
- Redaction before egress. A local model that strips identifiers before anything is sent to a hosted API is a privacy control that costs one small model and no vendor negotiation.
Where they still lose, and it is not close.
Being honest about the ceiling is what makes the hybrid design defensible:
- Long-horizon agentic work. Twenty-step tool-use loops are where small models fall apart — not by refusing, but by drifting, repeating a failed call, or losing the goal. Error compounds across steps, so a small per-step gap becomes a large per-task gap.
- Hard reasoning and long context. A stated 128K context window is not the same as usable recall across 128K tokens, and small models degrade earlier and more sharply.
- Breadth of world knowledge. Fewer parameters means less memorized fact. This matters less than it sounds when you are grounding answers in retrieved documents — which is one more reason a good local knowledge base is what makes a small local model viable.
- Robustness under adversarial input. Instruction-following holds up worse against prompt injection. Do not put a small model in the position of being the only thing standing between untrusted text and a dangerous tool.
The costs that do not show up in the price comparison.
"Free after hardware" is true per token and misleading per system. What you take on:
- A throughput ceiling instead of a bill. One machine serves a fixed number of concurrent requests, and the failure mode under load is a queue, not a higher invoice. Batch what you can; embedding is embarrassingly batchable, interactive generation is not.
- Serving is a system. Ollama and llama.cpp are the easy on-ramp for one user; vLLM or SGLang are what you reach for when many users share a GPU, and MLX is the Apple-silicon path. These are different operational commitments — see serving and access.
- Upgrade churn. Open-weight models improve fast, and switching is not free: prompts drift, and for embedding models a swap invalidates your entire index.
- Evaluation becomes mandatory. With a hosted frontier model you can coast on its general competence. Downsizing is only safe if you can measure that quality held — so a small eval set is the actual prerequisite for the cost saving.
The pragmatic default for most teams: local models for embedding, reranking, and classification; a hosted frontier model for the agent's reasoning loop. That single split captures most of the privacy benefit and most of the cost saving, because the local jobs are the high-volume ones and the hosted job is the one where the quality gap actually bites. Escalate to fully local only when a hard requirement — air-gapped operation, regulated data, offline use — makes you.
Related: choosing a model for the per-task checklist, open-weight vs closed models for the licensing dimension, and local-first retrieval for the hardware budget worked through end to end.