On-Device Agent Architecture

Three real wins and one that evaporates.

Local inference has become genuinely practical: 30-billion-parameter open-weight models shaped for agent loops now fit on one consumer GPU or a well-specified laptop, quantized to around four bits, and answer fast enough to sit inside a loop rather than beside one. The case for using them is strong and it is worth stating precisely, because the strongest-sounding argument on the list is the weak one.

• Marginal cost goes to zero. This is the win that changes designs rather than budgets. When a step is free, you can afford to run a classifier before every action, re-read the whole context after every tool result, or draft three candidates and pick one — all things you would never do at metered prices. Loops that are uneconomic on an API become ordinary.
• The network leaves the inner loop. A twenty-step task on a hosted model pays connection setup, queueing and time-to-first-token twenty times. Locally the same loop pays memory bandwidth. Even where the hosted model is faster per token, the loop can be slower end to end.
• It works with no connection. Underrated until the product is a phone, a vehicle, a factory floor or a hospital basement.
• It does not make the system private. The model is one component. The tools are the rest, and tools are where data leaves.

That last point is the one this page keeps returning to, because it is the failure mode that a "runs on your device" launch reliably produces in the systems built on top of it.

The loop has five parts, and each one localizes for a different reason.

"On-device agent" is not one architectural choice. It is five, and treating them as a bundle is how teams end up with an agent that is local in the expensive places and remote in the sensitive ones. Take the agent loop apart:

• The model. Local for cost and latency. This is the part everyone means, and it is the least security-relevant of the five.
• The index and memory. Local because that is where the sensitive corpus actually is — mail, notes, files, message history. An on-device index is the strongest privacy claim in this architecture and it does not require a local model, only a local embedding step.
• Tool execution. Mostly remote, because the tools are other people's services. This is the egress path and there is no version of "on-device" that removes it.
• Policy checks. Local is convenient and remote is enforceable. A limit that lives on the user's machine is a limit the user's machine can be persuaded to ignore.
• Session state. Local by default is good for privacy and bad for continuity — no handoff to another device, no server-side inspection when something goes wrong, and no resumability beyond what you write to disk yourself.

Latency is the design driver, and it behaves differently on a device.

A hosted deployment optimises throughput because it serves many users on one GPU. A device serves exactly one user and has no queue, which inverts several familiar trade-offs.

Speculative decoding wins here and loses in the data centre. It spends extra compute to cut per-token latency, so it is close to free on an idle personal GPU and a throughput regression on a busy shared one — the reason a model shipped with speculative decoding enabled is telling you it expects to be run alone. See speculative decoding.

Prefill is the cost you feel. On a device, decode is bounded by memory bandwidth and prefill by compute, and an agent loop re-reads a growing context on every step. Without cache reuse, step twelve pays for everything the first eleven steps produced. Local runtimes vary widely in whether they keep the KV cache warm across calls, and none of them keep it across an application restart, so a resumed session pays full prefill again. Budget for it explicitly — this is the mechanism described in prefill, decode & the KV cache, with the hosted provider's prefix caching removed.

Dense beats sparse when the GPU is yours. A mixture-of-experts model bills compute for the experts that fire and memory for all of them, which is a bargain on rented capacity and an expensive mistake on a 24 GB card. Dense models in the 7–30B range are the natural fit on-device for exactly this reason; see mixture of experts.

The first token after idle is the one users judge. Weights get evicted from memory, the runtime cold-starts, and a "local, therefore instant" agent takes nine seconds to say hello. Keeping a small model resident and paging in the large one on demand is usually a better experience than one mid-sized model that is never warm.

The escalation gate, and why it should be a static rule.

Almost every serious on-device agent is a hybrid. The interesting design question is what sends work off the device, and the tempting answer — let the local model judge whether a task is too hard for it — is the wrong one. Difficulty estimation is itself a hard judgement, it is made by the component you already suspect, and it fails in the direction that costs you most: a model that does not know a policy also does not know that it does not.

Route on a static property of the action instead. The rule that holds up in practice is escalate anything that binds the user:

• Anything irreversible — a purchase, a send, a delete, a booking.
• Anything governed by a written policy the agent is expected to follow while a person argues with it. This is the axis on which small models are weakest, by a wide margin, and no amount of prompting closes it.
• Anything that will be shown to a third party as if the user wrote it.
• Anything where being wrong is expensive and being slow is not.

Everything else — classify, extract, summarise, draft, rank, decide which tool to try — stays local. That is a routing table you can read, test and audit, which a learned difficulty estimator is not. It is the same conclusion model routing & cascades reaches from the cost side: static routing by task captures most of the benefit with none of the fragility.

The trust boundary you actually have.

Draw the real boundary and several assumptions break at once.

Local model, remote tool, same leak. An agent that reads the user's mail locally and then calls a hosted summarisation-free tool — a calendar API, a maps lookup, a payment endpoint — has put user data on the network, in a request whose contents the user never approved individually. Running inference locally removes one egress path and leaves every other one exactly where it was. What controls this is an egress policy, not a deployment target; see egress control for agents and data residency & sovereignty.

The local index becomes the crown jewel. Consolidating mail, files and message history into one embedded vector store creates an asset that did not previously exist in that shape. Any process on the device can read it, and unlike a server-side store there is no access log, no revocation and no incident response. Encrypt at rest with a key tied to the OS keychain, and think about what a device backup contains.

Prompt injection is worse, not better, on-device. The agent reads the user's own untrusted content — email from strangers, downloaded documents, web pages — with the user's own credentials and no server-side proxy in the middle to inspect or refuse. Every mitigation in prompt-injection defense that relies on a chokepoint has to be re-implemented on the client, where the attacker may also be standing.

You cannot revoke. A compromised server-side agent is a deploy away from being stopped. A compromised on-device agent is a software update away, gated on the user installing it. Design the kill switch as a server-side capability check the client consults, not as a client-side flag, and accept that an offline device will not hear it.

Operating a fleet you cannot see.

The architectural work is the easy half. The operational half is that every practice you rely on for a hosted agent assumes a fleet you control, and you now have a fleet you can only ask questions of.

• Version skew is permanent. A hosted agent runs one version. A shipped agent runs every version anyone declined to update, on hardware you did not choose, at quantizations the runtime picked for them. "Which model produced this trace" becomes a field you must record rather than a fact you know.
• Rollback is not a button. The rollout discipline still applies, but the unit is an app release and the timeline is days. Feature-flag the behaviour from the server so you can change what the local model is asked to do without shipping new weights.
• Traces are consent-gated and sparse. You will see a biased sample: the users who opted in, and disproportionately the ones for whom things went badly enough to report. Build the local agent to keep a rich on-device trace the user can export with one action, and design the failure path around that export rather than around telemetry you will not receive.
• Evaluate per hardware class, not once. The same weights at 4-bit on a laptop and at 8-bit on a workstation are different models for evaluation purposes, and quantization degrades instruction-following before it degrades fluency — which is precisely the failure your eval suite is least likely to notice.
• The economics invert. There is no per-token bill and there is a support cost, a binary size, a battery budget and a class of hardware you now implicitly support. Model this as unit economics with the meter moved, not as free.

Related: small & local models for choosing the model itself, self-hosted inference for agents for the server-side sibling of this problem, the agent design-pattern landscape for where a hybrid split sits among the other patterns, and memory stores for what to put in that on-device index.