Concepts

AI, machine learning, and deep learning are nested circles, not synonyms — and which one you are looking at predicts how a system fails.

A neural network is a big stack of adjustable knobs that turns numbers into numbers; learning is just nudging those knobs toward less error.

An LLM is a huge next-token predictor; scale turned that one simple objective into abilities nobody explicitly programmed.

Training builds the model’s frozen weights once; inference runs them per request and never changes them — which answers most cost and privacy questions.

Models see integers, not text; the hidden tokenization step explains your bill, your context limit, and odd failure modes.

Embeddings turn things into points in space so that similar becomes close — the engine behind search, recommendations, and RAG.

Self-attention lets every token look at every other token; that one idea fixed long-range memory and unlocked GPU-scale training.

The model returns a probability distribution, not an answer; temperature reshapes it, and temperature 0 is low-variance, not deterministic.

A model invents fluent answers because fluency, not truth, is what it optimizes — so fabrication is structural, not a bug awaiting a patch. Grounding puts the evidence in the context and requires the answer to come from it, which does not eliminate invention but makes it cheap to catch.

Model-generated training data is safe in proportion to how good your filter is — collapse is a property of the pipeline, not of the data.

Distillation trains a new, smaller model; quantization stores the same weights at lower precision. One is GPU-days and irreversible, the other is minutes and is not.

Refusals, sycophancy, formatting habits and the assistant persona itself are installed after pre-training, in a comparatively cheap layer rebuilt for every release — which is why a minor version bump can move behavior more than a capability upgrade.

One model call is two machines with opposite bottlenecks: prefill reads the prompt in parallel and is compute-bound, decode writes one token at a time and is memory-bandwidth-bound. That split explains time-to-first-token, why output costs more than input, and why prompt caching must be a prefix match.

Temperature 0 is a sampling rule, not a guarantee: inference servers batch your request with other people’s, and many kernels change their reduction strategy with batch size, so identical greedy requests return different text. The consequence for agents is that a failing run cannot be reproduced by re-running it — reproducibility has to come from records.

The cutoff is a gradient, not a wall — knowledge of the months just before it is thinner than knowledge of two years earlier, which is exactly where confidence outruns evidence. And a model with no clock turns every date question into a fluent, self-consistent, wrong answer.

The one speedup that provably cannot change what the model says — a cheap draft guesses ahead and the real model verifies — which is why it costs you no eval cycle, and why it quietly loses throughput on a busy GPU.

Parameter count stopped meaning cost: MoE splits the bill so compute follows the parameters that fire per token while memory follows all of them — which makes the same model a bargain on a rented API and an expensive mistake on your own GPUs.

LoRA won because the specialisation ships as a 50 MB file, so one base serves a hundred variants and switching becomes routing — and because rank is a capacity dial that decides whether you can teach behaviour (yes) or facts (no, that is retrieval).

An agent is a model placed in a loop with tools, choosing each next action toward a goal — the core mental model.

Reason → act → observe → repeat: tracing a tool call from the model through the harness into the environment and back.

A five-rung ladder from suggest to fully autonomous, and why the right level is a per-action engineering decision.

One question — who decides the next step — sorts any LLM system into chatbot, pipeline, workflow, or agent.

What a tool really is, read vs write actions, and why the environment — not the model — is where agents become dangerous.

The planning spectrum from reactive to deliberative, and the under-appreciated hard problem of knowing when an agent is done.

The three properties a task needs to justify an agent, the cheaper patterns that solve most cases, and clear do-not cases.

The four characteristic loop failure modes, the security shift autonomy brings, and what "safe agent" honestly means.

What prompt injection actually is, why it's not a bug a vendor can patch, and the three real defenses available to you.

The context window is an agent’s short-term memory and it resets every session; durable behavior needs an external long-term store you write to and retrieve from on purpose.

When there is no API, an agent can drive the screen itself — reading a screenshot and synthesizing clicks and keystrokes — which unlocks any software but is slow, brittle, and a fresh attack surface.

One strong agent is the baseline, not the goal; you reach for multiple cooperating agents only when a task is genuinely parallel or needs separate specialized contexts — and you pay in coordination cost, error propagation, and tokens.

Talking to an agent in real time makes latency the whole design problem, and forces one architectural choice: a swappable STT→LLM→TTS cascade you can inspect, or a single speech-to-speech model that trades that control for speed and natural prosody.

Human-in-the-loop is not the opposite of automation — it is where you place a human checkpoint. Gate the few consequential, irreversible actions and let the rest run; the trap is the rubber-stamped approval that adds latency and false confidence while catching nothing.

Code is the universal tool — and agent-written code is untrusted code, always, because its output depends on inputs you do not control. Isolation is five independent decisions (filesystem, egress, credentials, compute, lifetime), and network egress is the one most often left wide open.

Authentication, authorization, and attribution are three questions that one shared API key answers badly. An agent’s permissions should be the intersection of what the user may do and what the task needs — and because they are enforced outside the model, they are the one defense that still holds when prompt injection wins.

An agent re-sends the whole transcript every step, so total input grows with the square of the step count — which is why a cheaper model is almost never the fix.

When an agent picks the wrong tool or loops on a failing call, the bug is almost always in your tool — a tool is an interface for a reader with no docs, no memory between calls, and a token budget.

An agent that saves an hour is worthless if checking it costs fifty minutes — review cost is the metric your interface actually optimizes, and an editable plan plus a working undo beat every confirmation dialog you could add.

The one capability number stated in a unit you can plan with — the length of job an agent finishes on its own — except the headline is the 50% horizon and the deployable figure is the 80% one, roughly five times shorter. Measure your own on thirty real tasks and use it as the size limit on unattended work.

Every agentic benchmark number scores a model and the harness it ran inside, and only one of them is named — the loop, tool catalog, context policy and stop rule are the half you own, the half nobody publishes, and usually the half that was wrong. Spend a day on the harness before your next model upgrade and re-run the eval on the old model.

An agent in your logged-in browser profile did not get twelve tools, it got every site your cookie jar authenticates — and that unenumerated set, not the model, is what turns a prompt injection into a privilege escalation. Confirmation dialogs gate the actions someone thought of; the fix is to hand the agent scoped, per-task credentials so a persuaded agent holds nothing worth asking for.

The four levers that move output quality: instruction, context, examples, output shape.

Message roles, the instruction hierarchy, and never letting data act as instructions.

When examples beat instructions, how to choose/order them, and where they stop paying off.

The finite shared token budget, the three limit failures, and managing it actively.

The model proposes, your code disposes: the request/response shape, the loop, the safety rules.

Retrieve→augment→generate, RAG vs alternatives, and debugging it in two halves.

Why we chunk, embeddings as coordinates, nearest-neighbour search, hybrid + reranking.

From "ask for JSON" to schema-constrained decoding, plus schema design and defensive parsing.

Guardrails are pre/post-checks around a model call, not a wall around the model — what they catch, what they miss, and where they live.

An eval is a small, trusted scoreboard you run against your own task — why public benchmarks aren't enough, and what a useful eval set looks like.

Prompt engineering words one instruction; context engineering decides everything else that fills the window — retrieval, memory, tool results, history — and when to compact it. It is the core discipline of building agents.

Three ways to adapt a base model to your task change three different things — the instruction, the retrieved context, or the weights. Picking wrong wastes months; the right order is usually prompt, then RAG, then fine-tune.

An agent produces a trajectory, not a single answer, so grading only the final output hides broken paths — evaluating an agent means scoring the steps it took, on your own tasks, with cost and safety on the same scoreboard as accuracy.

Caching is a prefix match: stable content first, volatile content last, and one stray timestamp in the system prompt silently invalidates everything after it. Reads cost roughly a tenth of base input price, writes carry a premium — and in an agent loop that resends the whole transcript each step, this stops being an optimization and becomes structural.

A run is a tree of spans, not a log line: the prompt as actually sent, the tool arguments, the result, the cost, and a trace ID threading it all. Logging only the final answer records where the failure surfaced, never where it happened.

"Local" is three independent dials — where documents sit, where embeddings are computed, where generation happens — and most real setups are local on the first two only. Owning the pipeline buys provable data residency and a retrieval bill of zero; it costs you recall quality, index maintenance, and a full re-embed every time you change embedding models.

Vector search returns passages that look like the question, so it structurally cannot answer what is spread across documents. A graph stores relationships instead of prose — winning multi-hop and whole-corpus questions, and paying for it with an LLM pass over your entire corpus and an entity-resolution problem that never fully goes away.

Streaming does not make a model faster — it makes the wait legible, and it moves every output guardrail you own to a place where it may now run too late.

Every other cache can only be slow; a semantic cache can be wrong, because it decides a hit by similarity score rather than equality. It is not a caching feature — it is a retrieval system with a false-positive budget.

Adding a language breaks three things at once — tokens, retrieval, and evaluation — and generation quality, the part everyone tests, is the least broken of them. The cross-lingual retrieval miss is invisible from a fluent final answer.

A prompt is a parameter you are fitting by hand on a sample of three with no held-out set, which is why the version that reads better so often scores worse — the optimizer is a script, the scored dataset is the asset, and the winning prompt is a build artifact that expires the day you change models.

A vendor-neutral map of the major model families and a durable mental model for placing any new release.

Control, cost, privacy, licensing and lock-in — the real engineering trade-offs, without the marketing.

Text, vision, audio, code: input vs output modalities and why "multimodal" is a spectrum, not a checkbox.

Model size and the trade-off triangle that dominates production model economics, and how to engineer around it.

What inference-time "thinking" actually does, when it helps or wastes money, and why it is now a dial.

LangChain, LlamaIndex, provider SDKs and the broader landscape — by category and trade-off, not by brand.

Model choice and serving choice are orthogonal: first-party APIs, cloud catalogs, inference providers, self-host, gateways.

Why leaderboard rank rarely predicts your task, and why a small custom eval set beats every public number.

A repeatable, constraint-first decision procedure that synthesizes the whole topic and survives a fast-moving field.

MCP is an open standard that lets any model talk to any tool or data source through one interface — turning M×N custom integrations into M+N.

The question is not whether a model you can run yourself matches a frontier one — it does not — but which jobs never needed one. Embedding, reranking, routing and extraction are the high-volume steps, and they are exactly where a quantized 0.5–8B model on your own hardware is good enough and orders of magnitude cheaper.

MCP hands your agent a tool; A2A introduces it to a peer with its own goals, latency and failure modes — and the protocol solves discovery, not trust.

Routing pays only when judging "is this hard?" is cheaper and more reliable than answering — which is why static routing by task captures most of the savings, and a cascade needs a verifier you already have.

Residency is geography, sovereignty is jurisdiction, and a region toggle answers one of the four questions your legal team is asking. For an agent, the leak is usually the trace, not the model endpoint.

The same model at half price on both input and output, in exchange for a completion window measured in hours — and the work that qualifies is usually your highest-volume work. The structural catch: an agent loop can never use it.

A skill adds no capability — it is a folder of instructions the agent loads only when the task matches, so the hard part is not the instructions but the one-sentence description that decides whether they are ever read.

A managed agent runtime is sold as one product and is really five — compute, the loop, a tool gateway, an identity broker and a store for conversation state — and only the last one is hard to leave, which makes bundling rather than managed-ness the lock-in: ask what you would still hold if the loop product were frozen tomorrow.

Sold on catalog size, and the catalog is the part you will outgrow; the hard product underneath is a per-user token vault — OAuth per provider, refresh under concurrency, rotation, and a credential the model never sees. Buy the vault, and treat whose name is on the consent screen as the decision you cannot take back.

Routing, key custody, caching, policy and metering are five separable jobs sold as one product, and only the ledger is genuinely hard to rebuild — put the gateway in your request path and you have bought an availability dependency plus a percentage levied on every step an agent takes.

Three signals share the word "confidence" and only sampling agreement earns it, because the alignment step that made the model pleasant also made it overconfident — so the thing to build is not a number to display but an abstention threshold read off a coverage–risk curve.

A reasoning trace is generated text, not a log — models mentioned an answer-changing hint only 25–39% of the time — so approval gates, judges and injection detectors that read the thinking are grading a story; audit the tool calls instead, and never put the scratchpad in the reward.

A refusal is a policy applied at generation time on top of a capability that is still there, so "the model can't do that" is almost never true — which makes refusal rate the most volatile property of a deployed model, over-refusal a cost nobody measures, and the model's own willingness the worst place to put a control you need to hold.