Tool Catalog Lifecycle

An addition is a change to every existing task.

The model picks a tool by reading all the descriptions together and choosing the best match. That makes selection a competition, and introducing a competitor changes the outcome of contests it was never meant to enter. The failure is not exotic and it is not the model being stupid:

• Description overlap. A new search_documents sits beside an existing query_knowledge_base, and calls that used to route correctly now split between them. Neither description is wrong; together they are ambiguous, which is the failure tool design anti-patterns catalogues.
• Attention dilution. Past a certain catalog size, selection accuracy degrades across the board rather than on the new tool specifically, and it degrades first on the tools with the least distinctive descriptions.
• Prefix invalidation. Tool definitions sit in the cached prefix of every request. Adding one changes the prefix, so the first call after a catalog change is a full cache write for everybody — and the definitions are then re-sent on every step of every run forever, which is a permanent per-step cost, not a one-off. See prompt caching.
• Name and schema collisions. Two tools with the same name from different servers, or the same name with different argument shapes across environments, produce failures that look like model errors and are not.

Admission: a tool arrives with evidence or it does not arrive.

Most catalogs grow by accretion — a team needed a capability, wired it in, and nobody owned the aggregate. An admission process costs very little and is the only thing that keeps the catalog reviewable:

• A named owner. Not a team inbox. Someone who is paged when the tool errors and who signs off on its description changing.
• The task it enables, and the eval that shows it. If the capability cannot be stated as a task the agent could not previously complete, the tool is a wrapper around something the agent already had.
• The regression run. Full existing eval set, before and after, with the delta attached to the request. A drop on unrelated tasks is a blocking finding, not a note.
• A disambiguation statement. Which existing tool is this closest to, and what one sentence in each description now separates them? This is the step that prevents the overlap failure, and it forces an edit to the incumbent's description as often as to the newcomer's.
• A scope declaration. Which task types or agents should see it, defaulting to the narrowest set that satisfies the requester rather than to everyone.

Design guidance on what makes an individual tool good — granularity, error messages, schema shape — lives in tool design for agents and tool granularity. This page is about the property no individual tool review can see: whether the set still works.

The catalog the model sees is not the catalog you host.

The cheapest fix for almost everything above is to stop presenting one global catalog. A per-task allowlist decouples "how many tools does the organisation operate" from "how many tools compete for this decision", and the two numbers should diverge sharply as you grow:

• Scope by task type first. A refund workflow does not need the deployment tools. Static allowlists are unglamorous, deterministic, and beat every dynamic retrieval scheme on the cases where you already know the answer.
• Then by tenant and permission. A tool the caller is not authorised to use should not be in the prompt at all, because a tool the model can see is a tool it will attempt and a refusal it will have to route around.
• Retrieve dynamically only where the set is genuinely open. Selecting a subset of tools by relevance is a real technique and adds a retrieval step that can be wrong, silently, with no error — reach for it when the catalog is large and open-ended, not as a substitute for deciding what a task needs.
• Stamp the resolved set on the trace. Every run should record the exact tool set and catalog version it saw. Without it, "the agent stopped using tool X last Tuesday" is unanswerable, and you will spend a day on it.

Skills and progressive-disclosure mechanisms shift the same problem to a different layer rather than removing it — agent skills load instructions on demand, but whatever is loaded still competes for the same attention, and the loading decision is itself a selection problem with the same failure modes.

Removal is the part nobody plans, and it needs a tombstone.

Deleting a tool is not the inverse of adding one, because by the time you want it gone the name has propagated into places you do not control: cached prompts, few-shot examples, stored plans, agent memory, and skills written by other teams. The agent will keep calling it, and a bare "unknown tool" error produces a retry loop rather than an adaptation.

• Retire in three phases. Announce and stop new adoption; make the tool return a structured deprecation error naming its replacement while still working; then remove. Skipping the middle phase is what turns a retirement into an incident.
• Tombstone the name. The removed name should keep resolving to an error that says what happened and what to call instead, for at least as long as anything might still hold a reference. An error a model can act on is worth more than a clean 404.
• Removing a tool is also a behaviour change. Run the same regression gate as for an addition. The agent may have been routing around a defect elsewhere using the tool you are deleting.
• Hunt the never-called. Any tool with zero calls in thirty days is either dead weight paying rent in every prompt, or a capability the model cannot find — and those two need opposite responses, which is why the list has to be looked at rather than automated on.

The lifecycle discipline here is the same one model deprecation and migration argues for on model IDs: a name you publish is a dependency someone else has taken, and removing it is a migration rather than a delete.

Version the catalog, and put the version on every run.

Behaviour is the (model, prompt, tools) triple that rollout and versioning insists on pinning, and the tools element is the one most often left floating — partly because a catalog assembled at startup from several servers has no single artefact to point at.

• Make the catalog an artefact. Serialise the resolved set — names, descriptions, schemas — hash it, and treat the hash as the version. Now a catalog change is a diff a human can read and a value a trace can carry.
• Roll it like code. Canary a catalog change to a fraction of traffic behind a flag, gated on the eval delta, with a config-flip rollback. There is no reason a tool addition should ship less carefully than a prompt change, and it usually ships far less carefully.
• Watch for drift you did not author. A description you do not own can change under you with no commit and no error, which is the entire subject of third-party tool drift. The hash is what turns that from invisible to a diff.
• Pin per environment, and reconcile. Staging and production drifting apart on tool descriptions produces evals that pass against a catalog nobody is running.

Measure the set, not the tools.

Per-tool dashboards are standard and mostly tell you about availability. The numbers that describe catalog health are comparative and almost nobody keeps them:

• Selection error rate — runs where the wrong tool was called for the task, sampled and labelled. This is the number that moves when you add something overlapping, and the only one that catches it.
• Catalog size against effective size — tools hosted versus the median number presented per run. If these converge as you grow, your scoping is not working.
• Tokens per step spent on definitions — the standing cost of the catalog, paid on every step of every run. It is usually larger than teams expect and it is the honest price of "we'll just add it".
• Calls per tool over thirty days — with the zero list reviewed by a human, not auto-pruned.
• Time from a tool request to admission — because a gate nobody can get through is routed around, and a catalog assembled by workaround is the outcome this whole page is trying to prevent.