Third-Party Tool Drift

The dependency nobody declared.

You pin your model. You version your prompts. You review every line of application code. Then you hand the model thirty tool definitions — names, descriptions, parameter docs, enum values, response shapes — most of which were written by someone at another company and are fetched at run time from an MCP server, a hosted connector platform, or an OpenAPI document you convert on boot.

Those definitions go straight into the context window. A tool description is not configuration; it is instruction text that competes with your system prompt for the model's attention. Which means the honest statement of your deployment's behaviour is (model, prompt, tools), exactly as your versioning policy already says — and two of those three are pinned while the third is fetched fresh from a third party on every process start.

The failure this produces is unusual in production systems: a behaviour regression with no deploy, no config change, no incident trigger, and a git log that proves nobody touched anything. Teams spend the first day of that investigation looking in the wrong repository.

Four kinds of drift, sorted by how loudly they fail.

The intuition to correct is that breaking changes are the danger. Breaking changes are the safe ones — they throw, you page, you fix. The expensive drift is the kind that leaves everything green.

• Structural drift — a tool disappears, a required parameter appears, a type changes. Loud. You get a 4xx or a schema validation error, and your existing alerting already catches it. Treat this as the best case.
• Semantic drift — the schema is unchanged but the meaning moved. An enum gains a value your prompt never mentioned; a field that used to be a country name is now an ISO code; pagination defaults change from 50 to 20 and your agent silently sees less data. Nothing errors. The agent takes a slightly wrong action, or the right action on incomplete input.
• Description drift — the vendor rewrites the docstring to be clearer, or adds "use this tool only when the user explicitly asks". That is a prompt edit applied to your system by a stranger, and it can move tool-selection rates by tens of percent. This is the one with no error surface at all.
• Behavioural drift — the contract holds and the service changed underneath: slower, rate-limited differently, stricter validation, results ordered differently. Your capacity assumptions quietly stop holding.

Snapshot the catalog and make the diff a review event.

The mechanism is boring and it is the whole fix. At build time, fetch every tool definition your agent will be given, serialise them canonically, and write the result to a file in your repository. Hash it. Compare on every build.

• Normalise before hashing — sort keys and tool order, strip volatile fields like server timestamps and instance IDs. An unstable hash produces alert fatigue in a week and then gets muted.
• Hash per tool as well as per catalog. The catalog hash tells you something moved; the per-tool hash tells you what, and lets you ignore an unrelated tool churning while watching the three that matter.
• Fail the build on an unreviewed change, the same way a lockfile change fails review when nobody looked at it. The diff is small and readable — a changed description is a two-line diff and a thirty-second decision.
• Serve from the snapshot where you can. Some platforms let you pin a tool version or supply your own schema; a connector platform with schema modifiers lets you define the shape the model sees regardless of what upstream did. That converts drift from a behaviour change into a merge conflict, which is the correct place for it.

This is the same discipline as pinning a model to a dated snapshot, applied to the other unpinned half of the behaviour triple. Teams that already do one and not the other usually have not noticed the symmetry.

Contract tests that check meaning, not just shape.

Schema validation catches structural drift, which you were already going to catch. The tests worth writing assert the semantics your prompt quietly depends on. Keep a dedicated sandbox tenant per integration and run a small suite against the live third party on a schedule — nightly is usually enough, and it is deliberately not in the pull-request path, because a vendor's outage must not block your deploys.

• Assert the enum sets you branch on. If your prompt or your code knows about open | pending | closed, a test should fail the day a fourth value appears — before the model meets it.
• Assert a golden call end to end. One representative invocation per tool with a fixed input, checking that the result still has the fields you extract and that they still mean what they meant. Formats drift more often than schemas.
• Assert the defaults. Page size, sort order, timezone, currency, truncation limits. These are undocumented as often as not, and they change without a changelog entry.
• Diff the description text explicitly. No assertion is possible, so make the test a snapshot: the description changed, here it is, approve or adapt. Treat an approved description change as a prompt change and re-run the eval subset that exercises that tool.

See tool schemas & contracts for what a good contract looks like from the authoring side; this is the consumer's half of the same problem.

Detect it in production, where the real catalog lives.

Your build-time snapshot describes what you expected. Production is where the model met what was actually served, including per-tenant differences you cannot see from CI — connector platforms return different tool sets depending on which integrations a customer has authorised, so no two tenants necessarily hold the same catalog.

• Stamp the catalog hash on every trace. One field, alongside the model ID and prompt version you already record. It makes "did the tools change" a filter rather than an archaeology project, and it is the single highest-value line of code on this page. See tracing & observability.
• Watch tool-selection rates per tool, per day. A description edit shows up here and nowhere else: the same traffic, the same prompt, and suddenly a tool is chosen 40% more often. This is the leading indicator for description drift and it costs one dashboard.
• Watch the argument distribution, not just the call count. A shift in which enum values the model passes, or a parameter that used to be supplied and now is not, is semantic drift arriving. Cheap to compute from traces you already keep.
• Split error rates by class per tool. Validation errors, auth errors, rate-limit errors and timeouts drift for different reasons and blend into an unreadable line when aggregated.
• Alarm on the arrival of an unknown enum value or an unknown tool name, which is the earliest machine-detectable signal that upstream moved.

Respond at your boundary, not in your prompt.

When drift lands, the tempting fix is to patch the system prompt: "note that the status field may also be archived". It works today and it accumulates into a prompt that is a changelog of other people's decisions. Fix it one layer lower.

Put a thin facade between the agent and every third-party tool — your own name, your own description, your own parameter set, mapping onto theirs. It is a small amount of code and it buys three things: the model's view of the tool stops changing when the vendor edits prose, you can absorb a rename or an added enum in one adapter instead of across several prompts, and you can shrink a bloated vendor surface to the operations you actually use. That last one is a quality win independent of drift — see tool design anti-patterns.

Then run the response as a change, because it is one:

• Re-run the eval subset that touches the affected tool before accepting the new snapshot. A tool change is a behaviour change and deserves the same gate as a model swap — see quality regression detection.
• Keep a per-tool flag so you can withdraw one tool from the catalog without a deploy when a vendor ships something bad on a Friday.
• Record the drift event in the same place as incidents. Three description changes to the same vendor in a quarter is procurement information, and nobody will have it unless someone wrote the first two down.

Related: rollout & versioning for pinning the other two legs of the behaviour triple, MCP ops in production for running the servers these definitions come from, MCP tool poisoning for when the description change is hostile rather than careless, and agent inventory & registry for knowing which agents hold which tools when one of them moves.