Third-Party Model & Vendor Risk

Your supply chain is longer than your vendor list.

Procurement usually registers one vendor: the model provider. A deployed agent typically depends on five or six independent parties, each of which can change under you, and most of which never went through review because nobody classified them as vendors.

• The model provider — and, separately, whoever hosts the inference. The same open-weights model served by two providers is two different systems with different quantization, different serving stacks, and different latency and refusal behaviour. See inference providers.
• The gateway or routing layer, if you use one, which sits in the path of every prompt and every completion and therefore sees everything both ways.
• Every tool server the agent can call. A third-party MCP server is code that runs against your credentials and returns text into your model's context. It is a supplier with production access, and it is the one most likely to have been added by an engineer in an afternoon.
• The embedding and reranking models, whose replacement silently invalidates an index you may have spent a week building.
• The evaluation stack — including whatever model grades your outputs. If your judge model changes, your quality metric changes, and you will read that as a change in your agent.

The three clauses that decide whether your evals stay true.

Most of a vendor questionnaire is unfalsifiable at signing time. These three are specific, testable, and directly determine whether the system you validated is the system running next month.

• Model version stability. Can you pin an exact model version, or does your endpoint resolve to a moving alias? If it moves: what notice do you get, how long do pinned snapshots live, and does "the same version" cover changes to the system prompt, the safety filters, and the sampling defaults, or only to the weights? An alias that silently rolls forward means your evaluation results expire on a date the vendor chooses and does not tell you. This is the same exposure discussed in model deprecation and migration, seen from the contract side.
• Subprocessor change. Who else touches the data, and what happens when that list changes? The standard is advance notice with a right to object; the common reality is a page on a website that updates without a notification. If your own regulatory filings enumerate subprocessors, an unannounced addition puts you out of compliance without any action on your part.
• Retention and training use. How long are prompts and completions kept, who can read them, are they used to improve the service, and can any of that be changed unilaterally? Note that abuse-monitoring retention is often carved out separately from the headline "we don't train on your data" claim, and it is usually the longer of the two. This is the clause most often described accurately in the terms and inaccurately in the sales conversation. See data governance.

Write the answers down as three dates and three durations, not as prose. "Pinned snapshots supported, minimum six months' deprecation notice, subprocessor notice 30 days with objection right, prompt retention 30 days, abuse-monitoring retention 12 months, no training use" is a risk position you can act on. "The vendor is committed to responsible AI" is not.

What a certification does and does not tell you.

Certifications are worth collecting and worth understanding precisely, because the gap between what they attest and what people believe they attest is where the false comfort lives.

• SOC 2 Type II attests that specified controls operated over a period. It is real evidence about a company's operational discipline — access control, change management, monitoring. It says nothing whatsoever about model behaviour, output quality, or whether the model you call next week is the one you tested.
• ISO/IEC 42001 certifies an AI management system: that the organisation has a documented process for managing AI risk. Process, not outcome. It is a meaningful signal that someone is accountable internally, and a meaningless one about any specific model.
• Model and system cards are the most useful artifact of the three and the least standardised. Read them for what was evaluated and on what data — then notice that the evaluations were run on the model in isolation, and you are deploying it in a loop with tools and untrusted input, which is a different system with different failure modes.
• Regulatory status is the one that shifts obligations onto you. Under regimes like the EU AI Act, integrating someone else's model into your product does not transfer their compliance to you; you acquire your own duties as the deployer, and in some configurations you inherit provider duties. See the EU AI Act for agents.

The practical reading: certifications tell you whether a vendor is a serious operation, which is genuinely useful for deciding whether to sign at all. They tell you nothing about the risk you are actually managing, which is that the component's behaviour changes and your system quietly stops doing what you validated.

Ask for evidence, not assurances.

Replace the unanswerable questions with a short list of requests that have concrete deliverables. Every item here is something a competent vendor can produce in a week, and a vendor's response to being asked is itself the signal.

• A change log for the model endpoint, covering the last twelve months. Not a roadmap — a history. How often did behaviour change, and were customers told before or after?
• The deprecation record. Which versions were retired in the last two years, and how much notice did customers actually get, as opposed to how much the policy promises?
• Incident history and status-page archive. Availability is easy to check yourself; what you are reading for is the quality of the disclosure. Vague post-incident notes are a preview of what you will get during your own outage.
• The data-flow answer in writing: which regions the request transits, where it is stored, and which of those are configurable versus fixed. Relevant to data residency, and often different from what the marketing page implies.
• A named contact and an escalation path with a time. "Enterprise support" without a response-time commitment is a mailing list.

The controls you own regardless of what the vendor promises.

Contract terms allocate liability after the fact. These four change what actually happens, and none of them requires the vendor's cooperation.

• Pin versions and treat a version bump as a deploy. If the provider offers pinned snapshots, use them, and move to a new one through the same review a code change gets. If they do not, that fact belongs in your risk register in writing, because it means part of your production system is on someone else's release schedule.
• Route everything through one egress point. A gateway you control gives you the change-detection you were trying to get contractually: per-model request logs, the ability to shadow a new version against the old one, and a single place to switch providers. It is the same argument as the collector in observability — it converts a vendor from a dependency compiled into your services into a configuration line.
• Re-run evals on every vendor-side change you detect, and make the change detectable in the first place by recording the model identifier and any version header on every trace. Without that, a behaviour shift arrives as an unexplained quality regression that someone spends a week attributing to their own prompt edit. See eval-driven agent development.
• Keep a second provider warm enough to be real. Not necessarily live traffic — but prompts that are portable, an eval suite that runs against both, and a documented switch that someone has actually executed once. An untested failover is a plan, not a control.

Tool servers get the same treatment with a shorter leash: pinned versions, no auto-update, credentials scoped to exactly what that server needs, and review of what it returns into context. The security mechanics are in agent supply-chain security and MCP security anti-patterns; the governance point is simply that a tool server is a vendor and belongs in the register with the rest.

Who answers when it goes wrong.

The final governance question is not about the vendor at all. When an agent built on a third-party model causes harm to a customer, that customer contacts you, the regulator contacts you, and the vendor's terms of service — which almost certainly disclaim liability for model output and cap damages at some months of fees — do not help with either conversation. Assume the liability sits with you and design accordingly; you will be right nearly always, and the exceptions are worth the specific legal advice they require.

• Name the accountable owner per vendor, not per system. Someone must be responsible for noticing that a supplier changed. See accountability and roles.
• Make your audit trail vendor-independent. If reconstructing what happened requires the provider's logs, you cannot reconstruct it on your own timeline — and regulator timelines are not negotiable.
• Rehearse the vendor-caused incident specifically. Most incident plans assume the fault is yours. The drill worth running is: the model changed, quality dropped, the vendor has not acknowledged it. Who decides to roll back, and to what?

Related: model deprecation and migration for what to do when the change arrives, data governance for the data half of the same review, and governance in practice for how this fits the rest of the programme.