Production and Handoff
The build takes weeks and trust takes months, so an FDE who leaves at go-live leaves a system nobody uses or nobody can run. Treat handoff as an acceptance test the customer's operators pass without you, and treat adoption as a deliverable with its own definition, baseline and owner. This chapter ends with a handoff document and runbook you can copy.
What this phase is for, and how it fails
By now the hard-looking work is done. The agent clears the threshold, the review has signed, and the build runs inside the customer's environment (Building in Their Environment). It looks like the finish line. It is where the customer's business starts to change, or does not.
Production has two tests beyond the technical one. The customer's people must be able to run the system without you, and they must actually use it. Each has its own failure, and both look like success on the day the vendor leaves.
Only the engineer can operate it. Alpine Edge gives this, in those words, as the early warning of "handoff failure" (Alpine Edge, August 2026; T4, one consultancy's practice). Note where it sits: in a table of risks to address during discovery, with the fix "Name customer owners early; ship runbooks; run an incident drill." Handoff is planned in week one, or it fails at the end. The knowledge of why the system works lives in one person's head and shell history. When that person rotates to the next customer, the first model update, expired credential or odd input becomes an outage. The customer learns that it bought a dependency, not a capability.
It sits there unused. Tryolabs calls this "a common pattern": three months on, "the tool sits there unused, and you are the one who has to explain why". Its diagnosis is structural: "Discovery, build, and adoption end up owned by three different people who answer to three different priorities" (Tryolabs, June 2026; T4, a firm describing its own field). The FDE role exists to put those three under one owner. An FDE who treats adoption as someone else's job has split them again.
How long trust takes is the part people underestimate. Jarvis says that at Morgan Stanley "the main technical hurdle was probably solved within like 6 to 8 weeks", and that "a further like 4 months of like doing pilots, collecting evals, and iterating" followed before the advisors trusted it (Altimeter interview, November 2025; T2). What matters here is how they were spent. ZenML's summary of the talk says "technical readiness wasn't sufficient" and that the team "worked closely with advisors to label data, verify outputs" (ZenML LLMOps database; T3, an echo of the same talk: a secondary summary of one vendor's account of one customer). The trust period ran two to three times as long as the build, and it was spent with users, not on the model.
Jarvis has since made the general claim. TNW reports him, at HumanX, putting the problem down to deployment "in about 80% of cases". In its summary, companies "do not yet know how to roll AI out, govern it or prove they can trust it" (TNW, September 2026; T3, a journalist relaying a vendor executive whose team sells deployment). Discount it for the source's interest; the direction still holds.
So the argument is this. Handoff is not a meeting at the end; it is an acceptance test that the customer's operators pass without you in the room. Adoption is not a hope; it is a deliverable, defined and measured like the threshold was. Plan the last months of the engagement around both, and budget them: if the build took six weeks, do not book your exit for week eight.
The composite: the drill and the slow yes
Months three to six at the insurer. Routing accuracy reached the claims director's threshold roughly two to three months after the pilot, and go-live followed as a staged rollout inside the insurer's private cloud. The first stage sent suggestions to a minority of the auto-claims adjuster teams; the rest widened over several weeks once each stage held, with the old queue one switch away. Every release bundled the code, the prompts, the pinned model version and the routing rules, so rolling back meant redeploying the previous bundle, not reconstructing it. The pattern is Sierra's immutable releases, which let customers "roll back agent behavior instantly".
The drill. Partway through the rollout, weeks before the FDE's last, the insurer's platform operations team ran a handoff drill. Its engineers deployed a release from the runbook through their own change process. Then they dealt with failures the lead had prepared: the model endpoint refusing requests (claims should fall back to the human queue, and the alert should page someone), a batch of scanned documents the parser could not read, and a release that had to be rolled back. The FDE sat in the room and said nothing. Two gaps surfaced: the runbook assumed a credential the ops team did not hold, and nobody knew who should tell the adjusters when suggestions were switched off. Both were fixed, and a re-run a few weeks later passed without a word from the FDE. The pattern is Alpine Edge's operator test: deploy, "investigate a failed task and practise recovery using the documentation".
The slow yes. For the first weeks, most adjusters ignored the suggestions and triaged as before. The FDE did not add training slides. Instead every override (an adjuster moving a claim to a different queue from the one suggested) was logged as a label. The recurring ones were fixed in the next release, and the claims director showed the adjusters which of their overrides had changed the agent's behaviour. Use rose over the following months as adjusters saw their own corrections land. The FDE's brief to the COO put her cycle-time KPI beside time-to-first-action every time, so the board number and the number that moved first were never read apart. The patterns are Morgan Stanley's months of advisor pilots after a weeks-long build, and the adoption work Palantir writes into its Deployment Strategist role.
Three things in that beat carry the chapter. The rollout could be undone. The drill was run by the people who would own the system, with the vendor silent. And adoption moved when users saw their own judgment change the system, not when they were told to use it.
Ship in stages you can undo
Go-live is not one event but a sequence of stages, each small enough to fail cheaply: shadow mode, then one team or a small share of traffic, then wider slices, with the old process still available behind a switch. Alpine Edge's week-four checklist puts it in one line: "If ready, use a limited or shadow release with tested rollback" (T4). Write each stage's exit rule before the stage starts, and keep it dull: it has run for an agreed number of working days on real volume, no guardrail has been breached, the override rate is flat or falling, and the threshold owner has said yes. A stage that widens because a meeting felt good is not a stage.
Two things make a stage undoable. Insist on both before go-live.
A release is everything that changes behaviour. An agent's behaviour depends on more than its code. Sierra describes each agent release as "a snapshot that includes not only the underlying source code and prompts" but also "model version dependencies and an immutable snapshot of all of the knowledge available to the agent". As a result, "Immutable agent releases enable Sierra customers to roll back agent behavior instantly if necessary" (Sierra, June 2024; T4, a vendor describing its own platform). You do not need Sierra's platform to copy the idea. Version the prompts, pin the model version rather than a floating alias, snapshot any rules or reference data, and deploy them together. The wiki's page on rollout, versioning and pinning covers the mechanics, including the surprise that behaviour can change when the provider moves an alias and nobody deployed anything.
Off must mean "back to the old way", not "broken". A kill switch is only useful if what happens after it is safe and known. For a propose-only agent this is easy: suggestions stop and work flows through the old queue. For anything that acts, decide in advance what in-flight work does when the switch is thrown. Write the answer into the runbook, and make sure the people who will throw the switch know who to tell. The composite's drill found exactly that gap: the switch worked, and nobody knew who should tell the adjusters.
Sierra's tests "can be run in parallel during development and prior to every agent release". Copy that: the regression suite from the evals chapter becomes the gate on every release the customer ships after you leave, and it is one of the main things you hand over.
Adoption is a deliverable
A system that clears its threshold and changes no one's work has not delivered the outcome the engagement was sold on. If adoption is part of the deliverable, it needs what the threshold needed: a definition, a baseline, an owner and a date.
Define it before someone else does. The NHS Federated Data Platform shows what happens when adoption has no agreed definition. The Register reported that "fewer than a quarter (45) of England's 215 hospital trusts were actively using" the platform by the end of 2024. It also reported that "72 are already live as part of a phased rollout", and that "more than 120 NHS trusts had signed up to use the FDP" (The Register, May 2025; T3, a journalist relaying NHS figures). Signed up, live and actively using are three different numbers for one programme, and each side of the argument can pick the one that suits it. The same report says £8.8 million went to KPMG for adoption support: a measure of how seriously the buyer took the problem, not of whether the spending worked.
For an FDE engagement, write the adoption definition next to the threshold, in the handoff document, in terms of what users do: suggestions shown, suggestions accepted, suggestions overridden, and the share of eligible work that went through the new path. Measure it from the first day of rollout so that there is a baseline. Give it a customer-side owner, usually the same person who owns the threshold.
Then read the split, because the three failure shapes need different fixes. Ignored (shown, neither accepted nor overridden) usually means the suggestion arrives in the wrong place or at the wrong moment; sit with users and fix the workflow, not the model. Overridden means they looked and disagreed; that is an eval problem, and the overrides are its data. Accepted without looking (acceptance near total, review time near zero) is the dangerous one, because the human check you promised the model-risk function has quietly stopped. Watch for it as closely as for low use.
Overrides are the signal. Users vote with corrections long before they answer surveys. An adjuster who moves a claim to a different queue has produced a labelled example for free. The wiki's page on production feedback signals argues that the edit, not the thumbs-up, is the densest quality signal a system produces. Two things follow. Route overrides into the eval set and the regression suite, so that recurring ones become fixes. Then show users that it happened: people who see their corrections land start to treat the system as theirs.
Training is part of the build. Palantir writes training and breadth of use into the job of the non-coding partner on its deployments (box below). The evidence that this works is thin. The most-cited Palantir result is a Walgreens executive's remark, relayed by Palantir's chief revenue officer on an earnings call, that Foundry and AIP enabled "end-to-end workflows in 4,000 stores within eight months" (Palantir Q1 2025 call transcript; T2, a vendor reporting a customer's words). It measures how far the rollout reached, not who used it or what changed. Read it as a vendor treating breadth as the headline, not as evidence of adoption.
How employers write handoff and adoption into the job. Anthropic's Staff Forward Deployed Engineer posting (October 2026) asks the engineer to "deliver production systems the customer can operate" and to "hand over systems the customer can run" (posting). Palantir's Deployment Strategist posting asks the strategist to "Lead training sessions to ensure that the product is meeting user needs", and to make sure the product "is being used widely enough to have concrete impact on our partners' operations" (posting). Both are primary evidence of what these employers ask for, and T4 for how the work is actually done.
Briefing the executive
The sponsor stops reading engineering updates around go-live. The evals chapter covered the one-off meeting where the news is bad. Production needs a short, regular brief that holds the sponsor's attention after the drama, because the remaining decisions are theirs. This is practice, not data.
Lead with their number, and pair it with the one that moves first. The sponsor's KPI is usually a lagging indicator: cycle time, cost per case, revenue. It moves slowly and for many reasons at once. The engagement's own metric, such as time-to-first-action, moves first and is more clearly yours. Report them side by side every time, against the same week-one baseline. Report only the leading number and you ask them to trust a metric they report to nobody; report only the lagging one and the first bad month is yours.
Split the lagging number by path. Show the KPI for eligible work where the suggestion was accepted beside eligible work where it was ignored or overridden, over the same weeks. It is not a controlled comparison; the claims adjusters accept may be the easier ones, and the brief should say so in a line. It is still the most honest way to show the sponsor whether the system is moving their number, rather than a quiet month moving it for you.
Give them the sentence for their boss. Draft the one line the sponsor can repeat upward, in their KPI's units, with the caveat attached: what moved, for which share of the work, measured how. An executive who has to write that sentence alone will round it up, and the inflated version becomes the number you are held to.
Report adoption with its definition. Write "accepted suggestions as a share of eligible claims", not "adoption", and keep the definition identical from brief to brief.
Close with the decision in front of them. At this stage the decisions are concrete: widen the rollout, fund the operator time the runbook needs, sign the handoff, or trigger the next "not now" item. Alpine Edge frames it the same way: "The handover should let the customer decide whether to expand, revise or stop the project" (T4). One page, one chart, every two weeks until handoff and monthly after; detail beyond that belongs in the threshold owner's review.
The handoff drill
The only way to know the customer can run the system is to watch them do it. Alpine Edge's version of the test is specific: have "the future operator deploy the application, investigate a failed task and practise recovery using the documentation", then "Resolve gaps before the engineer leaves or define the support needed to cover them" (T4). Its week-4 checklist adds: "Name the operational owner and provide the runbook." Run the drill as an acceptance test, not a training session, and agree the pass condition in writing beforehand. A workable one: every scenario is detected, contained and recovered from the runbook alone, within a time the service owner accepts, with no FDE interventions.
- The operators drive. The people who will be paged after you leave do every step, from their own accounts and with their own credentials. If they need you to unlock something, the drill has already found a gap.
- The FDE stays silent. Every time you explain something, write down what you said; that sentence is missing from the runbook. Silence is uncomfortable and it is the point.
- Break it on purpose. Prepare at least three failures in advance: a dependency the system calls is down, an input it cannot handle, and a bad release that has to be rolled back. The wiki's page on incident response for agents lists the failure classes worth rehearsing, including an agent that keeps acting while you scramble.
- Record the gaps, fix them, and run it again. A drill that finds nothing was probably too easy. A drill that finds gaps and is not repeated has not been passed.
- Count cost the way they will. Alpine Edge advises: "Track cost per successful task, not per token." Finance will ask the operator what the system costs; hand them cost per claim triaged, not tokens per month.
Run the first drill early enough to fix what it finds and run it again: weeks before you leave, not days. If the re-run fails, your exit date moves, or the gap goes into the support window in writing.
Leaving on purpose
The FDE plans to leave from the first week, and production is where the plan gets tested. Jarvis put OpenAI's position bluntly: "From OpenAI's side, we should always be temporary" (TNW; T3). The ZenML summary of his talk describes the team as "a zero-to-one team focused on breaking the back of novel, high-value problems" rather than "long-term service delivery" (T3, an echo of the same talk). Tryolabs gives the test from the other side: "It is whether it is still in use after we are gone", and "When an FDE leaves, your team is more capable, not more dependent" (T4).
Three owners on the customer side, by name. Someone owns the service (deploys, alerts, credentials). Someone owns its behaviour (the threshold, the eval set, the decision to ship a change). Someone owns adoption (the definition, the review, the training of new users). These may be the same person; they must not be the vendor.
A written support window with an end date. "Call us if anything goes wrong" is not a handoff; it is an open-ended services contract nobody priced. Write what the vendor will do, for how long, through which channel, and what happens after.
Change control they can run. The system will change after you leave: a new model version, a new claim type, a rule the regulator adds. The handoff document must say who can approve a change, which suite runs before it ships, and what score blocks it. Without that, the first change after you leave is made by whoever is in a hurry.
One honest note. No public source this book found measures adoption of an FDE-built system independently, or checks whether it is still running and used a year after the FDE leaves. The practices in this chapter rest on practitioner accounts and one well-documented counter-example, the NHS Federated Data Platform. They are the best available guess at what makes a handoff last, and a guess is what they are.
What you learned does not stay with the customer alone. The parts that will work again are the subject of the next chapter, Feeding the Product.
The template: the handoff document and runbook
One document with two halves. The top half is the handoff: what the system does, who owns what, and what was proved. The bottom half is the runbook: how to deploy it, roll it back, stop it and fix it. Write it with the operators, not for them, and treat the drill as its test.
HANDOFF DOCUMENT AND RUNBOOK version / date: ____
1. THE SYSTEM
What it does: ____________________________________________
What it never does: ____________________________________________
Authority: propose only | act with approval | act
Rules enforced outside the model: ______________________________
2. OWNERS (customer side, by name)
Service (deploy, alerts, credentials): ________________________
Behaviour (threshold, eval set, changes): ______________________
Adoption (definition, review, training): _______________________
Vendor support: scope ________ channel ________ ends ________
3. WHAT WAS PROVED
Threshold met: metric ________ held-out set ________ date ____
Adoption definition: ___________________________________________
Adoption baseline (date): ______________________________________
4. RELEASES
A release contains: code / prompts / pinned model version /
rules / reference data snapshot
Deploy: ____________________ Who may deploy: ______________
Rollback: ____________________ Last rehearsed: ______________
Before every release: run ________ suite; block if ____________
5. HEALTH (what normal looks like)
Signal Normal Alert when Who
_________________ _________________ _____________ _______
_________________ _________________ _____________ _______
6. KNOWN FAILURES
Symptom First check Fix / workaround
_________________ _________________ ________________________
_________________ _________________ ________________________
7. KILL SWITCH
How to stop: _______________________________________________
What happens: _______________________________________________
Who to tell: _______________________________________________
8. ADOPTION REVIEW
Measures: shown / accepted / overridden / ignored / share of
eligible work; accepted-without-review rate
Overrides go to: ________ Reviewed by: ________ every ________
9. HANDOFF DRILL
Date: ______ Operators: ________________ FDE silent: yes / no
Pass condition: ________________________________________________
Scenarios: _____________________________________________________
Gaps found / closed: ___________________________________________
Re-run date and result: ________________________________________
10. SIGN-OFF
Service owner (we can run it): __________ date: ______
Behaviour owner (it is good enough): __________ date: ______
Sponsor (we accept the handoff): __________ date: ______
Vendor lead (we have handed over): __________ date: ______
Here it is filled from the composite. Ranges and descriptions stand in for the measured values a real document would carry.
HANDOFF DOCUMENT AND RUNBOOK: auto FNOL triage version n / months 5-6
1. THE SYSTEM
What it does: suggests a queue for each incoming auto FNOL
claim, shown in the adjuster's queue view
What it never does: writes to the claims system; acts on a claim
Authority: propose only
Rules enforced outside the model: litigation-flagged claims never
go to the fast queue; listed claim types always
go to a human queue
2. OWNERS (customer side, by name)
Service (deploy, alerts, credentials): platform ops team lead
Behaviour (threshold, eval set, changes): claims director
Adoption (definition, review, training): claims director
Vendor support: scope: defects channel: ticket
ends: agreed date, a few months after handoff
3. WHAT WAS PROVED
Threshold met: routing accuracy >= senior adjusters; same-day
first action; zero litigation-flag breaches on
the held-out set (claims operations owns it)
Adoption definition: accepted suggestions / eligible auto claims
Adoption baseline (date): measured from first rollout stage
4. RELEASES
A release contains: code / prompts / pinned model version /
routing rules / reference data snapshot
Deploy: pipeline in the insurer's private cloud, through
the weekly change board
Who may deploy: platform ops, after behaviour owner approves
Rollback: redeploy previous release Last rehearsed: drill 2
Before every release: run regression suite; block if below
threshold or any guardrail failure
5. HEALTH (what normal looks like)
Signal Normal Alert when Who
queue lag minutes above agreed ops
unreadable docs a small share share jumps ops
fallback to human rare any sustained ops
override rate falling slowly rises for days director
6. KNOWN FAILURES
Symptom First check Fix / workaround
no suggestions model endpoint claims fall to human
queue; restore endpoint
parser errors new scan format route to human queue
odd routing burst last release roll back, then triage
7. KILL SWITCH
How to stop: feature flag in the queue view config
What happens: claims flow through the old manual queue
Who to tell: claims director, then team leads by message
8. ADOPTION REVIEW
Measures: shown / accepted / overridden / ignored / share of
eligible work; accepted-without-review rate
Overrides go to: eval set Reviewed by: claims director
every: two weeks, then monthly
9. HANDOFF DRILL
Date: month 4-5 Operators: platform ops FDE silent: yes
Pass condition: each scenario detected, contained and recovered
from this document, no FDE help
Scenarios: endpoint refuses (fallback + alert); unreadable scan
batch; bad release rolled back
Gaps found / closed: missing ops credential (granted); nobody
to tell adjusters on kill (now section 7)
Re-run date and result: weeks later; passed, no FDE help
10. SIGN-OFF
Service owner (we can run it): platform ops lead
Behaviour owner (it is good enough): claims director
Sponsor (we accept the handoff): COO
Vendor lead (we have handed over): FDE lead
Two things in the filled version are easy to miss. The override rate is a health signal, not just an adoption measure: a sudden rise is often the first sign that the inputs have changed. And the vendor support window has an end. Without one, the document describes a dependency, whatever its title says.
Failure signals
Most of these show up in the weeks around go-live, while there is still time to fix them before you leave.
Ready to move on
The engagement is handed over when every box is ticked:
- The rollout ran in stages, each release bundles everything that changes behaviour, and rollback has been rehearsed.
- The kill switch returns work to the old process, and the runbook says who to tell when it is used.
- The customer's operators passed the handoff drill without the FDE's help, and every gap it found is closed or covered by a written support term.
- The service, behaviour and adoption owners are named on the customer side, and none of them is the vendor.
- Adoption has a written definition, a baseline and a review the customer runs.
- The sponsor has had a brief that pairs their KPI with the leading metric and ends with a decision.
- The vendor support window has a scope and an end date, signed with the handoff document.
Sit silent through the drill, and write down every sentence you want to say. Each one is a line missing from the runbook. When you can watch the customer's operators deploy, break and recover the system without saying anything, you have handed it over; until then, you are still running it for them.