Case File

The ten at a glance

Find the case, check its tier, and read the second half of the row before the first. Each name links to the full entry below.

NYC MyCity chatbot

What happened. New York City released its MyCity business chatbot in the autumn of 2023. Testing by The Markup and THE CITY in 2024 found it would "confidently provide incorrect and potentially harmful information", including suggesting landlords could discriminate against tenants with Section 8 vouchers. The city said it would improve it. A new mayor called it "functionally unusable", and The Markup's update of 4 February 2026 reported that the "beta test has ended" (The Markup, 2026).

Tier. T1 for the testing: The Markup, Documented and THE CITY ran it themselves, independent of anyone selling the system. The cost, "around half a million dollars" in the mayor's words, is T3.

What it proves. A confident wrong answer in a regulated domain can be measured from outside, by people with no access to the vendor. It also shows the slow fade: the evidence was public for nearly two years before anyone with authority called it.

What it does not prove. The testing measured the chatbot's answers, not the project behind them. It says nothing about who owned the system, what the build team knew, or why it was not fixed, and the book does not claim it was an FDE engagement. It is the only independently measured outcome among the ten, and it measures a failure. No success in this file has a T1 result behind it.

Used in. How to Use This Book (the T1 example); When Engagements Fail (the slow fade).

Morgan Stanley wealth management

What happened. Morgan Stanley built an assistant for its wealth-management advisors with OpenAI. Colin Jarvis, who runs forward deployed engineering at OpenAI, says "the main technical hurdle was probably solved within like 6 to 8 weeks", and that "a further like 4 months of like doing pilots, collecting evals, and iterating" followed before the advisors trusted it (Altimeter interview, November 2025). OpenAI's enterprise guide describes the first three evals as translation, summarisation and "Human trainers", which compared the model's answers with expert advisors' (OpenAI, AI in the Enterprise). The firm's David Wu says the first question they get is whether "our information" will be used "to train the public ChatGPT", and that the OpenAI team's "willingness to ensure zero data retention has been really impactful" (OpenAI customer story).

Tier. T2: a vendor describing its own customer, in an interview, a guide and a customer story. ZenML's summary of the same talk is T3 and an echo of it, not a second source (ZenML LLMOps database).

What it proves. It is the best public account of the gap between a working build and a trusted one: the trust period ran two to three times as long as the build, and it was spent with users, not on the model. It also shows the evals starting early and with the customer's experts, a first win picked on one of the bank's largest businesses rather than a side workflow, and a regulated firm asking about training and retention before it asks about quality.

What it does not prove. Nobody independent measured adoption. The headline adoption figure is an echo: Morgan Stanley's own release says "98% of Financial Advisor teams have adopted" (Morgan Stanley), and OpenAI's guide turns that into "98% of Morgan Stanley advisors use OpenAI every day". Teams that adopted a tool and individuals who use it daily are different measurements. The timeline is one vendor executive's recollection of one customer, and says nothing about how long trust takes elsewhere.

Used in. How to Use This Book (the 98% echo); Scoping and the First Win (high stakes); Security, Governance and Model Risk (the ZDR question, and the composite); Evals (start with evals, and the long climb); Production and Handoff (the trust period, and the composite); Feeding the Product; Judgment Under Pressure (non-determinism).

Klarna customer service

What happened. Klarna announced in February 2024 that its assistant handled "two-thirds of Klarna's customer service chats" in its first month, "the equivalent work of 700 full-time agents" (Klarna). By Jarvis's account, OpenAI's team wrapped each intent in its own evals, which let it scale from about 20 policies to about 400, and turned the method into a framework, Swarm, that was open-sourced and later fed OpenAI's agent SDK (Altimeter interview). About fifteen months after the launch, Klarna's CEO told Bloomberg the result had been "lower quality", and that "investing in the quality of human support is the way of the future for us" (Entrepreneur, May 2025).

Tier. T2 for the launch figures and the eval-per-intent method (self-reported by the customer and the vendor). T3 for the reversal (a magazine relaying one interview). It carries an echo: the Entrepreneur report gives the original share as 75%, not two-thirds.

What it proves. A per-intent regression suite is what lets an engagement grow its scope safely. Field work can become a product, and here it did so only after a second, harder customer (T-Mobile) used the framework with small changes. And the agent's share of the work is not evidence the work is done well: the success was declared on an automation metric and later reversed.

What it does not prove. Nothing public says whether the per-intent evals measured resolution or only containment, so the case cannot tell you whether better evals would have caught the quality problem. It is not, strictly, an FDE engagement: Jarvis himself calls it "a solution architect engagement but very much an FDE style of delivery", and Pragmatic Engineer says OpenAI's Solutions Architecture team created the framework (Pragmatic Engineer, August 2025). Flag that before you tell the story to a customer. When Engagements Fail does not count the reversal as an FDE engagement either.

Used in. Evals (eval per intent); Feeding the Product (the second-customer rule, and getting it adopted); When Engagements Fail (the reversal, and the composite).

OpenAI and a European semiconductor company

What happened. Jarvis describes an OpenAI engagement with an unnamed European semiconductor company whose request was deliberately open: look across the whole value chain and "take the biggest areas of… waste". The team "embedded on site, spent like a couple of weeks just like understanding their business in a lot of detail", across design, verification and post-ship performance measurement. It chose verification, where engineers "have to spend like 70 80% of their time" fixing bugs; tests run overnight, and each engineer arrives in the morning to a few hundred bugs. The first eval step was "making like a set of five with the experts from the customer" (Altimeter, 2025-11-26).

Tier. T2: the vendor describing its own engagement, with the customer unnamed. The "set of five" is quoted as doctrine and graded T4 where Evals uses it.

What it proves. It is the best public FDE example of discovery done on the floor: weeks on site before choosing where to work, and the choice made by where engineers' time was lost, not by what the model was good at. It also shows a big problem picked on purpose ("you got to pick a big a big kind of problem to solve") and an eval set that starts as a conversation with the customer's experts.

What it does not prove. The chapters report no outcome from it. The customer is unnamed, so nobody outside OpenAI can check the account, and it is one engagement told by the team that ran it. Do not confuse it with the semiconductor company in Stanford's study, whose agents "worked in demos but not in production" (see the runners-up).

Used in. Discovery (and the composite in The Insurer, On Site); Scoping and the First Win; Evals.

Palantir AIP bootcamps

What happened. In 2023 Palantir moved its commercial pilots to "bootcamps" that, on the Q3 2023 call, "deliver real workflows on actual customer data in five days or less versus our traditional pilots, which generally take one to three months"; Shyam Sankar called it "proof, not proofs of concept" (Q3 2023 call). By the next call it reported "more than 560 bootcamps across 465 organizations to-date" (Q4 2023 call). Later calls gave conversion examples: a healthcare company that converted five weeks after a December bootcamp to a five-year agreement worth $26 million in annual contract value (Q1 2025), and another that "completed two boot camps with us last summer and signed a $96 million deal" (Q4 2025).

Tier. T2: the vendor reporting to investors on its own earnings calls.

What it proves. A vendor can compress the pre-contract phase from months to days for some customers, and some of those customers sign large contracts quickly. The principle transfers even where the format does not: the customer's real data and a real workflow appear in the first week, and the event is judged on whether the workflow works.

What it does not prove. No conversion rate was disclosed; the Q4 2023 call talks about "converting bootcamps to enterprise deals" and gives no rate. The examples are the ones management chose to tell investors. They say nothing about whether the bootcamp workflows reached production, or how many FDE hours a conversion cost. Most vendors cannot run a five-day bootcamp at all.

Used in. Before the Engagement.

Palantir S-1 pilot economics

What happened. Palantir's 2020 S-1 filing says it acquires customers "through short-term pilot deployments of our software platforms at no or low cost to them", and that "During these short-term pilots, we operate the accounts at a loss", with sales cycles "often… six to nine months" (Palantir S-1). It sorts customers into Acquire, Expand and Scale phases. In 2019 Acquire customers brought in $0.6 million "which yielded a contribution loss of $65.4 million"; Expand ran at (43)%, Scale at 55%, the top quarter of Scale at 87%, and all customers at 21%. It names the role directly, saying FDEs "observe users' challenges firsthand".

Tier. T2: company-reported, in a regulatory filing.

What it proves. The losses came first and by design, and the margin turned on accounts where the deployment work stopped growing faster than the revenue. In the half-year after, the Expand accounts moved from (43)% to 35%. It is the oldest public record that a free pilot is a cost the vendor chooses to carry, which is the problem Before the Engagement is built against.

What it does not prove. The 87% is selected on the outcome: it is the best quarter of Scale customers ranked by margin. The phases are defined by the result, so the table cannot say how many customers move between them, and the filing gives no conversion rate ("there is no guarantee that we will be able to move customers from the Acquire phase into later phases"). Contribution margin leaves out research and development and overhead. Qureshi's later "80% gross margins in 2023" has no source, and its reappearance elsewhere is an echo.

Used in. How to Use This Book (the T2 example); Before the Engagement (and the composite); Origins and Economics (and the sceptics).

NHS Federated Data Platform

What happened. The Register reported that "fewer than a quarter (45) of England's 215 hospital trusts were actively using" the Palantir-built platform by the end of 2024, that "72 are already live as part of a phased rollout", and that "more than 120 NHS trusts had signed up", with £8.8 million going to KPMG for adoption support (The Register, May 2025). Palantir's Louis Mosley told MPs that writing back to NHS systems "is more challenging than reading from these systems, especially in systems that are very old", and that a manual write-back loses "a lot of the efficiency"; a trust's chief digital information officer agreed (The Register, July 2025).

Tier. T3: a journalist relaying NHS figures and testimony to MPs. The vendor has an interest in blaming the old systems.

What it proves. Adoption needs an agreed definition: signed up, live and actively using are three numbers for one programme, and each side of an argument can pick the one that suits it. Write-back is harder than read, and when the last step is a person keying a decision in, much of the gain is lost.

What it does not prove. The KPMG spend shows how seriously the buyer took adoption, not whether the spending worked. None of the figures is an independent measure of use, and the case cannot say whether the platform is worth what it cost. It is the book's one well-documented counter-example on handoff, not evidence of what makes one last.

Used in. How to Use This Book (the T3 example); Data Access and Integration (write-back, and the composite); Production and Handoff (adoption, and leaving on purpose).

MD Anderson and IBM Watson

What happened. MD Anderson's Oncology Expert Advisor, built with IBM Watson, cost "US$62 million" over more than four years. The University of Texas audit, as Network World reported it, found that after the hospital moved its records system to Epic in 2016, "OEA has not been updated to integrate with the current system", and that it was "not in clinical use". The hospital paid almost $40 million to IBM and over $21 million to PwC, "almost all of it without board approval", contractors "were paid regardless of whether the original contracted scope-of-work was completed", and the scope grew from one leukemia type to six, then to lung cancer (Network World, March 2017).

Tier. T3: press reporting of an internal audit the book has not read in the original.

What it proves. An integration nobody maintains decays: whatever the model could do, it could no longer read the hospital's records. It also shows governance failing with no help from the model: approvals and payments not tied to scope or evidence.

What it does not prove. It is old, it predates today's models, and it is not an FDE engagement. It shows the mechanism of decay and of loose approvals, not how common either is.

Used in. Data Access and Integration; Security, Governance and Model Risk.

McDonald's drive-through voice ordering

What happened. McDonald's tested voice ordering with IBM in about 100 locations. A 2022 analyst report put accuracy in "the low 80% range" against a "95%-plus" target, and the test ended in 2024, roughly two and a half years after the partnership began (Nation's Restaurant News, June 2024).

Tier. T3, and the accuracy figure is second-hand.

What it proves. A climb that never reaches the threshold is a failure in its own right. Nobody set a date by which it had to succeed, or said in advance what would happen if it did not.

What it does not prove. The accuracy figure is an analyst's, relayed by a trade paper, and the public record does not say why the test ended. It is not an FDE engagement, and it cannot tell you whether the target was reachable.

Used in. Evals; When Engagements Fail (an unreachable threshold).

Airbus A350 and Palantir

What happened. Nabeel Qureshi's first real Palantir engagement was Airbus: he "moved out to Toulouse for a year and worked in the factory alongside the manufacturing people four days a week". The software pulled "disparate sources of data — work orders, missing parts, quality issues" into one place, and he calls them "basic software things" (Qureshi, October 2024). Separately, Pragmatic Engineer reports that Anjor Kanekar, a Palantir FDE for seven years, "was on the final assembly line at aerospace manufacturer Airbus", with peers in environments that included airgapped ones (Pragmatic Engineer, August 2025).

Tier. T4 for Qureshi, a former employee's recollection. T3 for Kanekar, secondary reporting from interviews.

What it proves. What on-site work looks like at its most literal, and that the value came from basic software fitted to the real workflow, not from cleverness.

What it does not prove. Any result. Qureshi's "4x" for the A350 manufacturing pace has no method; do not repeat it as an outcome. Two accounts from Palantir alumni are not a measurement.

Used in. Building in Their Environment (and the least clever thing).

Runners-up

These cases each carry one point in one or two chapters. Most are T3 or a single team's account; none has an independently measured outcome. Use them for the point listed, and nothing wider.

What no public case shows

The gaps matter as much as the cases, because they mark where this book's advice is a recommendation rather than a finding.

• No T1 account of how an engagement runs. Independent, measured evidence on FDE engagements barely exists. Vendors publish their wins, customers publish their adoption numbers, and almost nobody outside either side measures anything. Every account of how discovery, scoping, evals or handoff actually went is a vendor's or a participant's (How to Use This Book). There is no independent study of FDE discovery itself (Discovery).
• The only T1 outcome is a failure. Of the ten cases above, the only one with an independently measured outcome is MyCity, and what was measured is that it gave wrong answers. Every success here is the vendor's or the customer's best account of itself.
• No first-hand account of an FDE killing an engagement. No public source describes an FDE deliberately ending a customer engagement and the criteria used; the research behind this book looked and found none. Failed engagements end quietly, under NDAs, so the walk-away advice in When Engagements Fail is built from adjacent evidence.
• No measure of handoff a year on. No public source measures adoption of an FDE-built system independently, or checks whether it is still running and used a year after the FDE leaves (Production and Handoff).

If you can publish any of these, with dates, numbers and who measured them, it would be worth more than another success story.