Failure Modes

Appendices

A failing engagement shows it weeks ahead, in symptoms most people file as scheduling problems; find yours here for its likely cause and one thing to do this week. The catalogue gathers every chapter's failure signals, removes the duplicates, and sorts them by phase, from the pilot terms to the career. Each row links to the chapter section that argues the fix, and each phase ends with the public case that shows it, where one exists, graded in Appendix A.

C.1

Using the catalogue

Find the phase you are in, then read its grid for the symptom you can see. Most symptoms belong to one phase, but several recur in later ones: a missing data path, an unowned threshold, an exit with no date. Each of those is listed once, where it is cheapest to fix, and its "What to do" cell links to the other chapters that raise it. The engagement as a whole (C.11) collects the symptoms that can appear in any week.

If you have five minutes, not fifty, ask the five questions from Five failures. Each tests one of the five recurring failures, and each turns a feeling into a yes or a no:

The wrong problem (C.3): "If this reaches the threshold, which number in your next operating review moves?"
No data (C.5): "Who must approve this access, and has each of them seen the request?"
No owner (C.9): "When this breaks after we leave, whose phone rings, and do they know it yet?"
A moving sponsor (C.11), asked by the lead, not by email: "Is this KPI still the one you are judged on this half? If not, which one is?"
An unagreed or unreachable threshold (C.4, C.7): "If we are still below the threshold on that date, what do we do?"

One symptom is noise. Two inside any four-week window are a pattern, and the rule from When Engagements Fail is to schedule the post-mortem then, on the second signal, not the fifth. That is the book's rule of thumb, not a measured trigger. The post-mortem is checked against the signed terms and the five conditions in the rescue-or-walk rule, not against impressions of commitment. Its template is in Chapter 13.

None of this is about an agent failing at runtime. A wrong answer, a loop or a tool error on a live system is an incident, not an engagement failure, and the wiki's page on failure taxonomies and triage covers how to classify and route those. The two connect at one point: runtime failures that recur in the same category become an eval problem (C.7), and if they recur across customers, a product problem (C.10).

The evidence under this catalogue is thin, and the chapters say so. There is no public, first-hand account of an FDE ending an engagement and describing the criteria used. The symptoms are practitioner and vendor testimony (mostly T2 to T4), checked against RAND's interviews (T1, though it measures what practitioners believe, not outcomes) and Stanford's study of successful deployments (T2, a draft). The public cases named under each grid are graded in their chapters.

C.2

Before the engagement

These are visible before the first day on site. One is a reason to reopen the terms. Two or more mean the pilot is, in effect, an unfunded proof of concept, whatever the contract calls it.

No named sponsor, or the sponsor is the evaluating team

Nobody on the customer side loses anything if it fails, so nobody will sign if it succeeds.

Ask whose operating number this moves, and ask for that person. See Terms that make a pilot real.

The KPI is "explore what AI can do"

Success cannot be shown, so the pilot will end in "interesting, let's try a few more things".

Propose one business number the sponsor already reports, and its baseline source. See Terms that make a pilot real.

Open-ended duration

The pilot drifts until the sponsor's attention moves on.

Fix 30 days, with 90 as the ceiling, and write the exit criteria. See Proof, not proofs of concept.

The FDE first sees the terms after signature, or the account arrives already staffed by a sales promise

The terms were set by someone who can't judge feasibility, and sales is choosing the portfolio.

Fill in the engagement brief in kick-off week and send the blanks to the AE and the sponsor. A manager makes "learn" and "land" a gate before any FDE is named. See The FDE's job in pre-sales and Capacity is strategy.

Two symptoms from Chapter 4 are filed later, where the fix is done: no data access before week 2 (C.5) and a security review "after the pilot" (C.6). The case: Palantir's S-1 shows the model the labs inherited, pilots run at a loss and a sales cycle of "often… six to nine months" (T2, company-reported). IDC's finding that four of every 33 AI proofs of concept reached production, with "highly underfunded or not funded at all" as the diagnosis, is a correlation (T3). Both are in What this phase is for; the S-1 is graded in Appendix A.

C.3

Discovery

These are cheap to fix in week one. By week four each has turned into rework.

Discovery done by email or questionnaire

You have the documented process and the stakeholders' opinions, not the work.

Ask for floor time. If travel is impossible, screen-share a real shift, not a walkthrough. See Go where the work happens.

A model or vendor chosen in week one

The technology has been fixed before the problem: RAND's third root cause.

Park the choice explicitly until the summary box is signed. See Start with the business lever.

Only executives interviewed, and no workflow observed

You know the lever but not where it is lost, and the build will follow the org chart and miss the workarounds.

Shadow the people who do the work; watch at least one case end to end per stakeholder group, and show them your workflow drawing. See What to observe and what to ask.

One stakeholder's definition of better

The others will appear later as objections or as a blocked rollout.

Record each group's "better" and feared failure, and show the conflicts to the sponsor. See What to observe and what to ask.

A good demo, after which nobody asks for the next step

The wrong problem: the scope serves something the business has stopped caring about, or never did.

Ask: "If this reaches the threshold, which number in your next operating review moves?" A different number, or none, sends you back to discovery, not to the model. See Five failures.

The case: RAND's pricing example, where leaders ask for the price that sells the most items when they need the one with the greatest margin, and the error shows only at integration (T1, interviews). See What discovery is for.

C.4

Scoping and the first win

These usually show in the first two weeks. The first row recurs in later phases in new words; whenever it appears, the fix is the same.

The threshold is "as good as possible", "roughly", or "let's see what it can do first"

Nobody owns the decision that it is good enough. The bar is going missing, not being lowered.

Find the person whose team absorbs the errors and ask for a number in their words, in writing, by a date. If none comes, apply the rescue-or-walk rule. See The metric before the build and The four load-bearing commitments.

No baseline by week 2

The win cannot be shown, and the pilot will be judged on impressions.

Stop building and measure: a sample, timestamps, two labellers. A small baseline beats none. See Measure the baseline in week 1.

The target workflow has no owner

You have picked a side process, or a process nobody will change.

Change the target. A pilot with no operational owner has nowhere to land. See High stakes, small scope.

The sponsor shrugs at "what if it works?"

The scope is safe and the win will be ignored.

Cut along the business, not away from it: one slice of a workflow the sponsor cares about. See High stakes, small scope.

The scope has no "not now" list

Every request has been either silently accepted or silently dropped.

Write the list, with reasons and triggers, and show it at the next meeting. See The "not now" list.

The demo is on synthetic data only

The customer's experts cannot judge it, and access is not moving.

Ask for a redacted export of real cases, and treat a refusal as a data-access alarm (C.5). See The first demo.

The case: Jarvis's contrast between customers who pick "an edge case… over to the side of the company" and those who pick "something with genuinely high stakes", with Morgan Stanley as his example (T2, a vendor describing its own customers). See What scoping is for.

C.5

Data access and integration

These show in the first three weeks, while there is still time to act. The first is the commonest form of what Chapter 13 calls "no data": data that exists but cannot be reached in time.

No production-like data by the end of week one

The request is with the wrong person, or asks for too much, or nobody has said what a delay costs. The pilot is running on fixtures and hope.

Narrow the ask, name each approver's question, and invoke the agreed term: pause the clock or rescope. Ask: "Who must approve this access, and has each of them seen the request?" See The access request is a negotiation and Terms that make a pilot real.

No data owner named by the end of discovery, or the request sits with "a committee"

Nobody individual owns the decision, so access will start late and the first demo will be on fake data.

Name the owner and send the request before discovery ends. Ask the sponsor to name one approver per question: load, identity, personal data. See The access request is a negotiation.

The agent's errors cluster by region, source or document type

A semantics gap, not a model gap: a code, field or join means something the translation layer does not say.

Fix the glossary or field map with the customer's experts before touching the prompt or the model. See The translation layer.

The first real-data run breaks in ways the fixtures did not

Normal. The fixtures held only the cases you thought of.

Add the new cases to the fixtures and the glossary, and budget days, not hours, for the first real run. See Build on fixtures while you wait.

The customer asks for write-back in the pilot

They see the efficiency gain, not the approvals, rollback and risk review it brings.

Offer the side store and the existing queue view now; put write-back on "not now" with its four conditions. See Read first, write later.

Nobody will own the connector after handoff

It will break on the next upgrade of a source system, and the model will be blamed.

Treat it as a design problem now: find an owner, or simplify until the existing team can run it. See Integrations decay.

The case: Qureshi's account of Palantir pilots of "8-12 weeks" spent "just getting data access", with "the final week scrambling to have something to demo" (T4, one former employee's account). See What this phase is for.

C.6

Security, governance and model risk

These show in the first three weeks. Each is cheaper to fix in a diagram than in code.

"We'll do the security review after the pilot"

The pilot will succeed and then wait months while the sponsor's patience runs out, or the review will block a design that is already built.

Invoke the day-1 review term, and bring a data-flow diagram to the first meeting. See Run the review as a design session and Terms that make a pilot real.

"Will our data train your model?" answered with "no"

A reassurance, which the reviewer cannot check and will not accept.

Answer with the clause, the sub-processor list and each store's retention, logs included. See The first question is about the data.

The agent has write access "for later"

The worst case of any injection now includes whatever that access can do.

Remove it. Raise the authority level later, on eval evidence and a track record. See Indirect prompt injection.

Injection answered with "we have a filter"

Nobody has asked what a successful injection could do.

List every untrusted input with its worst case and the control that is not the model, and show seeded-injection results. See Indirect prompt injection.

The tool is classed as a "productivity tool"

Model risk has been postponed, not passed.

Ask model risk what evidence it needs, and plan the eval and audit log to produce it. See Model risk, audit and who is accountable.

Approval with no conditions written down

Nobody knows what production requires, so it will be renegotiated later.

Write each condition with its evidence and owner, and attach it to the scope. See Model risk, audit and who is accountable.

A workaround touches production data "just this once"

The data path the review signed is being bypassed.

Refuse on the spot; send a say-no note naming the approval it would void. See How to say no so it holds.

No public failure case. The chapter's evidence here is surveys of what is in the way and one study of what happened to survivors. Stanford's draft found security "was not a project killer" in any of its successful cases and called the tax "front-loaded" (T2), which a sample of successes cannot test. See What the review is for.

C.7

Evals

These show before the readout, and most of them during the long climb from the first quick number to the threshold.

The FDE wrote all the cases

A pass will be read as a vendor claim, and the experts have not agreed what "right" means.

Get five cases from the customer's experts, with their reasoning, before building further. See Start with five, from their experts.

The held-out score keeps rising week on week

Someone is tuning on it, so it no longer measures anything.

Freeze it, measure only on agreed dates, and ask the customer to label a fresh one if it has leaked. See Three sets, three jobs.

The score is stuck and nobody can say why

The failures have not been read or sorted, so every fix is a guess.

Read the failures, sort them into categories, and attack the largest one. See Grow it from real failures.

A category fails almost every time

A broken grader or wrong labels, as often as a weak agent.

Read ten transcripts and check the labels with the experts before reporting it. See Graders you can trust.

The threshold sits above expert agreement

It cannot be met, and the engagement will end on a technicality.

Show the threshold owner the agreement figure and let them restate the bar in writing. See The long climb from 80%.

The eval passes but users override the agent

The eval does not measure what the users care about.

Have the experts label a sample of overrides; add any missing dimension as a grader and re-sign the plan. See Evals: failure signals.

The sponsor hears "nearly there" in week 3

The 80% illusion; the conversion meeting will be judged against it.

Report the number against the threshold, the ceiling and the expected climb, every week. See Delivering the bad news.

Accuracy sits below threshold with no stop date

The climb has stopped being managed and become hoped for.

Agree the re-measure date and what happens if it misses: "If we are still below the threshold on that date, what do we do?" See Five failures.

The cases: McDonald's voice-ordering test with IBM, reported in "the low 80% range" against a "95%-plus" target and ended in 2024, with no date set by which the climb had to succeed (T3, the accuracy figure second-hand). LinkedIn's team reached 80% in a month and then spent four more months trying to pass 95%, and wrote that the early pace "created a false sense of 'almost there'" (T2). Both are in The long climb from 80%; McDonald's is graded in Appendix A.

C.8

Building in their environment

These show during the build, usually before anyone outside the team notices.

The design names a service their platform does not run

It was drawn for the vendor's environment, and the platform team has not seen it.

Get the seven constraint answers in writing and redraw from them before building further. See Inventory their constraints.

A "never" rule lives only in the prompt, or someone has guaranteed the model's behaviour

It will hold most of the time and fail in production on the case that matters. A guarantee sets an unmeetable threshold by default.

Move the rule into code before or after the model, with a unit test that must always pass. Restate any guarantee as rules in code plus a measured rate and a review path, in writing. See Rules outside the model and "Can you promise it will never…?"

A rule's trigger appears only in the documents

The model is being trusted to decide a "never" on its own.

Let the model move cases toward caution only; code refuses any output that moves one away. See Rules outside the model.

Rule results are reported as a percentage

Rule tests are mixed into the eval, so one serious failure hides among many harmless ones.

Separate them: the eval may be below 100%, the rule tests may not. See Rules outside the model.

Nobody can explain why a case was routed wrong

The design has more moving parts than the task needs, or versions are not logged.

Step down the ladder and log every version with every decision. See The least clever thing.

Only the FDE can deploy it

The system ships around their pipeline, not through it.

Push a change through their full path now, and let their engineer run the next one. See Build it so they can run it.

Behaviour changed and nobody changed the code

The model endpoint was upgraded underneath you.

Pin the version, ask for change notice, and rerun the eval and rule tests on every upgrade. See Building: failure signals.

Every complaint gets a same-day fix

Whack-a-mole: each patch makes the next failure likelier.

Log and categorise; act the same day only on broken rules or the four commitments. See Not every alarm needs action.

No public case. The chapter is one of the thinnest in the book: no measured study says which architectures succeed inside customer environments. See What building is for.

C.9

Production and handoff

Most of these show in the weeks around go-live, while there is still time to fix them before you leave. The first and the last are the two faces of Chapter 13's "no owner".

Every production change goes through the FDE, or nobody can name who runs it after you leave

The customer has a dependency, not a capability, and a working system is about to become an orphan.

Name the service owner, a person and not a team, before the next release, and have them make the next change with you watching. See The handoff drill and Five failures.

Nobody can say what "adoption" means

The number will be chosen later by whoever needs it to look good or bad.

Write the definition in user actions, baseline it from the first rollout stage, and give it an owner. See Adoption is a deliverable.

Suggestions are ignored for weeks

Users do not trust it yet, or it does not fit how they work.

Check where suggestions appear first; then read the overrides, fix the recurring ones, and show users which corrections landed. See Adoption is a deliverable.

Acceptance near total, review time near zero

The human check the risk review relied on has stopped happening.

Tell the behaviour owner; sample accepted items for review and show the misses. See Adoption is a deliverable.

Rollback has never been tried

The first bad release will become an incident instead of a redeploy.

Rehearse it in the drill, with the operators driving, before the next stage widens. See Ship in stages you can undo.

The sponsor gets only engineering metrics

The brief will be delegated, and the next decision will be made without you.

Put their KPI beside the leading metric, define adoption, and end with the decision you need. See Briefing the executive.

Someone wants headcount cuts on pilot metrics

A share-of-work number is being read as a quality number.

Say in writing what the metric does and does not measure, and cite the public reversals. See Five failures.

The support window has no end date: "Could your engineer just stay on?", or old customers still call you, not their team

The exit is dissolving. The vendor is becoming the operator, the engagement an unpriced services contract, and the FDE's on-call tail is growing.

Write the scope, channel and end date into the handoff document and get it signed; the lead, not the FDE, answers the request to stay, with the drill and its date. Route old calls through the dated window. See Leaving on purpose and The three loads.

The cases: Tryolabs' "common pattern" of a tool that, three months on, "sits there unused" (T4), and Alpine Edge's warning sign of handoff failure, "Only the engineer can operate it" (T4), are in What production is for. For how long trust takes: at Morgan Stanley the technical problem took weeks and the pilots and evals before advisors trusted it about four months more (T2). On headcount, Klarna's assistant was announced as doing "the equivalent work of 700 full-time agents" (T2, self-reported), and its CEO later said the result had been "lower quality" (T3). Commonwealth Bank of Australia reversed 45 cuts it had attributed to a voice bot, calling its assessment an "error" (T3). Both are in Five failures. Appendix A grades Morgan Stanley and Klarna.

C.10

Feeding the product

These show after go-live, usually when the FDE is already on the next engagement, and some only from the manager's seat. That is why they are easy to miss.

You are building the same thing for the third customer, or the same failure appears at several accounts, each fixed locally

Nothing came back from the first ones. A product gap is being paid for as field work, and the role has become services work.

Write one brief now, with every engagement as evidence and a named receiver. Name the next customer who will need it. See Feeding the product: failure signals and Failure signals for a manager.

The brief says "customers want…" with no numbers

No field log was kept, so the evidence is memory.

Start the log on the next engagement. For this one, rebuild what you can from eval results and tickets, and say it is reconstructed. See Keep a field log from week one.

Product wants the customer's data "to train on" or "as examples"

The line between pattern and customer has not been drawn.

Send failure descriptions, not records. Take anything else to the lead and the contract before it leaves. See Separate the pattern from the customer.

A platform is being designed from one engagement

Generalising too early: the first customer's quirks are about to become product.

Ship a template, and make the second customer the test of what is general. See Let the second customer decide.

Briefs go out and nothing comes back

There is no receiver with roadmap authority, or the briefs ask for too much.

Address a named person, ask for the smallest version, and set a decision date. Let usage make the case. See Get it adopted, not just heard.

No time is booked for the brief: no brief this quarter, or the next engagement starts the week this one ends

The next engagement absorbs it, the finding dies with your memory, and the loop exists only in the charter.

Book brief days into every engagement plan with the lead before kickoff, and ask for the gap between engagements in writing, naming what it is for. See Feeding the product: failure signals and The three loads.

The cases: the engineer who took an FDE job, found "a typical IT services mindset", and "quit 4 weeks later" (Pragmatic Engineer, T4, two anonymous engineers); and Jarvis naming "generalizing too early" as one of his team's biggest mistakes (T4). Both are in What feeding the product is for.

C.11

The engagement as a whole

Chapter 13 names five recurring failures, and none is a model failure: the wrong problem, no data, no owner, a moving sponsor, and an unagreed or unreachable threshold. Their symptoms are filed where each is cheapest to fix: the wrong problem in discovery (C.3), no data in C.5, no owner in C.9, the threshold in C.4 and C.7, and the moving sponsor here. It also names three ways the handling goes wrong: the slow fade, where nobody says the word; the late kill, after the money and the goodwill are gone; and the blame post-mortem, which costs the next attempt its sponsor. The symptoms below can show in any week, and most look like scheduling problems, which is why they are ignored.

The sponsor delegates or skips the steering meeting twice

Their priorities have moved, and nobody has said so.

The lead asks the sponsor, live: "Is this KPI still the one you are judged on this half? If not, which one is?" See Five failures.

Someone offers to "handle" the sponsor for you

The sponsor is about to stop hearing the number.

Keep the weekly note going to the sponsor directly, bad news included. See Signals you are negotiating the wrong thing.

Labelling or data work stalls on the customer side

The people the work depends on have been moved to something more urgent.

Name a restorer with authority over their time, and a date, or record the condition as missing. See Rescue or walk.

"Let's pause it for a quarter"

A walk, phrased so nobody has to decide.

Turn it into a decision: hand over, write the return trigger, stop billing on a date. See Walking away well.

An FDE is "fully booked" but nothing has moved for weeks

A stalled engagement is being counted as capacity in use.

Mark it as a signal in the weekly account review and apply the rescue-or-walk rule. See Failure signals for a manager.

Someone answers the trouble with a base rate: "95% of these fail anyway"

The rates in circulation are weaker than they look, and none can say whether this engagement is failing.

Answer with the engagement: "Nobody has a reliable failure rate for projects like this one. What we do have is our own five conditions, and here is where each stands." See Don't argue from base rates.

You argued for an hour about a demo date

Your standing is being spent on a negotiable.

Concede the date in one sentence; save the argument for the four commitments. See The four load-bearing commitments.

Every refusal you made this month protected margin

The no's have become commercial, and the customer can tell.

Apply the paid-hours test: would we refuse this if every hour were paid? Price the ones that fail it openly. See How to say no so it holds.

The cases: New York City's MyCity chatbot is the slow fade: testing found it would "confidently provide incorrect and potentially harmful information" (T1, the journalists' own measurement), and the evidence was public for nearly two years before anyone with authority called it (see What this phase is for, graded in Appendix A). Stanford's semiconductor company is "no owner": agents that "worked in demos but not in production", where "The problem was not technical" (T2, a draft). Its recruiting rescue shows a reset that restored the conditions rather than the model (T2). They are in Five failures and Rescue or walk. RAND's account of leaders who "rapidly switch their priorities every few weeks or months" is the moving sponsor (T1, interviews).

C.12

The career, and the team around it

The last failures are slower and more personal: a role that turns out to be services work, a level climbed by endurance, a team run on utilisation. The first five rows are for the FDE, the last three for the manager.

The job has no product loop and no say over the problem

It has dropped the stages that make it an FDE job; it is a contractor's job with the title.

Ask the five questions in your final interview or first month. If three or more answers point to services, judge it on services terms: pay, utilisation, travel and the exit path. See Read your employer's model and The sceptics are partly right.

Your promotion case is a customer count, or section 2 of your growth plan is empty

The evidence is L2 repeated, not a higher level; nothing is being handed to anyone, and the plan is to do more.

Rewrite the case around one thing that happened without you. Name one task you will stop doing and one person who will take it, before the next engagement. See Leverage, not endurance.

You are on site in the build weeks

Travel is covering for a broken data path or an absent sponsor.

Fix the cause (C.11), and cap travel by phase, not by percentage. See The three loads.

You can name only one exit

One domain is carrying the career; the others are at L1.

Pick the weakest domain's next level as next year's main goal. See What comes next.

The employer will not date a support window or leave a gap between engagements

It is probably selling your hours.

Ask for all three load caps in writing; if the answer to each is no, believe it, and confirm with the five questions. See The three loads.

One FDE carries two heavy phases

One of the two engagements will lose a commitment in whichever meeting comes second.

Move one, or delay its start. A delayed kickoff costs less than a conceded threshold. See Team shape and load.

Finance asks for utilisation or billable hours

The team is about to be run as services.

Offer the six measures instead, and agree in writing that an early call counts as a success. See Measure what the thesis measures.

The best FDEs are always on the stickiest accounts

A customer who never lets a strong engineer go has turned the vendor into its operator.

Rotate the FDE out on a planned date; if the account cannot survive it, the handoff never happened. See Failure signals for a manager.

The case: the same disappointed hires as in C.10, read from the engineer's side: such FDEs "don't contribute back into the platform" (T4, two anecdotes relayed by a newsletter). Two people are not a survey, but the ownership test predicts the complaint. See The sceptics are partly right.

Read this catalogue on a calm week, not a bad one. Pick the phase you are in, check its grid against the last four weeks, and write down any symptom you find with its date. One dated symptom is a note; two are the agenda for a post-mortem while the engagement can still be saved. Then write the rescue-or-walk rule into the terms before you need it, so the second symptom triggers a decision instead of another meeting.