Scoping and the First Win

What this phase is for, and how it fails

Discovery, the previous chapter, ends with an observed problem: the thing that actually costs the customer time or money, which is rarely the thing they asked for. Scoping turns that problem into something you can win. That means a workflow small enough to build in weeks, a number that says whether you built it, and the signatures of the people whose judgment of "better" counts.

Scoping fails in two directions, and each looks like a reasonable choice when you make it.

The safe target. The team picks a workflow nobody minds failing: an internal FAQ, a side process, a department with no budget. It ships, and nobody notices. Colin Jarvis, OpenAI's head of forward deployed engineering, describes the opposite pattern in the customers who succeed: "they don't do like a kind of edge case like over to the side of the company. They pick something with genuinely high stakes" (Altimeter interview, November 2025; T2, a vendor describing its own customers). His example is Morgan Stanley, whose wealth-management practice is one of the bank's largest businesses. A win on a side workflow proves only that the model works on a side workflow, and nobody has to believe it.

The unagreed number. The team picks a real target but never fixes what success means. The demo goes well, everyone says it looks promising, and then the debate over whether it is good enough runs for months. RAND's practitioner interviews, which Discovery covered, put this at the root of the most common failure. Stakeholders "misunderstand—or miscommunicate—what problem needs to be solved using AI", and models end up "optimized for the wrong metrics" (RAND, 2024; T1, though it rests on interviews, not on project outcomes). In scoping the error looks harmless: a model tuned to a number nobody agreed, which becomes obvious only after it is delivered.

Those two failures pull against each other. A target with high stakes tends to be large, and a large target does not fit in a pilot. Scoping is how you hold both: keep the stakes and shrink the surface. Pick the workflow the sponsor cares about, then cut it to one slice of it, with one number and one date.

The composite: one line of business, one number

Three things in that beat carry the chapter. The target sits in the core business but is cut to one line of it. The number is two-dimensional and measured before anyone builds. The person who sets the threshold is the person whose team pays for a wrong answer. The rest of this chapter shows how to get there.

High stakes, small scope

The best evidence that small starts work comes with a catch. Stanford's Digital Economy Lab studied 51 enterprise AI deployments that delivered measurable value, and found that "73% of implementations started small deliberately, and 63% framed their pilots explicitly as experiments" (Pereira, Graylin and Brynjolfsson, April 2026; T2, and the copy online is marked as a draft). The catch is in the title: Lessons from 51 Successful Deployments. The sample contains only successes, so it cannot tell you how many failed projects also started small. Read it as "successful teams usually started small", not "starting small makes you succeed".

The authors' explanation is more useful than the percentages. They call controlled scope a "failure strategy": "Small pilots fail cheaply. Cheap failures do not end careers." In one of their cases, a professional-services firm with two failed technology projects behind it moved forward with a sponsor who "accepted 80% accuracy as good enough to move forward". A small scope makes an honest threshold politically possible.

MIT's NANDA report points the same way from the vendor side. Its takeaway is that startups that succeed "land small, visible wins in narrow workflows, then expand" (MIT NANDA, July 2025; T2). The method limits how far that goes. The report rests on interviews with 52 organisations, surveys of 153 leaders and a review of 300+ public initiatives, and it calls its own figures "directionally accurate", based on interviews rather than official company reporting.

None of this argues for a safe scope. Small and visible is not the same as small and peripheral. Jarvis makes the point again with OpenAI's European semiconductor engagement: "you got to pick a big a big kind of problem to solve" (Altimeter interview, T2). The way to hold both is to cut along the business, not away from it:

• One line of business out of several (auto, not property).
• One step of a long workflow (triage, not settlement).
• One region, one team, or one share of traffic, with the rest kept on the old process.
• Propose, don't act: the agent suggests and a person decides, which shrinks the risk without shrinking the stakes.

There is a test for whether you cut too far. Ask the sponsor what they would do if the pilot hit its threshold. If the answer is "roll it out to the rest of the claims operation", the scope is a slice of something that matters. If the answer is a shrug, you have picked the side workflow.

One more piece of evidence sets the frame. RAND advises leaders to "commit each product team to solving a specific problem for at least a year. If an AI project is not worthy of such a long-term commitment, it most likely is not worth committing to at all" (RAND, T1). A 30-day pilot does not contradict that. The first win is the opening of a year-long commitment to a problem, not the whole of it. If nobody at the customer is prepared to stay on this problem for a year, a successful pilot will have nowhere to go.

The metric before the build

Write the success criterion before the first line of agent code. Once a prototype exists, every conversation about the threshold turns into a conversation about the prototype, and the number drifts toward whatever it already does.

Anthropic's guide to defining success criteria gives a shape worth copying exactly (Claude docs; T4, vendor guidance). Its example of a well-defined criterion reads: "The sentiment analysis model should achieve an F1 score of at least 0.85 … on a held-out test set of 10,000 diverse Twitter posts … which is a 5% improvement over the current baseline." That one sentence holds four parts, and a scope needs all four:

1. A metric that the business recognises, not just the model team: routing accuracy, minutes saved, cases closed.
2. A threshold, written as a number or a rule with no room for interpretation.
3. A held-out set that the threshold is measured on, which the builders never tune against.
4. A baseline that the improvement is measured from (section 6.5).

The same guide adds that "most use cases need multidimensional evaluation along several success criteria." For an FDE this matters more than for a product team, because a single accuracy number hides the errors the customer fears most. Pair the main metric with a speed or cost metric, and with at least one guardrail: a class of error that must never happen, such as the composite's litigation-flagged claim reaching the fast queue. A guardrail is usually what the customer's risk function will ask about first, and setting it now saves an argument later.

Who sets the threshold. Not the FDE, and often not the sponsor. The threshold belongs to the person whose team absorbs the errors: the claims director, the head of the contact centre, the clinical lead. The sponsor signs that the metric matters; the operational owner signs that the threshold is good enough to use. If those are the same person, good. If nobody will own the threshold, you have found the workflow with no owner, and that is a reason to change the target, not to proceed.

Who owns the held-out set. The customer, always. Their experts label it, they keep it, and the vendor sees it only when the threshold is measured. That sounds bureaucratic, but it protects you as well as them. When the vendor builds both the agent and the test, a pass is a vendor claim. When the customer owns the test, a pass is the customer's own finding, and it is much harder to argue with at the conversion meeting. Seed it from the baseline sample in week 1, so that the same documents measure the old process and the new one.

Translate the threshold into the sponsor's words. "F1 of 0.85" means nothing to a COO. "At least as accurate as your senior adjusters, and same-day first action" does. Write both versions in the scope: the precise one for the eval, the plain one for the sponsor. If they ever disagree, the plain one is what you will be held to.

This chapter sets the threshold; the evals chapter (Chapter 9) builds the harness that measures it and grows the held-out set into an eval suite. The wiki's Evals 101 covers the vocabulary, and eval-driven agent development covers how an eval becomes the spec an agent is built against.

Measure the baseline in week 1

No baseline means no win. Without a measured "before", a good result can always be called what the old process would have done anyway, and a bad one can be blamed on a bad week. Alpine Edge's four-week template for a forward deployed engagement puts the baseline first: "Measure the baseline before any AI touches it: cycle time, rework, cost per task" (Alpine Edge, August 2026; T4, one consultancy's practice). Its four weeks are worth knowing as a shape:

• Week 1: "Discover and baseline".
• Week 2: "Build one complete workflow".
• Week 3: "Harden and pilot".
• Week 4: "Assess production readiness and results".

The same post lists the warning sign for this phase: "A prototype exists but no production-like data access by end of week one." Treat that as an alarm, not a delay. The data-access chapter that follows (Chapter 7) covers what to do about it; for scoping, the point is that a baseline measured on data you cannot reach is not a baseline.

A baseline does not need a data platform. It needs a sample, a stopwatch and a labeller:

1. Sample. Pull a few hundred recent cases from the target workflow, spread across weeks and teams so that one bad Monday does not set the number. Write down how you drew them.
2. Time. For speed metrics, take timestamps the systems already hold (received, assigned, first action, closed). Where none exist, shadow the people doing the work for a day and time it, and say in the scope that you did.
3. Label. For accuracy metrics, have two of the customer's experts label the same cases independently. Where they disagree tells you two things: which cases are genuinely ambiguous, and the ceiling. No agent will be judged better than two experts agree with each other.
4. Keep the sample. These labelled cases become the first version of the held-out set.

The baseline is also your first discovery check. If the measured number is far better or worse than everyone said in discovery, find out why before you set the threshold. Either the process is not what you were told, or the metric is not measuring what you think it is. Both are cheaper to learn in week 1 than in week 4. The SDLC book's chapter on the 90-day pilot makes the same argument for engineering teams: decide the comparison before the run, not after.

The "not now" list

Scope grows in every meeting. A stakeholder who was not in discovery asks whether it could also handle property claims. The sponsor's peer wants the same thing for underwriting. Someone points out that a chatbot was in the original proposal. Each request is reasonable, and each one, accepted, moves the first win further away.

Refusing does not work, because a refusal looks like a vendor protecting its margin. Writing it down does. Alpine Edge's mitigation for scope creep is one line: "One workflow until the core metric moves; keep an explicit 'not now' list" (T4). The list does three jobs:

• It tells the person asking that the request was heard and has a place to go.
• It turns "no" into "after the threshold is met", which gives the stakeholder a reason to want the first win.
• It becomes the expansion plan. When the pilot succeeds, the "not now" list is the first draft of what comes next.

Write each item with a reason and a trigger: "Property claims: documents too varied for a 30-day pilot; revisit once auto meets threshold." An item with no trigger is a promise you have not decided to keep.

The list is not frozen. Jarvis describes OpenAI's approach to ambiguous engagements as "bias for moving fast, proving out any brick walls and then adjusting the scope to the most useful thing we can do" (Pragmatic Engineer, August 2025; T3, a secondary quote). Scope changes when you hit a brick wall, such as data that does not exist or a system that cannot be written to. It does not change because a new stakeholder came to the meeting. When it does change, change the one-page scope, the threshold and the signatures together, and record what moved onto or off the list and why.

Employers describe this trade-off as the core of the job. Their postings are primary evidence of what they ask for, though not of how the work is done (box below).

The first demo

The first demo is a communication tool before it is a technical milestone. Its job is to make the scope real to the people who will use the result, early enough that they can still change it. Four rules make it work.

Show the real workflow on real or redacted data, never a slide. A deck asks the audience to imagine the product; a demo on their own documents lets them judge it. Synthetic data is fine for building, but a demo on synthetic data alone tells the customer's experts nothing they can trust, because they cannot see their own edge cases in it. If access is still pending, ask the data owner for a small redacted export. It is a much smaller request than a replica, and it is usually granted faster.

Demo what the user does next, not what the model can do. The adjuster does not care that the model extracted every field on the form. They care that the claim arrived in their queue, already sorted, with the police report on top. Open the demo in the screen they work in, or the closest mock you can build, and walk through their next action.

Say out loud what isn't done. Name the "not now" items, the cases the prototype gets wrong, and the fact that nothing has been measured against the threshold yet. Alpine Edge lists a second warning sign, which it calls the proof-of-concept trap: "Demo excitement, no baseline" (T4). A demo that runs ahead of the measurement creates enthusiasm the numbers then have to live up to. Saying what is missing keeps the threshold, not the applause, as the bar.

Pick the audience for the decision you need. The first demo goes to the threshold owner and two or three of the people who will use it, not to the executive committee. You want them to correct you. The executive demo comes later, with a number attached.

"The exec wants a demo on Friday" is a common pressure in week 1. Agree to it, but on your terms: real documents, the user's next step, and a slide-free statement of what is and is not measured yet. If you cannot show anything real by Friday, show the baseline instead. A measured "before" is a stronger thing to put in front of an executive than an unmeasured "after".

After the demo, the first exposure to real use should also be small. WeightWatchers, working with Sierra, planned to test its customer-service agent on "a small percentage of traffic", "a hundred or so inbound inquiries at a time" and "a small set of use cases" before deciding whether to widen it (Sierra customer story; T2, vendor-published, and the page does not say who measured its results). The team then "quickly expanded the breadth of use cases". The page reports that the agent contained "nearly 70% of all cases within the first week". Treat that as the vendor's account of its own best customer; the useful part is the shape, which is a small, bounded slice of real traffic first and expansion only after it holds.

The template: the one-page scope

One page, because nobody signs a longer one carefully. It carries the observed problem from discovery, the cut, the number and the names. Keep it in a place the customer can edit, and re-sign it whenever the threshold or the scope changes.

ONE-PAGE SCOPE                                   version / date: ____

1. PROBLEM (observed, from discovery)
   Stated problem:   ______________________________________________
   Observed problem: ______________________________________________
   Business lever it moves: _______________________________________

2. IN SCOPE (one workflow)
   Workflow / step:  ______________________________________________
   Slice:            line of business / region / team / % traffic
   Agent authority:  propose only | act with approval | act
   Workflow owner:   ______________________________________________

3. NOT NOW (each with a reason and a trigger)
   - item: ____________  reason: ____________  revisit when: ______
   - item: ____________  reason: ____________  revisit when: ______

4. SUCCESS METRICS
   Metric        Baseline (how measured, when)   Threshold   Owner
   main:         ______________________________  _________   _____
   speed/cost:   ______________________________  _________   _____
   guardrail:    ______________________________  never       _____
   Threshold in the sponsor's words: _____________________________

5. HELD-OUT SET
   Source and size: _______________   Labelled by: ________________
   Owner (customer): ______________   Vendor tunes on it: NO

6. FIRST DEMO
   Date: ________   Audience: threshold owner + users: ____________
   Data: real / redacted (never synthetic only)

7. WEEK-BY-WEEK PLAN
   Week 1: __________________________________________________________
   Week 2: __________________________________________________________
   Week 3: __________________________________________________________
   Week 4: __________________________________________________________

8. TOP RISKS                      Earliest signal           By when
   - ____________________________  _______________________  _______
   - ____________________________  _______________________  _______

9. SIGN-OFF
   Sponsor (metric matters):        __________   date: ______
   Threshold owner (good enough):   __________   date: ______
   Vendor lead (we can build it):   __________   date: ______

Here it is filled from the composite. Every figure is a range, as the composite rules require; a real scope would carry the measured numbers.

ONE-PAGE SCOPE: auto FNOL triage                 version 2 / week 2

1. PROBLEM (observed, from discovery)
   Stated problem:   "a customer chatbot"
   Observed problem: adjusters spend roughly a third to a half of
                     each day triaging incoming claim documents
   Business lever it moves: claims cycle time (the COO's KPI)

2. IN SCOPE (one workflow)
   Workflow / step:  first-notice-of-loss document triage
   Slice:            AUTO claims only, all regions
   Agent authority:  propose only (suggestion in the adjuster queue)
   Workflow owner:   claims director

3. NOT NOW (each with a reason and a trigger)
   - item: property claims   reason: documents too varied for 30 days
                             revisit when: auto meets threshold
   - item: customer chatbot  reason: not the observed problem
                             revisit when: triage is in production
   - item: write-back to claims system  reason: needs replica + review
                             revisit when: read access is stable

4. SUCCESS METRICS
   Metric        Baseline (how measured, when)   Threshold   Owner
   main:         routing accuracy; few hundred   >= senior   claims
                 week-1 claims, 2 seniors label  adjusters   director
   speed:        time to first action; system    same        claims
                 timestamps, week 1: 1-3 days    work day    director
   guardrail:    litigation-flagged claim sent   never       claims
                 to the fast queue                           director
   Threshold in the sponsor's words: "at least as accurate as my
   senior adjusters, and nobody waits a day for a first look"

5. HELD-OUT SET
   Source and size: week-1 sample      Labelled by: 2 senior adjusters
   Owner (customer): claims operations  Vendor tunes on it: NO

6. FIRST DEMO
   Date: end of week 2   Audience: claims director + 2 senior adjusters
   Data: redacted export of real auto claims

7. WEEK-BY-WEEK PLAN
   Week 1: discovery; timestamp baseline; request replica access
   Week 2: discovery sign-off; label held-out set; demo on
           redacted real claims
   Week 3: end-to-end workflow on sanitised fixtures; shadow-run on
           live auto claims once read access lands
   Week 4: measure against threshold on held-out set; readiness review

8. TOP RISKS                      Earliest signal           By when
   - replica access late          no read access            day 5
   - threshold owner disengages   misses the demo           week 2
   - model risk blocks pilot      no review date booked     week 2

9. SIGN-OFF
   Sponsor (metric matters):        COO                 date: week 2
   Threshold owner (good enough):   claims director     date: week 2
   Vendor lead (we can build it):   FDE lead            date: week 2

Two things in the filled version are easy to miss. The guardrail has no baseline target, only "never", and that is correct: some errors are not traded against speed. And the risks have dates. A risk with no earliest signal is a worry; a risk with a signal and a date is something you can act on in time.

Failure signals

Each of these shows up early, usually in the first two weeks, and each points to one fix.

Ready to move on

Move into data access and the build when every box is ticked:

• The metric, its baseline and its threshold are written down and signed by the sponsor and the threshold owner.
• The held-out set exists, was labelled by the customer's experts, and has a named owner on the customer side.
• The "not now" list has been agreed, with a reason and a trigger for each item.
• The first demo is scheduled with real users, on real or redacted data.
• A week-by-week plan exists, and each top risk has an earliest signal and a date.