Measuring Agent ROI
The share of firms reporting EBIT impact from AI did not move in a year while agent deployment nearly doubled. What is scarce is measurement, not value — and the baseline you need is destroyed the day you ship.
The most expensive thing you can leave out of an agent deployment is a number you could only have measured the week before it launched. McKinsey's State of AI in 2026, published 25 August 2026, found 89% of organisations using AI, 44% scaling it enterprise-wide, and 37% reporting a positive EBIT impact — the same 37% as a year earlier, while agent scaling at large firms went from 27% to 40%. A flat return number under a doubling deployment number is not a verdict on agents; it is a verdict on instrumentation, and the one measurement you cannot take retroactively is what would have happened without the agent. Build the counterfactual before you build the agent, because the day the agent goes live the baseline stops being observable.
Step 1The famous 95% is a measurement statistic that gets read as a failure statistic.
MIT's NANDA report The GenAI Divide found that roughly 95% of generative-AI pilots showed no measurable P&L impact, from 150 interviews, a 350-person employee survey and an analysis of 300 public deployments. It is widely quoted as "95% of AI projects fail". It does not say that. It says the impact was not measured — and its own authors note the finding partly reflects missing measurement data. The report was preliminary and not peer-reviewed, which is itself a reason to quote it for what it actually establishes.
That distinction is the whole discipline. A pilot with real value and no baseline is indistinguishable, in a finance review, from a pilot with no value. Both produce a usage report and a shrug. The asymmetry is brutal: value without measurement reliably reads as zero, while measurement without value reads as a negative number you can act on. Only one of those two failure modes teaches you anything.
If your ROI case is assembled after the agent is in production, you are not measuring ROI — you are reconstructing it, from data generated by a world the agent has already changed. That reconstruction is the artefact finance reviews kill, and it deserves to be killed.
Step 2The "agent replaces a human" framing is a category error.
An agent rarely replaces a whole role; it absorbs a slice of tasks and changes the shape of the rest. Pricing ROI as "headcount removed × salary" assumes the human disappears — but usually they move to reviewing the agent, handling its escalations, and doing the judgment work it cannot. The honest question is not "what human did this replace" but "what is the marginal value of the work now done that was not being done before, net of the human effort still required." Often the win is throughput or coverage that no headcount plan would have funded, not a salary line you deleted. The fractional time savings page is about why the minutes an agent gives back frequently do not aggregate into a billable hour.
Step 3No counterfactual baseline means no ROI, only a story.
ROI is value relative to what would have happened anyway. If tickets were already trending down, or an existing help article already deflected a third of the same intents, the agent only earns the delta over that baseline. The strongest baseline is a randomised holdout you keep running after launch; next is a pre/post with a control segment; weakest is a counterfactual estimate stated with its assumptions exposed.
# ROI is the delta over the counterfactual, net of new costs value_with_agent = outcomes_delivered * value_per_outcome value_counterfactual = baseline_outcomes * value_per_outcome # happens anyway incremental_value = value_with_agent - value_counterfactual roi = (incremental_value - loaded_agent_cost) / loaded_agent_cost # crediting the agent with value_with_agent alone is the lie
Keep a holdout even when it is uncomfortable. A 5% slice of traffic routed to the old path costs you 5% of the upside and buys you the only number in the model that is not an assertion. It is also the only instrument that survives the thing that kills pre/post comparisons: everything else about the business changing at the same time.
Step 4A worked quarter, with and without the baseline.
A support agent runs for one quarter. It closes 12,400 tickets with no reopen. The fully-loaded human handle cost for that ticket class is $6.20, so the headline value is 12,400 × $6.20 = $76,880. Now the cost side, fully loaded: inference $9,100; human review of 2,100 flagged outputs at 4 minutes each = 140 hours × $42 = $5,880; 1,860 escalations at $8.90 (the hard cases cost more than the average) = $16,554; and 0.25 of an engineer on evals, prompts and model-migration churn = $41,000. Total loaded cost $72,534.
# the same quarter, scored two ways naive = (76_880 - 72_534) / 72_534 # +6.0% -> "ship it" # the help article already deflected 31% of these intents last year baseline = 12_400 * 0.31 # 3,844 tickets, no agent needed incremental_value = (12_400 - 3_844) * 6.20 # $53,047 honest = (53_047 - 72_534) / 72_534 # -26.9% -> "not here, not yet"
Same agent, same quarter, same data: +6.0% or −26.9%, and the difference is one number — the deflection rate of the help article on those intents before the agent existed. Note what makes it unrecoverable. Once the agent is answering, the article's traffic and its deflection rate both change, so the baseline cannot be measured afterwards from production data; it can only be estimated, and it will be estimated by whoever wants the program to survive. The largest single term in the cost stack, meanwhile, is the engineer — which is why ROI models that count only tokens land in the wrong sign, and why cost attribution has to run on the provider's returned usage rather than your estimate of it.
Step 5Define value as a verified outcome, and time-to-value as the number that decides funding.
"The agent handled 40,000 requests" is activity — a number that goes up whether the work helped or not. Value must be a verified outcome tied to a metric the business already trusts: tickets resolved without reopen, qualified leads that converted, documents accepted downstream. If you cannot name the verified outcome, you do not have an ROI case, you have a usage report.
Then score the cumulative curve, not the asymptote. Agents ramp: low success and heavy review at launch, improving as evals tighten. Steady-state ROI can be excellent while the cumulative cash position is deeply negative for months, and a program judged on month-one numbers — or funded with less runway than its time-to-value — is cancelled before it earns out. Two defences: state the breakeven month in the funding ask, and report the curve rather than the current rate, so an improving negative reads as what it is.
- Pre-register the metric. Name the outcome, the baseline and the holdout before launch, in writing. A metric chosen after seeing the data is a metric chosen to flatter it.
- Re-measure on every model migration. A swap changes success rate, review load and cost per task at once; the old ROI number does not carry over, and unit economics invert rather than drifting.
- Separate the pilot's economics from the platform's. The first agent carries the whole platform build; the fifth carries almost none of it. Charging all of it to agent one kills the program that would have paid for agent five.
Step 6When the honest ROI number is "not yet, and maybe not here."
If the value is unverifiable, the baseline is unknowable, or the loaded human cost erases the gain, the honest answer is that this is not yet an ROI-positive use case — and saying so is cheaper than discovering it in a year. Roughly 6% of organisations in McKinsey's 2026 survey qualified as AI high performers, which should be read less as a warning than as a description of a field where most of the measured variance is still in execution and instrumentation. A faithful negative result redirects investment to a case that clears the bar, and scaling back is a legitimate outcome rather than an admission.
Before the next agent ships, write down three things and get them countersigned by whoever controls the budget: the verified outcome you will count, the baseline rate for that outcome measured now, and the share of traffic that will stay on the old path as a holdout. That page is worth more than any post-hoc model, because it is the only version of the number that was not chosen after the fact. An ROI model that cannot survive a hostile finance review is not conservative analysis, it is a pre-scheduled cancellation.