8 min read

B2
Operation · Economics & ROI

Agent Unit Economics

Cost per successful task is the right unit, and the number you compute for it is only as trustworthy as the thing that decides what counted as a success.

A unit-economics figure with no error bar on it is a marketing number, and the error comes from your success oracle rather than from your bill. Cost per successful task is still the only unit worth putting on a dashboard — but a judge that passes 8% of failures quietly moves it by more than any token optimisation you were considering, and the sampling interval around a hand-labelled success rate is usually wider than the margin being claimed on top of it.

Step 1The bill got six times cheaper and almost nobody's cost per outcome moved.

Teams instrument cost per token and cost per API call because those fall out of the invoice. Both are measures of effort, not of value delivered: a token is not a deliverable, a completed task is. The industry name for the unit that matters is cost per successful task — the total cost of every attempt divided by the count of attempts that actually produced the outcome.

The gap between the two is not academic, and the last three years prove it. Frontier input tokens now cost roughly a sixth of GPT-4's 2023 list price, and a cheap capable model a small fraction of that again, yet very few agent deployments got six times cheaper, because an agentic workflow burns five to thirty times the tokens of a single chat completion. Anthropic's own published figure for its multi-agent research system, written up in June 2025, is about 15× the tokens of a chat interaction, with an ordinary single agent around 4×. Deflation landed on a unit nobody buys. Price deflation and cost per task has the full version of that argument; what matters here is the consequence: the only line that stayed meaningful across the whole price collapse was cost per completed outcome.

Step 2Write the unit out and the leverage is obvious — it is all in the denominator.

An agent that costs $0.40 a run looks cheap until you learn it succeeds 60% of the time. The customer pays for successes; you paid for every attempt. Put the retry tax in as well and the shape is this:

# cost per successful task — the denominator does the work
attempts        = 1000
cost_per_run    = 0.40          # you pay this every attempt, win or lose
success_rate    = 0.70
retries_per_win = 1.3           # failed tries before a win still bill

cost_per_win = (cost_per_run * retries_per_win) / success_rate
# = 0.52 / 0.70 = $0.74  — not the $0.40 on the dashboard

Raising the success rate from 70% to 85% cuts cost per win by about 18% with no change to model price at all, and it improves the product at the same time. That is a far larger move than almost any token optimisation, and it is why fewer tokens versus cheaper tokens is a second-order question: both levers act on the numerator, and the numerator is the smaller term.

Before optimising the model, optimise the denominator. A reliability fix compounds three ways — it lowers cost per win, raises delivered value, and shrinks the retry tax. Token tuning touches one term.

Step 3The denominator is a property of your verifier, not of your agent.

Here is the part that is almost never instrumented. You do not observe the success rate. You observe a verifier's opinion of the success rate, and that verifier has its own error rate which propagates straight into the unit.

Suppose 1,000 attempts, a measured pass count of 700, and the pass verdict comes from a model judge. Judges on agent trajectories are not close to perfect; a false-positive rate in the high single digits is an ordinary finding rather than a pessimistic one, because a plausible-looking trajectory with a wrong result is exactly the case a judge is worst at. Carry an 8% false-positive rate through:

# a judge's false positives inflate the denominator you divide by
measured_passes = 700           # what the judge said
judge_fpr       = 0.08          # share of true failures it passes anyway
true_failures   = 300

inflated   = true_failures * judge_fpr          # = 24 silent failures counted as wins
real_passes = measured_passes - inflated        # = 676, i.e. 67.6% not 70%

# the unit moves with it: 0.52 / 0.676 = $0.77, against the $0.74 you reported

Three points of measured success rate is a 4% error in the unit — larger than most of the token savings anyone ships in a quarter, and in the wrong direction. Worse, it is a systematic error: it does not average out across months, and it grows precisely when the agent is pushed into harder work, because that is when the judge's false positives rise. If the same model that did the work also grades it, you are measuring confidence rather than correctness, and the denominator is inflated by exactly the failures nobody saw. LLM-as-judge for agents is how you calibrate it; where the economics breaks calls the resulting line the silent-failure tax.

So the prerequisite for the unit existing at all is an objective, automatable verdict a customer would agree with — a resolved ticket, a merged pull request, an approved invoice — plus a measured error rate for whatever produces that verdict. Without the first you have a cost number with no unit. Without the second you have a unit with no precision.

Step 4Quote the unit as a band, and size the sample that earns the band.

Even a perfect human verdict is sampled. If you hand-label 300 attempts and 210 pass, the point estimate is 70% and the 95% interval is roughly ±5 points — the normal approximation gives 1.96 × sqrt(0.7 × 0.3 / 300) ≈ 0.052. Push that through the unit and $0.74 becomes a range of about $0.69 to $0.80, a 15% spread. If your claimed gross margin is thinner than that spread, you do not yet know whether the product makes money.

Work the sample size backwards from the precision you need instead of picking a round number. To know a 70% success rate to within ±2 points at 95% confidence costs 1.96² × 0.7 × 0.3 / 0.02² ≈ 2,017 labels; ±1 point costs four times that, because the cost of precision is quadratic. Decide what you can afford to know before you promise a figure you cannot defend. The cost of evaluation prices that labelling work, and why agent eval is hard explains why the per-label cost is higher for a trajectory than for a classification.

Report three numbers together or none of them: the point estimate, the interval, and the verifier that produced the verdicts. A CPST of "$0.74" and a CPST of "$0.74 ± $0.06, judged by a model with an 8% measured false-positive rate on 300 human-labelled trajectories" are the same arithmetic and completely different claims. Only the second one can be audited, and only the second one survives a board meeting where somebody asks how you know.

Step 5Load the cost side fully, including the bill nobody computes.

Margin is price minus the fully-loaded cost per win, not price minus model spend. Four lines sit between the two, and the usual mistake is to treat the first as the whole list:

  • Human escalation. If 8% of tasks fall back to a person at $6 of loaded labour, that is about $0.48 on the average win — frequently larger than the model line. The cost of human review makes the sharper point: it is the one line that does not shrink as the agent improves, because a reviewer has to read the correct outputs too.
  • Eval and observability. The judge calls, the traces, the sampled labels from STEP 4. This is real per-task cost that exists because you decided to know your own numbers, and it does not become overhead by being inconvenient.
  • Errors that escaped. The refund, the rework, the lost customer. The cost of being wrong argues this line runs one to two orders of magnitude above the token bill on most deployments; it is also the line that STEP 3's silent failures feed directly.
  • The standing bill. Index, eval suite, trace retention, capacity floor, on-call. Fixed costs and the pilot tax covers why dividing it by a pilot's task count produces a number that describes nothing — and why breakeven volume is the honest figure at low volume.

Price off the loaded number. A product priced against raw model cost has negative margin the first time the escalation rate or the tail moves, and both move.

Step 6The distribution decides survival, and a metered unit can move underneath you.

Agent cost per task is heavily right-skewed: most tasks are cheap and a few burn 50–100× the median through long loops, deep fan-out and retry storms. The mean can look healthy while the 95th-percentile task eats the month. Underwrite against the p95 cost month rather than the average — a business profitable only at the mean loses money exactly when it is busiest. Forecasting agent spend is the arithmetic for doing that properly.

One newer wrinkle changes the arithmetic rather than the amount. When you buy an agent priced in vendor credits, your cost per successful task is denominated in a unit with no fixed exchange rate to money. Notion's October 2026 packaging gives Business and Enterprise seats 1,500 routed credits plus 500 user-choice credits a month, with no rollover and no published conversion, while the vendor's own router decides how many credits a task consumes; GitHub went the other way on 1 June 2026 and pegged one AI Credit to one cent. In the unpegged case you need two units — credits per successful task to manage the quota, and currency per successful task to manage the business — and only the first is observable from inside the product. Agents priced in credits covers what to ask for in writing.

And keep one use for cost per token: it is the correct lens for comparing two implementations of the same task at the same measured success rate, where cheaper tokens are pure margin. The error is promoting it to the headline business metric. A cheaper token on a task that fails is a faster way to lose money.

Do this in order. Define success as a verdict a customer would agree with. Measure the verifier's false-positive rate against human labels before you quote anything. Hand-label enough attempts for the precision you need — about 2,000 for ±2 points — and publish the unit as a band with the verifier named. Load escalation, eval, escaped errors and the standing bill into the numerator. Underwrite at p95, not the mean. If you keep one sentence: the token bill is measured to four decimal places and the denominator is a guess, so spend your next instrumentation hour on the denominator.