Energy & Carbon per Run

The per-prompt number is a boundary, and you have to state it.

Google's 2025 measurement of its own serving stack is the most carefully bounded public figure available, and it is the clearest illustration of the problem. The headline is a median Gemini Apps text prompt at roughly 0.24 Wh, plus about 0.26 mL of water. The same paper reports 0.10 Wh for the same prompt when you count only active accelerators — a number Google itself calls the optimistic one, because it drops host CPUs, idle provisioned capacity, and data-centre overhead.

• What the 0.24 Wh excludes is as important as what it includes: network transit, the user's device, model training, and data storage are all outside the boundary. None of those are zero, and two of them are yours.
• Independent estimates land an order of magnitude away. A third-party calculator put a 400-token prompt to a large reasoning-capable model at about 14.6 Wh. That is not evidence anyone lied; it is evidence that "energy per prompt" is under-specified until you name the model size, the output length, the utilisation assumption, and the overhead factor.
• Efficiency moves faster than your spreadsheet. Google reported roughly a 33× reduction in per-prompt energy over twelve months. Any figure you hard-code into a sustainability report is wrong within two quarters, in the favourable direction — which is an argument for publishing the method rather than the constant.
• Reasoning changes the arithmetic, not just the margin. Extended thinking adds output tokens that are billed and burned without appearing in the answer, and published estimates for reasoning-engaged queries run into the tens of watt-hours. Your effective default effort level is therefore an energy parameter, and most teams have never set it deliberately.

Your unit is the run, and the multiplier is the whole story.

Every public per-prompt figure was measured on a single-turn chatbot exchange. An agent finishing a task is a different object: a loop of model calls, each one re-sending a context that grew since the last, plus tool calls, plus retries, plus the sub-agents nobody counted.

# The only unit that means anything

  Wh per completed task
    = (model calls per run)
    x (tokens per call, input + output)
    x (Wh per token, with a stated boundary)
    / (task success rate)

# Why the first line dominates everything else

  chatbot turn        1 call      ~1x
  short agent task    8-15 calls  ~10x
  long agent task     40+ calls   ~40x
  failed run          same cost, divided by zero value
  retry storm         unbounded

# The denominator is not decoration
  a 70% success rate inflates the real per-task
  figure by 1.43x before anything else is counted

• Input tokens are the bulk, and they are repeated. In a twenty-step loop the same system prompt, tool schemas, and accumulated history are re-sent on every call. The prompt everyone measures is the one you send once; the context you re-send twenty times is the one that burns.
• Divide by success, as unit economics does for money. Energy spent on runs that produced nothing is still energy. A deployment whose completion rate is 70% is paying a 43% surcharge on every real task, and that correction is larger than almost any model-choice saving.
• Count the fan-out. Sub-agents, parallel tool calls, judge models in the eval path, and reranking all consume model time that the "cost per prompt" mental model omits. Energy accounting has the same blind spot as cost attribution, and the same fix: propagate through the fan-out or do not report a total.

Build the estimate from your own traces, not from a vendor page.

You already have the hard half of this measurement. Token counts per run are in your traces; what you are missing is one conversion factor and the discipline to apply it consistently.

• Pull tokens per completed task from tracing, not from the bill. The invoice is aggregated at the wrong granularity and nets out cache discounts in a way that hides re-sent context. Your trace store has input, output, and cached-read tokens per span; sum them per run id and keep the p50 and the p95, because the tail is where the energy is.
• Pick one conversion factor and version it. Derive Wh-per-thousand-tokens from the most carefully bounded public figure for a model of comparable size, record which figure and which boundary you used, and change it only on purpose. A consistent, documented, slightly wrong factor is auditable; a factor that silently improves every quarter is not.
• Self-hosted inference is the one case you can actually measure. If you run your own GPUs, this stops being an estimate: node-level power draw divided by completed tasks is a direct measurement, and it is the only path to a defensible number. That is a real argument in favour of the on-premises option in self-hosted inference, and it comes with the obligation to include idle time — a reserved GPU burns power between runs, which is where a low-traffic deployment's footprint actually lives.
• Do not model the data centre. PUE, water, and grid mix are the provider's to publish and yours to cite. Build your estimate as tokens × factor and let the factor carry the infrastructure assumptions, so that a provider's disclosure update is a one-line change rather than a re-derivation.

Carbon is a different number from energy, and the gap is an accounting choice.

Converting watt-hours to emissions introduces a second boundary decision, and the two standard answers can differ by a factor of several for the identical workload.

• Location-based accounting uses the actual grid intensity where the compute ran. It is the physically honest number and it moves with region and time of day.
• Market-based accounting applies the provider's purchased renewable instruments. It is the number most vendor disclosures lead with, and it can approach zero while the local grid is burning gas.
• Report both or you will be asked to. Any procurement or ESG process serious enough to ask for the number is serious enough to know the difference, and offering only the flattering one is the fastest way to lose the conversation. Put them in the same table, labelled.
• Region selection is a lever you may not be able to pull. Routing inference to a low-intensity grid cuts location-based emissions materially — and collides directly with data residency obligations, provider capacity, and the latency budget. Where it is available at all, it is usually available for batch work and not for interactive runs; batch and async inference is the shape that can actually use it.

The levers, in the order they matter — and model size is near the bottom.

Once the unit is a run rather than a prompt, the ranking of interventions inverts against intuition. Everything above the line attacks the multiplier; everything below it attacks the per-call cost, which is the smaller term.

# Ranked by effect on Wh per completed task

1  raise the success rate        removes whole wasted runs
2  cut retries and loops         retry storms are unbounded
3  stop early, deliberately      a deadline is an energy budget
4  raise the cache hit rate      re-sent context is most of the tokens
5  shrink what you re-send       prune history, shape tool results
6  set the effort level          reasoning tokens are real tokens
7  route the easy tasks down     a small model for the high-volume step
8  choose a smaller model        last, and smallest effect

# The two that look like savings and are not
  shorter answers      output is the minority of tokens
  cheaper provider     a price change is not an energy change

• Prompt caching is an energy control, not just a cost control. A cache hit on re-sent context skips the prefill compute, which is where most of an agent loop's work actually happens — the economics are worked through in context caching economics, and the physical saving tracks the money closely enough to use the same dashboard.
• Retry amplification is the worst case and the easiest to miss. A loop that re-sends a growing context and retries on failure can consume more energy failing than ten runs consume succeeding; cap it where deadline budgets are already enforced rather than inventing a second mechanism.
• Route the high-volume steps to small models. Classification, routing, extraction and reranking are the calls you make most often and the ones a small model handles well. This is the one model-size decision with a real multiplier behind it, precisely because it applies to the frequent step rather than the reasoning step.
• Resist the gesture fixes. Asking the model for shorter answers barely moves a number dominated by input tokens, and switching to a cheaper provider changes your invoice without changing a joule. Both are popular because they are easy to announce.

Write the disclosure so it survives a procurement questionnaire.

The reason to do any of this is that someone will ask — a customer's ESG annex, a public-sector tender, an internal reporting cycle — and the answer is due in a week. A method you can restate is worth more than a figure you cannot defend.

# The disclosure, five lines

  unit          Wh per completed task (not per prompt)
  boundary      full serving stack: accelerators, host,
                idle capacity, DC overhead. Excludes
                training, network, end-user device.
  method        tokens/run from traces (p50, p95)
                x published Wh/1k tokens, version dated
  carbon        location-based AND market-based, both shown
  caveat        provider efficiency improved ~33x in 12
                months; figure is restated each quarter

• Lead with the unit. Half the confusion in this area is two parties comparing a per-prompt figure with a per-task figure and concluding that one of them is lying. Naming the unit first ends that in one line.
• Keep the training question separate and answer it honestly. If you fine-tune, that is an episodic cost you can report as an amortised line; if you do not, say that the training footprint of the base model is the provider's to disclose and outside your boundary. Vagueness here reads as evasion.
• Restate quarterly, and expect the number to fall. Per-token efficiency is improving fast while run counts grow faster. If your per-task figure drops and your total rises, publish both — the alternative is a well-earned accusation of reporting the convenient half, the same trap price deflation and cost per task describes on the money side.
• Don't let this become the project. For most deployments the energy story is a paragraph and a method, not a workstream. The exception is a high-volume deployment where the multiplier is large and the brand is exposed, and there the measurement pays for itself through the efficiency work it motivates.