The right axis, the wrong unit

Reflection did something for agent buyers this month that no frontier lab has bothered to do: it put tokens-per-attempt on the same chart as the benchmark score. Beam, its 501B-parameter mixture-of-experts model with 23B active per token, is reported at 80.1% on Terminal-Bench 2.1 while generating roughly 25,000 tokens per attempt, against roughly 80,000 for a rival scoring 81.0%. Under a point behind on capability, three times ahead on verbosity. That is the comparison anyone running an agent loop has needed and nobody publishes. The claim bolted to it — three to four times less inference compute — is an estimate over generated tokens that leaves out prefill, and on a twenty-step task a 3× brevity win pays out as 1.7× while a plain 3× price cut pays the full three.

At a glance

Beam was announced on 5 October 2026 as Reflection's first open-weight model. Everything in the table is the vendor's own figure unless noted, and independent reproductions were not available at launch. Two caveats carry through the whole piece: the scorecard was published before the weights, which arrived on a Hugging Face page whose technical report is still marked as coming; and coverage does not agree on the rival's score, with one tracked comparison putting GLM-5.2 at 81.0 and another table giving a different figure.

Item	Reported	How to read it
Architecture	501B total, 23B active per token, 1M context	Sparse MoE: serving cost tracks the 23B, memory tracks the 501B.
Terminal-Bench 2.1	80.1% (BF16); 78.9% FP8; 78.4% NVFP4	Self-reported. The quantised builds you would actually serve score lower.
Tokens per attempt	~25,000, against ~80,000 for GLM-5.2 at 81.0%	One outlet's reading of Reflection's chart. Order of magnitude, not a measurement.
Efficiency claim	Matches GLM-5.2 on reasoning with 3–4× less inference compute	An estimate, and the method omits prefill, attention and serving overhead.
Licence	Apache 2.0	Genuinely permissive. See open-weight licences.

Under a point apart on the score. Three times apart on the tokens. Only one of those gaps is usually published.

The axis is right, and it is overdue

A pass rate with no budget attached is not a comparable number, because the budget is a dial on the harness rather than a property of the model. Raise the step ceiling, allow more sampling, let the model think longer, and the score moves — which is the entire content of inference-time scaling. A leaderboard that reports only the score therefore ranks submissions partly on how much each one was willing to spend, and the field has known this for a year without changing the chart.

Princeton's HAL made the first serious move by publishing cost-per-solve next to pass rate, and the argument for it is in HAL and asynchronous agent eval. Reflection's chart is a different and complementary move: cost-per-solve tells you what a submission spent, in dollars that depend on whoever was selling tokens that week, while tokens-per-attempt is a physical quantity that survives a price change. For planning an agent loop it is the better unit, because tokens are what you budget, what fills the context window, and what you wait for while they decode one at a time.

So credit where it is due. If this becomes a convention — every agentic score published with the token budget that produced it — agent evaluation gets materially more honest, and the reason is spelled out in the agent harness: every benchmark number is a score for two things and only one of them is named on the chart.

Then the unit goes wrong, twice

First, "inference compute" is not what you pay. The 3–4× figure is reported as an estimate rather than a measurement, derived from generated tokens and active parameters, with prefill, attention cost and serving overhead excluded. For a chat completion that is a tolerable simplification. For an agent it is the wrong half of the bill. A model is stateless, so every step of a loop re-sends the system prompt, the tool schemas and the whole transcript so far — the quadratic arithmetic worked through in agent cost control. Prefill is not an overhead you can round away; on a long agentic task it is most of the input volume, and input volume is most of the invoice.

Second, a tokens-per-attempt number is a harness number. It was produced by one scaffold, on one benchmark, by the party selling the model. Change the system prompt, the tool set, the retry policy or the stop rule and the figure moves, which is why the same model can post very different trajectory lengths in two teams' hands. And the builds you would actually serve — FP8 and NVFP4 — are reported a point and a half lower on the score, with no separate token figure given at all.

Neither objection makes the chart useless. They make it a starting point for arithmetic you have to do on your own trajectories, not a conclusion you can adopt.

Do the arithmetic: a generated token is billed more than once

The cost of a generated token depends on where in the trajectory it was written.

Write a token at step i of an N-step task and you pay for it once at the output rate, then again as input at every later step. The marginal cost is p_out + (N − i) × p_in, and at twenty steps with input at $0.60 and output at $2.40 per million, a token written mid-trajectory costs about $8.10 per million — roughly three and a half times its sticker output price. This is why verbosity is worth attacking at all, and it is also why the attack under-delivers.

Model one task concretely. Twenty steps, a 4,000-token stable prefix, 600-token tool results, input at $0.60 and output at $2.40 per million. A verbose model writes 1,200 tokens a step; a brief one writes 400 — exactly the 3× Beam's chart claims.

                      input tokens   output tokens      bill
verbose, 1200/step         422,000          24,000    $0.311
brief,    400/step         270,000           8,000    $0.181   1.72x cheaper
verbose, 1/3 the price     422,000          24,000    $0.104   3.00x cheaper

input = N*prefix + (write + tool)*N*(N-1)/2      N = 20

Two discounts of the same nominal size. Only one of them applies to the whole bill.

A 3× cut in generated tokens produced a 1.72× cut in the bill. The reason is visible in the arithmetic: the stable prefix is re-sent twenty times whatever the model writes, and the tool results are the environment's output, not the model's. Brevity shrinks one of three terms. A price cut shrinks all three, which is why a model that is merely a third as expensive beat the brief model outright on this trajectory — $0.104 against $0.181.

Now turn on prompt caching, with 90% of input served from cache at a tenth of the input rate. The verbose task falls to about $0.106 and the brief one to $0.050, so the brief model's advantage widens to 2.1×. That is the counter-intuitive result and the useful one: caching removes the floor that was diluting brevity, so the two levers are complements rather than substitutes. A brief model is worth paying a premium for precisely when your cache hit rate is high — see prompt caching.

Where brevity wins outright

The bill is the weakest case for a terse model. Three others are stronger, and none of them appears on a cost chart.

Latency. Output tokens decode serially. Three times fewer tokens per step is close to three times less time per step, and on a twenty-step task that is the difference between a tool a person waits for and one they walk away from. The trade-off is priced in pricing latency.
Context headroom. Every token written is a token occupying the window for the rest of the run. A verbose model hits compaction sooner, and compaction is where agents lose the thread — context budgeting.
Attention degradation. Effective recall falls well before the advertised context limit, so a shorter transcript is a more accurate one, not merely a cheaper one: effective versus advertised context.

Those three are why the axis deserves to become a convention even though the dollar case is weaker than it looks. The durable version of this whole argument, with the break-even rule, is fewer tokens vs cheaper tokens.

What to actually do with Beam

Four steps, in order, and the first two cost an afternoon.

Measure your own trajectory shape before shopping. Median steps per task, prefix size, tool-result size, tokens written per step, cache hit rate. Without those five numbers no vendor efficiency claim can be converted into your bill, and with them every claim can.
Re-run the comparison at a matched budget. If one model needs 80,000 tokens to score 81.0% and another needs 25,000 to score 80.1%, the ranking at a 30,000-token cap is not the ranking on the leaderboard. Cap first, then score — budget-matched evaluation.
Price the quantised build, not the headline one. FP8 and NVFP4 are what fits on affordable hardware and both are reported lower. The relevant score is the one for the artefact you will serve, as in distillation and quantization.
Wait for a third-party score before committing. Self-reported numbers on an agentic benchmark are a claim about a model and a scaffold together, and the scaffold belonged to the vendor. Reading benchmarks is the checklist.

And treat the renting-versus-running question separately. An Apache-2.0 licence is a real exit option and a real right to fine-tune, but a hosted endpoint serving the same weights gets you the discount without the operations — the division of labour argued in open-weight vs closed models.

FAQ

Is Beam a better choice than a frontier closed model for an agent?

Not on the published evidence. At 80.1% self-reported on one benchmark it is behind several rivals, including one open-weight model reported above 90% on the same test. The interesting claim is the token budget, not the capability, and capability is what ends a task in fewer steps — which is itself a cost lever.

Does 23B active parameters mean it serves like a 23B model?

For compute per token, roughly. For memory, no: all 501B parameters have to be resident to route to any expert, so the hardware floor is set by the total and the throughput by the active count. That asymmetry is what makes sparse models cheap to rent and awkward to self-host — mixture of experts.

Why is a 3× price cut worth more than a 3× brevity win?

Because a price cut applies to every token on the invoice, including the re-sent transcript and the tool results, while brevity only shrinks the tokens the model itself writes. On the modelled twenty-step task the prefix and tool results were two thirds of the input volume, and no amount of terseness touches them.

Then why does prompt caching make brevity matter more?

Because caching discounts exactly the part brevity cannot reach. Once the re-read prefix costs a tenth as much, the model's own output becomes the larger share of the bill, and a lever that cuts output three times has more of the bill to act on. In the modelled task the brief model's advantage rises from 1.7× to about 2.1×.

Should we trust the 25,000-token figure?

As an order of magnitude, with two caveats. It comes from one outlet reading a vendor chart rather than from a published table, and it was produced by Reflection's own scaffold on one benchmark. Use it to decide whether to run your own measurement, not as the measurement.