Task horizon.
A task horizon converts an agent's ability into a unit you can plan with: the length of job — measured in human working time — that it finishes on its own at a stated success rate. The number everyone quotes is the 50% horizon, and the number you deploy against is the 80% one, which runs roughly five times shorter. Quote the wrong one and you will promise a day of unattended work from a model that reliably delivers an hour.
A capability measured in minutes, not percent.
Most benchmarks answer "what fraction of these problems does the model get right?" A horizon answers a different question: "how long a task can it finish before it falls apart?" You take a set of tasks, record how long a competent human needs for each, run the agent, and find the duration at which its success rate crosses some threshold. That duration is the horizon at that threshold.
The measure comes out of METR's work on long software tasks, and it caught on because it says something a percentage cannot. A score of 71% on a benchmark tells you nothing about whether you can leave the agent alone for an afternoon. A horizon does, in the unit your planning already uses. It is also the cleanest way to state the thing practitioners keep rediscovering: agents do not fail at hard steps, they fail at long ones, because the per-step error compounds and nobody is watching.
The headline finding is a trend. Across roughly six years the frontier's 50%-success horizon has doubled about every seven months — from a couple of seconds for GPT-2 to tens of minutes for models of the Claude 3.7 Sonnet generation, then to hours. Recent re-estimates put the post-2023 doubling closer to four months. Treat the exact slope as contested and the direction as not.
The 50% horizon is a coin flip, and it is the one in the headline.
Here is the part that gets dropped in the retelling. A 50% horizon means the agent finishes half of tasks that size. Half. Nobody deploys anything at a coin flip, so the operating number is the horizon at a reliability you would actually accept — and METR's own reporting puts the 80% horizon at roughly one fifth of the 50% figure.
Run that through a concrete case. A model with a 50-minute horizon at 50% has something closer to a ten-minute horizon at 80%. A model advertised at five hours is a one-hour model if you need four runs in five to land. The gap is not a rounding error, it is a different product.
Reliability collapses faster than length grows because a long task is a chain: fifteen steps at 98% each is 74% overall, and at 95% each it is 46%. This is the same arithmetic behind pass^k scoring on agent benchmarks — succeeding once is a capability claim, succeeding k times running is a deployment claim, and only the second one predicts a Tuesday.
So when a vendor, a paper or a colleague quotes a horizon, the first question is which threshold it is at. If nobody says, assume 50%, divide by five, and plan against that.
The trend line was measured on a task distribution that is probably not yours.
Horizons are estimated on suites of software-engineering and research tasks: clean start states, a machine-checkable finish, no second party. That is a deliberate methodological choice — you cannot time a task whose completion is a matter of opinion — and it is also the reason the number transfers badly.
- Your tasks may be messier than the suite. Ambiguous requirements, a stale environment, a tool that was renamed last week. METR has itself published a note on the limits of the measure, and the honest reading is that the horizon describes performance on well-specified work.
- Or they may be cleaner. A narrow, repetitive, fully-instrumented pipeline can run far past the published horizon, because the failure modes the suite samples are the ones you engineered away.
- Human time is a proxy for difficulty, not a description of it. A four-hour task that is four hours of typing is not a four-hour task that is four hours of judgement, and models are much better at the first.
- A second party resets the clock. Anything with a person in the conversation — approvals, clarifications, someone pushing for an exception — is a different regime, and the durations do not carry across.
The useful move is not to distrust the number but to place it: a published horizon tells you what the class of model can sustain on well-posed work, and your own measurement tells you what this model sustains on yours.
Measure your own, and spend it on autonomy decisions.
Measuring a horizon in-house is cheaper than it sounds and does not require a benchmark harness. Take thirty real tasks you already have traces for. Label each with the human minutes it takes — an estimate to the nearest half hour is fine, precision is not the point. Run the agent five times per task and record how often it fully succeeds. Plot success against duration and read off where it crosses 80%. That crossing is your number, and it will be smaller than you hoped.
Then let it set policy rather than sit in a slide:
- Size the unattended unit of work below the horizon. If yours is forty minutes, a two-hour job should be checkpointed into pieces under forty, each with its own success check — the argument for decomposition made in goals, planning & termination.
- Set the autonomy level per rung of task length. Below the horizon, run and report. Around it, run and require review. Above it, do not start without a human owner.
- Track it as a release metric. A horizon that moved after a model swap or a prompt change is the single most informative number you can put in a release note, and it is more actionable than an aggregate score.
- Re-measure when the environment changes, not just when the model does. Horizon is a property of the pair, and a tool rename can shorten it without anything else moving — see third-party tool drift.
Do three things with this idea. When you read a horizon anywhere, find the threshold before you quote it, and if it is 50% assume the deployable figure is a fifth of it. Measure your own 80% horizon on thirty real tasks — half a day of work, and it will be the most useful number your team has about its agent. Then use it as the size limit on unattended work, so an agent is never handed a job longer than the length at which it reliably finishes.
Related: agent evaluation for how horizons fit alongside the rest of your measurement, reading benchmarks for the general skill of asking what a number was measured on, eval variance & statistical power for how many runs you need before a horizon estimate means anything, and durable state & resumability for surviving the tasks that exceed it anyway.