Annotation & Labeling Ops

The ceiling nobody computes.

An LLM judge is a classifier fitted to human labels. Its reported accuracy is agreement with those labels, so the labels define the maximum attainable score — and if the labels disagree with each other, the maximum is well under 100% and you cannot see where it is without measuring.

• Do the arithmetic once and it changes your roadmap. If two qualified annotators agree on 75% of items, then on the quarter of items where they split, any judge is right by coin flip against whichever label happened to be recorded. A judge at 78% and a judge at 85% are not meaningfully different classifiers; they are two samples from the same noise band, and the difference between them will not replicate next month.
• Single-annotator label sets hide this completely. One person labelling everything produces a perfectly self-consistent set with an unknown relationship to anyone else's judgement, and a judge trained to match it inherits that person's idiosyncrasies as ground truth. The failure is silent, and it surfaces months later as "the eval says we improved and users say we did not".
• Report agreement as a headline number alongside judge accuracy. "Judge agrees with humans 84%; humans agree with each other 88%" is an honest and immediately actionable pair. The first number alone invites a quarter of wasted tuning.
• Chance-corrected agreement is worth the extra step on skewed tasks. When 90% of traces are fine, two annotators who both say "fine" almost always hit 90% raw agreement while carrying no signal at all. Cohen's kappa or Krippendorff's alpha divides out that baseline; on the imbalanced pass/fail data typical of agent evaluation the raw number is actively misleading.

Low agreement is a rubric defect, not a people defect.

The instinct on seeing 68% agreement is to train the annotators harder or hire better ones. That is almost always the wrong diagnosis. Competent people disagree when the question is underspecified, and the question is your artifact.

• Most disagreement traces to one of four ambiguities. An undefined severity boundary ("is a wrong-but-harmless citation a failure?"); a compound criterion that bundles two judgements into one label; an unstated audience ("correct for whom?"); and a missing tie-break rule for partial success. Each is fixable in a sentence, and each recurs across hundreds of items until someone writes that sentence.
• Write the rubric as decision procedure, not as description. "Rate helpfulness 1–5" produces disagreement forever. "Mark FAIL if the response asserts a fact absent from the retrieved documents, even if the fact is true" produces agreement, because it names the decision rather than the feeling. Every criterion should be answerable by pointing at something in the trace.
• Anchor every level with a real labelled example, from your own traffic. Two or three examples per level, argued in a sentence each. Annotators calibrate against examples far more reliably than against adjectives, and the examples double as the few-shot set when you build the judge.
• Give "cannot tell" a first-class button. Forcing a judgement on an ambiguous item manufactures a label out of nothing and pollutes the set. Abstentions are data: a criterion with 15% abstentions is a criterion that needs rewriting, and you only learn that if abstaining was allowed.
• The rubric is versioned, and labels are stamped with the version. Rubrics change — that is the process working — and labels collected under v3 cannot be pooled with v5 without checking whether the change was material. Unversioned label sets quietly become uninterpretable within two quarters.

Disagreement is the most valuable output, so route it rather than average it.

The standard pipeline takes three labels and majority-votes. That is defensible for producing a training set and wasteful as a process, because it discards the signal that tells you where your criteria actually are.

• Split items are the decision boundary. Items everyone marks pass are uninformative; items everyone marks fail are uninformative; the contested ones sit exactly where your policy is undefined. They are simultaneously the best adjudication queue, the best few-shot examples for the judge, and the best regression set.
• Adjudicate, and write down why. A senior reviewer resolves the split and, critically, records the reasoning in one line. Those lines are the raw material of the next rubric version. An adjudication that produces a label but no reason has fixed one item and left the other four hundred like it unfixed.
• Track disagreement rate per criterion, not just overall. The aggregate hides the one criterion doing all the damage. Usually a single criterion accounts for most of the disagreement, and rewriting it moves the whole number in an afternoon.
• Feed the contested set to the judge as an explicit test. A judge that scores well on easy items and poorly on contested ones is worse than its headline accuracy suggests, because contested items are over-represented in the traffic you actually care about. Stratify judge evaluation by human agreement level — see judge calibration and meta-evaluation.
• Never let the judge's own output seed the labels it will be scored against. Pre-filling a label for a human to confirm raises throughput and destroys independence: confirmation bias makes agreement rise while accuracy does not, and you lose the ability to detect the judge's systematic errors — permanently, because the labels now contain them.

Who labels, and the seat that cannot be outsourced.

Agent traces are not sentiment data. A single item can be a twenty-step trajectory with tool calls, retrieved documents and a partially-correct outcome, and judging it requires knowing what the product was supposed to do.

• Domain experts set the standard; a vendor pool can scale it. The expert defines the rubric, labels the calibration set and adjudicates splits. Scaled annotators then work against that standard on the items it covers. Inverting this — a vendor pool defining quality for a clinical or legal agent — produces labels that are internally consistent and externally wrong.
• Product and engineering must both label, regularly, in small quantities. Two hours a month each. Not for throughput — for the thing you cannot get from a dashboard, which is the texture of how the system actually fails. Teams that stop reading traces start optimising metrics that have drifted away from the product.
• Annotators need the same context the judge will get, and no more. If the judge will see only the final response, a human labelling with the full trajectory produces labels the judge structurally cannot reproduce, and you will misattribute the gap to the model. Match the observation window deliberately.
• Budget for onboarding and expect a calibration period. A new annotator's first few hundred labels agree with the standard measurably less than their later ones. Label a shared calibration set on day one, review it with them, and hold back their early work from the golden set rather than discovering the drift downstream.
• Watch for fatigue and position effects. Agreement falls measurably in long sessions, and items presented later in a batch get graded differently from identical items presented early. Randomise order, cap session length, and re-inject a handful of known-answer items as an ongoing quality check.

Sampling is where the budget is won or lost.

Expert labelling is usually the largest line item in an evaluation programme, and the default policy — label a random sample — spends most of it on items whose outcome was never in doubt. What you buy with a labelling budget is resolution at the decision boundary.

• Stratify before you sample. By task type, by tenant, by trajectory length, by whether a fallback fired. A random sample of production traffic is dominated by the easy majority class and will tell you nothing about the segment that generates the complaints.
• Spend disproportionately where the judge is uncertain. Take the items where the judge's confidence is low or where two judge configurations disagree and label those. This is the highest-information-per-dollar policy available, and it typically reaches a usable calibration with a fraction of the items random sampling needs.
• Reserve a genuinely random slice anyway. Uncertainty sampling is biased by construction and cannot estimate a rate. Keep a small random stratum purely for "what fraction of production is actually failing", and never mix the two sets in the same number.
• Mine the free labels first. Edits, retries, escalations, undo, and the difference between what the agent produced and what the user shipped are labels you already own and are not paying for — the whole argument of production feedback signals. Human annotation should be spent on what those cannot tell you, which is why a failure happened.
• Cap the golden set and curate it, rather than growing it forever. A few hundred well-chosen, adjudicated, versioned items beat ten thousand cheaply-labelled ones, and they are affordable to re-label when the rubric changes — which is the operation that kills large sets. The cost model is in the cost of evaluation.

Labels have a date, and the numbers to run the programme on.

A label is a judgement about a specific output under a specific rubric at a specific time, and all three move. Treat the golden set as a maintained asset with an owner, or it decays into a set of assertions nobody can defend.

• Store provenance with every label. Annotator, rubric version, timestamp, observation window, whether it was adjudicated. Without provenance you cannot answer "did quality change or did our standard change?", and that question arrives during exactly the incident where you need it — see detecting quality regressions.
• Re-label a slice on a schedule. Quarterly, re-label 5% of the golden set blind and measure agreement with the stored labels. Drift against your own past judgements is normal and worth knowing about; discovering it during a launch review is not.
• Inter-annotator agreement, per criterion. The health metric for the rubric. It should rise after each rubric revision; if it does not, the revision did not address the real ambiguity.
• Judge-to-human agreement, reported against the human ceiling. Always as a pair. A judge at 82% against a ceiling of 85% is finished work; the same 82% against a ceiling of 97% is a broken judge.
• Cost per adjudicated label, and labels per merged eval change. The efficiency of the programme. If the second number is large, you are labelling to feel thorough rather than to decide something.
• Abstention rate. The early warning that the rubric has stopped covering the traffic — usually the first visible sign that the product changed and the eval did not.

Related: LLM-as-judge for agents for what the labels calibrate, why evaluating agents is hard for the ground-truth problem underneath, outcome vs trajectory evaluation for what an annotator is even looking at, and uncertainty & calibration for the confidence numbers the sampling policy leans on.