Production Feedback Signals

Explicit feedback is a tiny, biased sample of the wrong population.

The thumbs widget is the first thing every team ships and the last thing they should rely on. It is not that the data is bad — it is that the data describes a self-selected minority whose composition changes with your UI, and the ways it misleads all point the same direction.

• The response rate is low enough to make slices meaningless. Most products see explicit ratings on well under one percent of interactions. That is enough to rank two variants over a quarter and nowhere near enough to answer "did last Tuesday's prompt change hurt the finance workflow", which is the question you actually have.
• The population is bimodal by construction. People rate when something is remarkable. The vast middle — adequate answers that got the job done, and mildly wrong ones that were quietly fixed — never registers, and the mildly-wrong bucket is where nearly all your recoverable quality lives.
• Thumbs-up measures satisfaction, and satisfaction is not correctness. A fluent, confident, wrong answer rates well from a user who cannot check it — the exact failure mode you most need to detect. Any objective that rewards the up-rate is an objective that rewards confidence, and models are extremely good at supplying confidence.
• The signal moves when you move the button. Placement, prompting copy and timing change the rate by multiples, which means a quarter-over-quarter comparison across a UI redesign is comparing two different instruments. If you keep the widget, freeze its presentation and treat any change to it as an instrument recalibration.
• Free-text comments are the part worth keeping. Low volume, high information, and they name failure modes your taxonomy does not have yet. Read them; do not aggregate them into a sentiment score, which throws away the only thing they were good for.

The edit is the label. Capture the diff between the output and what shipped.

Almost every agent that produces an artefact — a draft, a patch, a query, a summary, a reply — hands it to a human who then changes it and uses it. That change is a free, dense, per-instance correction produced by a domain expert with full context, and in most products it is thrown away the moment the user hits send.

• Store the pair, not the score. The agent's output and the final artefact, joined by task ID, with the trace attached. An accepted-unchanged pair is a positive example; a heavily-rewritten pair is a labelled failure with the correct answer supplied. This is the highest-value data your product generates and it costs a database column.
• Edit distance is the metric; the diff is the evidence. Track normalised edit distance as the continuous quality signal — it is dense, computable in real time, and correlates with usefulness far better than any rating. Then read the diffs, because the metric tells you that quality moved and only the diff tells you what the agent got wrong.
• Cluster the diffs and the taxonomy writes itself. Group edits by what they changed — tone, a wrong constant, an omitted caveat, an over-long preamble, a hallucinated field. In practice a handful of clusters cover most edits, and each one is a concrete, addressable defect rather than "quality is a bit low".
• Distinguish correction from personalisation. A user rewriting a factual claim is a defect. A user changing "Hi there" to "Hey" is a style preference, and treating it as a defect sends you optimising noise. Separate them by cluster and only the first class enters the eval set.
• Accepted-unchanged is ambiguous and must be split. It means either "this was right" or "I did not read it". Disambiguate with whatever proxy your surface allows — time between presentation and acceptance, whether the artefact was opened, whether the user later reverted — because a rubber-stamping population inflates your best metric while quality is unmeasured.

Read the behavioural signals for what they actually mean, not for what you would like them to mean.

Beyond the edit, the session emits a handful of behaviours that carry real information. Each one is directional, none is self-explanatory, and the common mistake is to promote one to a headline metric without establishing what it measures.

• Immediate rephrase-and-retry is the strongest negative signal you have. The user asked, got something, and asked again in different words within seconds. High volume, low latency, unambiguous in direction. Segment it carefully: a retry after an explicit clarifying question from the agent is the system working.
• Abandonment mid-task tells you where, which is the useful part. The step at which people leave localises the defect far better than an overall completion rate — a fifth of users leaving at the same confirmation screen is a design bug with an address.
• Escalation to a human is a labelled failure with a resolution attached. The handoff carries both the context that defeated the agent and, shortly after, what the correct handling was. It is the single richest event in the whole set and it arrives pre-adjudicated.
• Copy, export and send are weak positives; treat them as such. They indicate the output was used, not that it was right. Useful as a denominator, dangerous as a target.
• Re-contact within the week is the one that predicts churn. The same user bringing the same problem back is the clearest available evidence that the first resolution was cosmetic, and it is invisible in every per-session metric.
• Every signal is confounded by traffic mix. Retry rate rises when a new customer onboards with unusual phrasing, not because the model regressed. Segment by cohort, task type and surface before you believe any move — the same discipline quality regression detection insists on.

Feedback is a sampling instrument, not a metric. Point it at your eval set.

The reframing that makes all of this pay is to stop asking "what is our feedback score" and start asking "which fifty interactions this week most deserve a human look". Feedback is very good at the second question and structurally incapable of answering the first.

• Build a triage queue, prioritised by signal strength. Thumbs-down with a comment, high edit distance, retry-then-escalate, abandonment at a late step. Ranked, deduplicated, and short enough that a human actually clears it — fifty a week reviewed beats five thousand aggregated.
• Every confirmed defect becomes a golden case with the correct output attached. This is the whole point. The correction already exists in the diff or the escalation resolution, so promoting it to an eval case is close to free — which is what makes eval-driven development sustainable rather than a heroic one-off labelling project.
• Use the signals to sample for judging, not to replace judging. An LLM judge over a feedback-weighted sample gives you far more information per token than a judge over uniform traffic, because you have spent your budget where the interesting cases are.
• Never optimise a feedback ratio directly. Point an objective at thumbs-up rate and you will get an agent that flatters, hedges less, and stops declining things it should decline. Point it at edit distance alone and you will get shorter outputs. Feedback selects what to examine; the eval set — held out, with labels — is what you optimise against.
• Keep a uniform random sample running alongside. Feedback-weighted sampling is deliberately biased, so it cannot tell you the base rate. A small unbiased stream, judged the same way, is what makes the weighted stream interpretable.

Close the loop in one system, or the signal decays into a report nobody reads.

The failure mode is not collecting the data — teams do collect it. It is that the signal lands in an analytics tool, the eval set lives in a repository, and the connection between them is a person who remembers. That connection has to be a pipeline.

• Join feedback to the trace at write time. A thumbs-down with no trace is an opinion. A thumbs-down joined to the model version, prompt version, tool calls, retrieved context and latency is a diagnosis. Carry the ID through the surface so this is a foreign key rather than a reconstruction — the reason tracing is a prerequisite for all of this.
• Make promotion to the eval set a one-click action in the review tool. Friction here is the whole difference between a suite that grows weekly and one that was written at launch. If a reviewer has to open a repository and write YAML, the suite stops growing on day nine.
• Version the eval set and report on it, including what was added and why. A growing suite silently changes what your quality number means; without a changelog, a score that moved because the set got harder is indistinguishable from a regression.
• Feed confirmed defect clusters into the release gate. A cluster that appears three weeks running is not a bug list item, it is a missing capability, and the gate is where that becomes visible to whoever prioritises.
• Report back to the user when their report changed something. Response rates on explicit feedback rise sharply when people learn it goes somewhere, and this is the cheapest available intervention on the one signal you cannot instrument your way around.

The traps: consent, survivorship, and the feedback loop that eats itself.

Implicit feedback is powerful precisely because it is collected without asking, which is also the source of every problem with it. Three deserve naming before you build the pipeline rather than after.

• Edits and drafts are user content, and using them is a governance question. Retaining what a person wrote in order to improve a model needs a lawful basis, a retention period, a deletion path and — in most enterprise contracts — an explicit term. Settle it with data governance before the pipeline exists, because retrofitting consent onto a corpus you already trained on is not a thing you can do.
• Survivorship bias is built in and it compounds. Every signal comes from users who stayed. The people the agent failed hardest left in week one and contribute nothing, so a feedback-driven roadmap optimises steadily for the population that already tolerates the product. Counter it deliberately with the first-session cohort and with exit signals, or your metrics will improve while your market narrows.
• Feedback-weighted training narrows the distribution. If your eval set is built only from cases that generated signal, you are progressively fitting the tail and losing the middle. The uniform sample from step 4 is the corrective, and it is the first thing dropped when the budget tightens.
• Anything visible and rewarded will be gamed. If a support team's bonus depends on thumbs-up rate, you now have a thumbs-up rate and no signal. Keep the instrument away from anyone's compensation, exactly as you would with any other measurement you intend to keep trusting.
• Do not let the judge and the feedback validate each other. Tuning the judge on cases selected by feedback, then using the judge to interpret feedback, produces a closed system that agrees with itself and drifts from reality. Anchor both against periodic human adjudication — the argument in judge calibration.

Related: online vs offline evals for where each signal belongs, SLOs and error budgets for agents for the tiering these signals sit inside, and evaluating agents for the ground floor.