Content Moderation Agents

Your policy document is a summary of a body of case law.

Every mature moderation operation runs on internal enforcement guidance that dwarfs the public policy: worked examples, carve-outs, regional exceptions, the memo from the incident two years ago, and a queue of escalations that quietly set precedent. A rule like "no graphic violence" resolves nothing about a news photograph, a medical illustration, a video game clip, a historical archive, or a documentary about a war. Those distinctions live in decisions, not in text.

• The easy cases were already automated. Hashes catch known material, classifiers catch the obvious, spam heuristics catch the volume. What arrives at a human is, by construction, the residue that those layers could not resolve — so an agent measured on a random sample is being graded on a distribution it will never see in production.
• The hard cases turn on context the item does not contain. Who posted it, to whom, in reply to what, with what history, in which country, in which language, at which moment in a news cycle. The same sentence is harassment or reclamation depending on the speaker.
• Consistency is the product. Users and regulators do not primarily demand that you be right; they demand that you treat like cases alike. An agent that is 92% accurate but incoherent between two identical posts is worse than a duller one that is consistent, because inconsistency is the thing that gets screenshotted.
• Therefore the objective is not "classify this item" but "apply the precedent that governs this item". That is a retrieval problem with a decision attached, and framing it as classification is the error the whole rest of this page follows from.

Build it as retrieval over decided cases.

The asset you already own and never use is the decision history: millions of items with a verdict, a policy code, a reviewer, an outcome, and often an appeal result. That corpus is the enforcement guidance, in the only form that is complete and current.

• Retrieve neighbouring decisions before deciding. For an item under review, pull the most similar previously-decided items with their verdicts and rationales. The agent's job is then to reason about whether the precedent applies — a task models are considerably better at than open-ended judgement, and one whose output a human can check in seconds.
• Make the citation mandatory. Every verdict names the policy clause and the prior decisions it followed. A reviewer who disagrees can now argue with something specific, and an appeal can be answered with a reason rather than a category.
• Recency-weight the corpus, because policy moves. A verdict from before the last policy revision is a trap, not a precedent. Version your policy, tag decisions with the version in force, and let retrieval prefer the current era.
• When precedent conflicts, that is a signal, not a tie to break. Two near-identical items with opposite verdicts means the policy is genuinely ambiguous there. Route those to a human and, more importantly, to the policy team — an agent that surfaces incoherence in your own enforcement is delivering more value than one that papers over it.
• Hybrid retrieval, not pure vectors. Slurs, brand names, product terms and coded language are exact-match problems, and semantic similarity dilutes them. See hybrid search and reranking.

The base rate, not the accuracy score, sets your reviewer headcount.

Moderation is the domain where excellent-sounding model metrics produce unusable systems, because violations are rare and the clean population is enormous. Do this arithmetic before you promise anyone a review-cost reduction:

# 10M items/day, 0.2% actually violating
items      = 10_000_000
violations = items * 0.002                    # 20,000

# A model at 95% recall and 99.5% specificity — strong numbers:
true_pos  = 0.95 * violations                 # 19,000
false_pos = 0.005 * (items - violations)     # 49,900

# Precision    = 19,000 / 68,900  =  27.6%
# Review queue = 68,900 items/day, 72% of which are fine

• Precision, not recall, determines the queue. Half a point of specificity is worth more than five points of recall at this base rate, and the two are usually traded in the opposite direction because recall is the number that sounds like safety.
• Tier by confidence and by harm, and be explicit about the asymmetry. Auto-remove only where the harm is severe and the classification is near-certain; auto-approve the vast clean majority; send the middle to humans ranked by expected harm. The middle band is the system.
• Ranking the queue is a bigger win than deciding it. Getting the worst item in front of a reviewer first reduces real-world harm more than automating the tail of easy calls, and it carries almost none of the risk. Ship that first.
• Cost is real at this volume. Ten million model calls a day is a serious bill; route cheaply and reserve the expensive path for the ambiguous band. See model routing and cascades and agent cost control.

Appeals are the only free labels, and the only window onto false positives.

The structural blindness of every moderation system is that removals are invisible to everyone except the person removed. You learn quickly when you leave something up that you should have taken down — someone reports it, a journalist writes about it, a regulator asks. You may never learn that you removed ten thousand legitimate posts, because the affected users mostly leave rather than argue.

• Treat every overturned appeal as a labelled training case, immediately. It is ground truth, produced by your own reviewers, arriving daily, at no cost. A moderation agent that is not fed by the appeals pipeline is throwing away its best data source.
• Track the overturn rate per policy area and per agent version. A rising overturn rate in one category is the earliest and clearest signal that the agent has drifted, and it precedes any external complaint by weeks.
• Sample and re-review auto-approved content as a standing cost. This is the only mechanism that measures the errors in the direction nobody reports. It is always the first thing cut when the agent looks like it is working, and cutting it is how a silent regression runs for a quarter.
• Appeal handling must not be the same agent. An appeal reviewed by the model that made the decision is not a review; it is the same computation with extra steps and a strong prior toward its own answer.

The explanation and the human review are legal artifacts, not features.

Moderation is one of the few agent domains where the output has a statutory shape. Under the EU's Digital Services Act, a provider restricting content owes the user a statement of reasons explaining the grounds and the available redress, submits those statements to a public transparency database, and must run an internal complaint-handling system whose decisions are taken under the supervision of qualified staff and not solely by automated means. Build accordingly, and check the obligations that apply to your own jurisdictions and scale.

• The reason must be derived from the decision, not generated after it. Asking a model to explain a verdict it already produced yields a plausible reconstruction, and a plausible reconstruction of a wrong decision is a well-written falsehood. The rationale comes from the retrieved precedent and the cited clause, recorded at decision time. Structured refusal and why-trails is the mechanism.
• Keep the record immutable and attributable. Item, policy version, model version, retrieved precedents, verdict, and which human approved it if any. This is an audit trail in the legal sense, not a debugging log, and it will be read by someone adversarial.
• A human in the complaint path is a design constraint, not a cost line. Architect the appeal queue so that qualified staff decide, with the agent preparing the case rather than resolving it — and make sure "qualified" survives contact with the throughput target.
• Name the accountable owner for each policy area. An agent does not absorb responsibility; the enforcement decision remains the platform's. See accountability and roles and the regulatory landscape.

Evaluate on drift and disagreement, because accuracy is a moving target.

An eval set built this quarter decays faster here than in any other domain in this section: adversaries adapt within hours, coded language turns over weekly, and the policy itself is revised in response to events. A static benchmark measures how well you did against last month's internet.

• Score agreement with reviewers, stratified by difficulty. Overall agreement is dominated by easy cases and always looks excellent. Report separately on the escalation band, which is the only population the agent is actually being asked to help with.
• Measure consistency directly. Feed near-duplicate items and assert identical verdicts. Feed the same item twice and assert the same outcome. Self-inconsistency is a defect you can detect without any ground truth at all, and it is the one users notice.
• Keep an adversarial slice that grows. Injection attempts, obfuscated slurs, homoglyphs, benign-looking phrasing of known-violating content, and last month's evasion technique. Every incident adds a case; the suite is a ratchet.
• Watch verdict-mix drift as an unlabelled alarm. The proportion of removals per category is stable in a healthy system. A category whose removal rate moves ten per cent overnight is either a real event on your platform or a regression in your agent — and you want to know which within the hour, not at the next review.
• Do not use the same model family as agent and judge. An LLM judge sharing the agent's blind spots will confirm them enthusiastically. Where the stakes are this asymmetric, the judge of record is a human sample.

Related: security-operations agents for the sibling domain with adversarial input, human-in-the-loop for where the checkpoint belongs, and adapting a playbook for the method behind this one.