E-discovery & document review agents.
The best published comparison of generative AI against a managed review workflow found the AI reached roughly 88% recall at 29% precision while the managed workflow reached 64% recall at 39% — so the agent found a third more of the responsive documents and handed back nearly twice as many documents to look at. That is the whole engineering problem in one sentence: you did not buy a cheaper review, you bought a different cost allocation, and the artefact you have to defend is not the model's score but a measured, documented elusion test on the pile you decided to throw away. Build that measurement harness first and the model becomes swappable.
Step 1The two numbers that have actually been measured.
Most claims in this market are vendor-reported. One study is not: a 2026 Redgrave LLP working paper ran a comparison on a 45,004-document population drawn from the UCSF Opioid Industry Documents Archive — a public corpus, so the exercise is reproducible in a way vendor case studies are not. The headline figures, as summarised in October 2026 trade coverage of that paper:
- Generative AI review (Relativity's aiR for Review): 87.7% recall, 28.6% precision.
- Managed review using an active-learning classifier (Relativity Assisted Learning): 64.4% recall, 39.2% precision.
- The paper's own argument is that a properly prompted generative workflow can be fitted into the established technology-assisted-review reference model — in other words, that the defensibility framework does not have to be reinvented, only re-instrumented.
Treat the exact decimals as one study on one corpus with one prompt, and the shape as the transferable finding. The shape is an inversion: the generative reviewer is substantially more sensitive and substantially less specific. It pulls in more of what you were looking for and much more of what you were not. Every downstream decision in this playbook follows from that single asymmetry, and the common mistake is to read the recall number alone and conclude the review got cheaper.
Vendor-reported figures in this space cluster around 90–95% recall at 80%+ precision. Those are not comparable to the numbers above, because precision depends entirely on how the responsiveness criteria were drawn and how rich the corpus is. When a vendor quotes precision, ask for the prevalence of responsive documents in the test population; a high precision number on a 40%-rich corpus tells you nothing about your 8%-rich one.
Step 2Price the recall before you buy it, because the cost reappears downstream.
Work the asymmetry through on a concrete matter. 100,000 documents after de-duplication and culling; assume 12% are truly responsive, so 12,000 responsive documents exist. Apply each set of operating characteristics:
# 100,000 documents, 12,000 truly responsive (12% prevalence) recall prec. surfaced responsive missed non-resp found in pile surfaced generative review 88% 29% 36,414 10,560 1,440 25,854 active-learning TAR 64% 39% 19,692 7,680 4,320 12,012 # surfaced = responsive_found / precision delta: +2,880 responsive documents found, +16,722 documents to review # at a second-pass review cost of $1.00/doc (your number will differ): # +$16,722 of review buys 2,880 more responsive documents # = ~$5.80 per additional responsive document recovered
That last line is the number to put in front of whoever signs the budget, and it is the number nobody quotes. Sometimes $5.80 a document is obviously worth it — a matter where a missed document is a sanctions exposure or a case-dispositive admission. Sometimes it obviously is not. The point is that it is a decision with a price, and the recall figure alone hides the price completely.
Two adjustments make the estimate honest. First, the second pass over the surfaced set is not full-rate first-pass review; a human confirming or rejecting a model call with a model-written rationale attached is faster, which is the effect measured in the cost of human review. Second, token cost is real but usually secondary: a document review is a per-document inference over a long input, so the bill scales with corpus size times document length and is dominated by the input side — see unit economics for the shape. Run the arithmetic on your actual prevalence before the pilot, not after.
Step 3The deliverable is a tested discard set, and its sample size does not scale with the corpus.
Here is the structural insight that most teams arrive at too late. Nobody will ever audit your classifier. What gets scrutinised is the pile you decided not to produce, and the only evidence about that pile is a random sample of it. The measurement is called elusion: the responsive rate among documents the process discarded.
Continue the example. The generative run discards 63,586 documents containing 1,440 responsive ones — an elusion rate of 2.26%. The TAR run discards 80,308 containing 4,320 — 5.38%. Those two percentages, not the recall figures, are what a protocol dispute is actually about.
And the sample you need to bound them is set by the bound, not by the corpus:
# elusion sampling: n is driven by the rate you must rule out, # NOT by how many documents are in the discard pile. sample 500 from the discard set, find 0 responsive -> 95% upper bound on elusion ~= 3/500 = 0.6% # "rule of three" sample 500, true elusion is 2.26% -> expect ~11 responsive documents. you will find them. sample 500, true elusion is 0.5% -> expect ~2.5. too noisy to act on; n must rise to ~2,000 # corollary: a 10-million-document discard pile needs the same # sample as a 60,000-document one. "the corpus is too big to # validate" is not a statistical claim, it is a budget claim.
So the protocol writes itself in the right order: decide the elusion ceiling you will accept before the run, derive the sample size from it, draw the sample randomly (not by date, not by custodian, not by the model's confidence), have humans code it blind, and compute the bound. If the bound exceeds the ceiling, you remediate — lower the responsiveness threshold, re-prompt, re-run the affected custodians — and you sample again. That loop is the product. The model is a component inside it. Threshold-setting as a general discipline is decision-model thresholds; the labelling operation underneath it is annotation and labeling ops.
Step 4The responsiveness criteria are the new seed set, so version them like code.
In an active-learning workflow the contested artefact was the seed set — which documents were coded, by whom, to what standard. In a generative workflow the contested artefact is the criteria document: the written description of what responsive means for this request, which becomes the prompt. It is now the single highest-leverage object in the review, and it is usually stored in somebody's chat history.
- Treat it as a versioned file. One file per request for production, in source control, with a date and an author. A change to the criteria invalidates every coding decision made under the old text, exactly as a change to a seed set would.
- Pin the model and its version alongside it. A silent provider-side model update changes your operating characteristics mid-review and nothing in the output says so; this is the failure mode in unpinned vendor defaults, and in a review it means two halves of one production were coded by two different reviewers.
- Record the per-document rationale, not just the call. A responsive/not-responsive bit is unreviewable. A one-sentence citation to the passage that triggered the call is what makes the second pass fast and the validation sample codeable, and it is the raw material for any later challenge — the record-keeping argument of audit trails.
- Re-run the validation set on every criteria change. Keep a fixed, human-coded set of a few hundred documents spanning the hard calls, and treat it as a regression suite in the sense of evals. Criteria edits are prompt edits, and prompt edits regress.
One upstream dependency decides more than any of this: whether the text the model read was the text in the document. Scanned exhibits, spreadsheets, email threads with quoted history, and embedded attachments all fail differently, and a document whose extraction dropped the table is a document the model never saw. Measure extraction fidelity separately before you measure review accuracy — document parsing for RAG covers the failure modes, and Marker vs MarkItDown vs MinerU vs Docling compares the extractors.
Step 5Privilege is a different problem, and no recall target covers it.
Responsiveness tolerates error statistically: you argue about a bound and you remediate. Privilege does not work that way. A single inadvertently produced privileged document is a discrete event with its own consequences, so the acceptable error rate is not a percentage you negotiate — which means the architecture has to be different, not just the threshold.
- Let the model rank, not decide. On the privilege pass the useful output is a calibrated score plus the reason — counsel named, legal advice sought, work product indicated — feeding a queue a lawyer works through. The decision stays with the person who signs. This is the human-in-the-loop pattern, implemented as a review queue rather than an approval dialog.
- Run it as a separate pass with its own criteria. Folding privilege detection into the responsiveness prompt makes both worse and makes neither measurable. Two passes, two criteria documents, two validation sets.
- Use the model for the thing it is unusually good at: the privilege log. Drafting log entries from the document's own metadata and text is high-volume, structured, verifiable work with a human check at the end — a much better fit than the call itself.
- Watch for the confident-wrong case specifically. The model that fabricates a plausible reason for a privilege call is the dangerous one, because the rationale is what the reviewer trusts; see hallucination and grounding and require every rationale to quote the document.
What "adequate supervision" requires varies materially by jurisdiction and forum, and this is not legal advice. The framing that holds generally: in adversarial civil discovery a human signs the production and certifies that the search behind it was reasonable, and no tool transfers that responsibility. Build so that the person signing can describe the process, show the validation result, and explain the remediation — and do not cite rule numbers, bar opinions or case names in your protocol unless counsel on the matter has verified them.
Step 6Ship it as a measured pilot, in this order.
The sequence matters because each step produces the evidence the next one needs, and because the expensive mistake — running the whole corpus and then discovering the criteria were wrong — is entirely avoidable.
- Start from a corpus somebody already coded. A closed matter with human coding decisions is a ground-truth set you already own. Run the agent against it and you get recall, precision and elusion on your own documents, your own extraction pipeline and your own criteria style — which is worth more than every published benchmark, for the reason set out in human baselines in agent evals.
- Write the acceptance criteria into the protocol before the first production run — elusion ceiling, sample size, who codes the sample, what triggers remediation. Where the forum and the other side allow it, agreeing the validation protocol in advance converts a later fight about methodology into a fight about numbers, which is a much better fight to have.
- Hold back a blind slice. Reserve a few hundred documents the model never sees until the end, coded independently, and use them as the final check. A validation set that has been iterated against is a training set.
- Produce a validation report as a deliverable, not a by-product. Population size, prevalence estimate, criteria version, model and version, recall and precision on the control set, elusion bound on the discard set, remediation taken. One page. It is the artefact that makes the review defensible and the only one anybody will read.
- Keep the whole chain under the matter's retention rules. Criteria versions, prompts, per-document rationales, sample codings and the report are all now part of the record — see retention and legal hold, and note that model-provider retention of your document text is its own diligence question.
If you do one thing from this page, do the back-test: take a closed matter that was reviewed by humans, run the agent over it, and compute recall, precision and elusion against the existing coding. That single run tells you your real prevalence, your real precision at your real criteria, and therefore the actual dollar cost of the extra recall — the three numbers that decide whether to deploy, none of which any vendor can tell you. Then build the elusion sampling loop before you run a live matter, because the discard set is what you will have to defend, and a sample drawn after the fact from a pile you have already produced against is worth very little. For the broader domain framing — where legal agents work, where they fabricate, and the supervision landscape — see legal agents.