Fraud & Disputes Agents

The cost function is upside down from the one everyone builds.

Start with the asymmetry, because every later decision is downstream of it. Nilson Report figures published in January 2026 put global card fraud losses at $33.41bn for 2024. Industry projections for the other side of the ledger — legitimate orders wrongly declined — run above $231bn for 2026, and Javelin's widely cited estimate puts the cost of false declines at roughly 13× actual fraud. The average false-decline rate sits near 1.5% of e-commerce sales, and studies of declined-order populations find that somewhere between 30% and 70% of them were good.

Read what that does to your objective. A model that stops one more fraudulent $200 transaction by tightening a threshold that also declines eleven good ones has lost money, and the loss lands in a place your fraud dashboard does not display: the customer who was declined at checkout, tried a different card, and does not come back. So:

• Denominate every decision in two currencies, not one. Expected fraud loss avoided, and expected value of good customers refused. If your review queue is ranked by fraud score alone, it is ranked by half the objective.
• Make the threshold a business input, not a model output. The score is a probability; the cut-point is a price. Whoever owns revenue should own the cut-point, and should be shown what it costs in declined good customers — see calibration, because an uncalibrated score makes this arithmetic impossible.
• Segment before you threshold. A $12 repeat purchase from a four-year-old account and a $4,000 first order to a freight forwarder do not belong on the same scale, and one global threshold silently over-declines the first to control the second.

Your declines generate no labels. Buy them on purpose.

Here is the structural problem that no amount of model quality fixes. An approved transaction eventually tells you whether it was fraud: a chargeback arrives, or it does not. A declined transaction tells you nothing, ever. It never settles, so it can never be disputed, so it produces no outcome to learn from. Every model you train and every eval you run is fitted on the population your existing system chose to approve — and the more accurate that system gets, the more skewed the training data becomes. This is the selective-labels problem, and in fraud it is not a subtlety, it is the whole measurement.

There is exactly one honest fix, and it costs money by design: approve a small random sample of transactions you would have declined, and eat the fraud.

• Size it as a budget line, not as an experiment. A fraction of a per cent of declines, capped in absolute dollars per month, with the expected loss written down in advance and signed off by whoever owns the P&L. Presented as "the cost of knowing our false-decline rate", it is a defensible number; presented as an engineering request, it will be refused.
• Exclude what you must, and say so. Confirmed-compromise cards, sanctions hits and anything with an AML dimension never go in the holdout. The sample is drawn from the ordinary score-based declines, which is where the false positives live anyway.
• It is the only source of the number you actually manage. Fraud rate on the approved population is measurable without it; false-decline rate is not measurable at all without it. Teams that skip this step end up estimating their most expensive error from customer complaints, which under-counts it by the ratio of people who complain to people who leave.

Everything else — reason-code analysis, card-testing detection, the agent's own reasoning traces — is supporting evidence. The holdout is the ground truth, and it is the reason this domain cannot borrow the evaluation playbook from anywhere else; see online vs offline evals for the general form of the problem.

The calendar is a design constraint, and it is longer than your dashboard.

Write the real timeline on the wall before agreeing to any metric:

• Up to 120 days for the dispute to be raised. Visa and Mastercard rules generally give a cardholder 120 days from the transaction date — or from the expected delivery date for goods not received — to initiate a dispute. So a decision you make today has an outcome that is not final until roughly four months from today.
• 10 business days, then 45, on the inbound side. Under Regulation E, a US financial institution investigating an alleged error must resolve it within 10 business days or provisionally credit the customer and take up to 45 days — extended to 90 days for point-of-sale, foreign-initiated, and new-account transfers. These are legal deadlines with a clock that starts on notice, and they make dispute intake an SLA problem before it is an intelligence problem.

Three consequences follow, and each of them breaks a habit imported from other agent projects:

• There is no online eval on the metric that matters. Whatever your dashboard says about this week's fraud rate is a partially observed cohort. Report by decision cohort with an explicit maturity age — "the 60-day-matured cohort" — and refuse to compare a two-week-old cohort against a mature one. Anyone reading the freshest number is reading the number that is most wrong, and it is always wrong in the flattering direction.
• A/B tests take a quarter. Plan for it. What you can read early is behaviour — approval rate, queue depth, checkout abandonment, analyst overturn rate — and those are leading indicators, not outcomes. Treat them as the production feedback signals they are.
• Retraining on unmatured data trains on optimism. A pipeline that pulls "the last 30 days" is systematically labelling recent fraud as legitimate, because its chargeback has not arrived yet. Exclude the immature window or weight it, and never let an automated retrain job define the window for itself.

Keep the agent out of the authorisation path. Put it in the case.

The real-time decision is not the place for a language model, and the reason is a latency budget rather than a capability argument: an issuer authorisation response is expected in the low hundreds of milliseconds, of which the risk decision gets a slice measured in tens. That is a gradient-boosted model on tabular features, and it should stay one — deterministic, calibrated, fast, and explainable in the terms a regulator asks about.

What is genuinely broken is everything around that decision, and it is all evidence assembly:

• The investigation. An alert arrives as a score and an id. Someone then opens six systems — device history, prior disputes, linked accounts, merchant history, KYC file, the customer's contact record — and reconstructs a story. That retrieval-and-synthesis job is exactly what an agent is good at, and it is where the analyst's hour actually goes.
• The case file. The output is a written narrative with citations to the records it rests on, a recommendation, and a confidence. Grade it the way you would grade any analyst: would a competent reviewer reach the same conclusion from this file alone? If the file does not support the recommendation, the run failed, regardless of whether the recommendation was right.
• The dispute response. On the merchant side, representment is a document-assembly problem with a rulebook — the network's compelling-evidence requirements specify what counts, and most losses are procedural rather than substantive. An agent that assembles the evidence bundle to the reason code's own checklist is doing unambiguous work.

Two documents, never one — and other places the compliance surface is sharp.

This domain has legal edges that most agent playbooks never touch, and one of them will bite an ordinary summarisation feature:

• The customer-facing message and the internal case note are different documents and must be generated separately. Where a suspicious activity report is involved, US law makes the existence of that report confidential — you may not disclose it to the customer, directly or by implication. An agent that helpfully drafts one narrative and reuses it for the customer notification is a tipping-off incident with a template. Enforce it structurally: separate prompts, separate outputs, a hard rule that internal case text can never be a source for customer text.
• Adverse decisions need a reason a human can defend. Where the decision is a credit decision, notice requirements attach; more broadly, an automated decision with a significant effect on a person is the category regulators are most interested in, and "the model said so" is not a reason. Keep the human approval on the adverse path, and keep it real — an approver who cannot see the evidence is a rubber stamp with a name attached, which is worse than no approver at all. See accountability and roles.
• Log the decision, its inputs, and the model versions, at decision time. Four months later, when the dispute lands, "why did we approve this" must be answerable from the record, not reconstructed from a model you have since retrained. This is the audit trail requirement with a long tail attached.
• Do not let the agent be the channel for a customer's fraud claim. Reg E's clock starts on notice, and notice can be oral. An intake agent that "handles" a complaint without registering it as a notice of error has not deflected a ticket, it has started an unlogged legal timer.

Related territory: KYC and AML onboarding agents for the identity side of the same file, and security operations agents for the alert-triage pattern this shares.

The scoreboard, and the one number nobody wants to publish.

Four numbers, reported together, on matured cohorts:

• False-decline rate, estimated from the approve-anyway holdout. The number this whole playbook exists to make computable. Publish it beside the fraud rate, always, because a fraud rate shown alone can be improved to zero by declining everyone.
• Net decision value. Fraud loss avoided minus the estimated value of good customers declined, per thousand decisions. This is the only number in which a threshold change can be evaluated honestly, and it is frequently negative for the last increment of tightening that someone shipped as a win.
• Analyst agreement on the agent's case files. The share of cases where a reviewer, given the agent's file, reaches the agent's recommendation — measured on a sample re-reviewed blind. Falling agreement is the earliest signal that retrieval has drifted, and it moves months before the chargeback data does.
• Reg E clock compliance. Provisional credits issued within 10 business days; investigations closed within 45 or 90. Boring, binary, and the one that becomes an examination finding.

Cases closed, alerts triaged and hours saved are vanity metrics here in a specific way: all three improve when the agent gets more decisive, and getting more decisive is exactly what the asymmetry in Step 1 says you should be cautious about.

Related: finance agents for the reconciliation discipline, quality regression detection for catching drift without waiting for outcomes, and human in the loop for what an approval gate has to include to be worth its latency.