Grading & Assessment Agents

Step 1Decide what the agent is allowed to emit, before anything else.

There are three postures, and they are not a maturity ladder — they are three different products with three different liability profiles. Pick one per assessment type and write it down, because the drift from the third to the first is what happens by default when throughput pressure arrives.

# Three postures, by what reaches the transcript

EVIDENCE ONLY        agent emits rubric-criterion findings with spans
                     human assigns every mark
                     saving: reading time, ~40-60% of marking effort
                     use for: anything summative, anything appealable

PROVISIONAL MARK     agent emits a score + evidence; human confirms
                     or overrides; human's action is the grade
                     saving: adds triage, confirm-without-change is fast
                     use for: high-volume formative, low-stakes quizzes

AGENT OF RECORD      agent's score is the grade
                     use for: practice sets with no transcript effect,
                     self-assessment, and nothing else

• The posture is per assessment, not per product. A weekly low-stakes quiz and a final essay are different legal objects even inside the same course, and the same agent can serve both if the posture is a configuration rather than a habit.
• Make "confirm" an explicit act with a cost. If confirming is one click on a pre-ticked box, you have shipped posture three while documenting posture two. Require the marker to touch the evidence — open one span, acknowledge one criterion — and measure the override rate as a live signal, exactly the way review queues handles any high-volume approval surface.
• Regulators have already specified this shape elsewhere. The hiring domain is the worked precedent — a countable, attributable per-candidate decision, which is why free-text "strong, 8/10" output fails an audit. Read hiring and recruiting agents for the architecture, then apply it here, because assessment is the same object with a different name.

Step 2The rubric is the tool contract, so decompose it until it is checkable.

Handing the model an essay and a holistic 1–6 scale produces a number with no defensible provenance. Handing it one criterion at a time, each returning a finding bound to a span of the submission, produces something a marker can audit in seconds and a student can argue with.

• One call per criterion, not one call per submission. Separate calls keep the criteria independent, which is what stops a strong thesis statement from inflating the citation-accuracy score. Holistic scoring correlates criteria that the rubric deliberately separated.
• Every finding carries a span. "Criterion 3, partially met" is unusable; "Criterion 3, partially met — the counter-argument appears at lines 41–48 but is not addressed" is a marking comment, an appeal exhibit, and feedback to the student in one object. Span-bound output is also what makes the agent's claims falsifiable rather than merely plausible.
• Split the mechanical criteria off entirely. Word count, citation format, required sections, submitted-on-time, code compiles, tests pass — these are deterministic checks and should never reach a model. Moving them out shrinks the model's surface and removes the most embarrassing error class.
• Interesting result worth knowing before you spend a month on rubric prose: published work on automated rubric refinement reports that a simple initial rubric, iterated against model behaviour, reaches agreement comparable to or better than a detailed hand-authored one. Write the short rubric, then improve it empirically.

Step 3The metric is agreement with adjudicated humans, and it is not accuracy.

There is no ground truth in marking; there are markers who disagree. So the target is agreement with a human standard, measured the way the assessment literature measures it, which means quadratic weighted kappa against double-marked and adjudicated scores — not accuracy against one marker's output.

# The agreement set you have to build first

  200-400 submissions, stratified by
    grade band, submission length, first-language status,
    cohort, and anything your rubric treats as neutral

  each marked independently by 2 humans
  disagreements adjudicated by a third -> the standard

# Then report, per stratum, not just overall

  QWK(agent, standard)
  QWK(human_1, human_2)        # the ceiling you are chasing
  mean signed error             # direction of bias
  rate of >=2-band disagreement # what appeals come from

• Read QWK against the human ceiling, never against 1.0. Reported LLM essay-scoring agreement clusters in the 0.6–0.8 "substantial" band — one study at 0.68, many-shot prompting reaching 0.82 against a 0.59 zero-shot baseline, a multi-model jury at 0.76 against 0.67 — while a small pilot on 20 essays landed at 0.46. The spread across prompting strategies is larger than the spread across models, which tells you where the engineering is.
• A high kappa can be the bad news. Researchers specifically warn that strong agreement may indicate the agent has learned a bias the human markers already had, and is now perpetuating it at volume. Agreement is necessary and not sufficient; the subgroup breakdown is where you find out.
• Stratify by first-language status explicitly. Published analyses find residual bias on features including current language-learner status. If you cannot report QWK per stratum, you cannot claim the agent is fair, and this is the stratum regulators and students will ask about first.
• Do not let QWK be your only number. The metric has known limitations as a sole criterion, and changing the kappa weighting changes what any acceptance threshold means. Pair it with the two-band disagreement rate, which is the quantity that actually generates appeals and reviewer load — see human baselines in agent evals for why the human number belongs in the report.

Step 4Refuse to build the cheating detector. This is the hill.

Somebody will ask for AI-text detection in the first stakeholder meeting, and the honest engineering answer is that the measurement does not exist at the error rate the decision requires. The canonical study ran seven GPT detectors over 91 practice TOEFL essays written before ChatGPT existed and recorded a mean false-positive rate of 61.3%, while the same tools classified 88 essays by US eighth-graders correctly.

• The mechanism explains why this will not be fixed by a better vendor. Detectors key on perplexity — how predictable the word choices are — and a writer working in a second language makes simpler, more predictable choices. The detector is measuring fluency and reporting it as authorship. Every control that keys on a proxy the subject can cheaply change belongs in this family.
• It fails in both directions at once. In the same study, asking a model to rewrite the AI text with "elevate the provided text by employing literary language" dropped detection to roughly 3%. So the tool is unreliable against the honest student and trivially evaded by the dishonest one — the worst possible pairing for a disciplinary process.
• What to build instead, and it is cheaper. Process evidence beats text forensics: draft history, revision timestamps, a two-minute oral or follow-up question generated from the student's own submission, and assessment designs where the artefact alone was never the evidence. The agent is genuinely good at the last one — generating a personalised follow-up question from a specific paragraph is a task it does well and a human rarely has time for.
• If the institution insists, insist back in writing. A detector score may inform a conversation; it must never be a finding, appear in a record, or trigger a process on its own. Put that constraint in the impact assessment, where it survives staff turnover.

Step 5Build the appeal path before the marking path.

Assessment is one of the few agent domains where the contest process is a legal feature rather than a nice-to-have, and retrofitting it is expensive because it dictates what you logged.

• A student must be able to learn that an agent was involved. Not buried in a handbook — on the feedback they receive, in one sentence, with the posture named. This is the ordinary disclosure obligation, and in assessment it is also what makes the appeal meaningful.
• Log the evidence, not just the score. An appeal six weeks later needs the criterion findings, the spans, the rubric version, the model and prompt version, and the identity of the human who confirmed. That is a decision receipt, and the retention period is the academic appeal window, not your default trace TTL — reconcile it with trace retention deliberately, because sampling a grading trace away is destroying evidence.
• Re-marking must be able to ignore the agent entirely. If the appeal route re-runs the same model with the same prompt, you have offered a formality. A contested submission goes to a human who has not seen the agent's output, which also makes every appeal a free addition to your agreement set — the pipeline contestability and appeals describes.
• Rubric versions are immutable. A grade is only defensible against the rubric that was live when it was issued. Version the rubric, pin it per cohort, and never edit in place.

Step 6Roll out on the double-marking budget you already have.

Most institutions already spend a fixed amount of human effort on double-marking a sample. That budget is the correct funding source and the correct control loop — it converts the deployment from a leap of faith into a measured substitution.

• Shadow-mark first, for one full cycle. The agent marks everything; nobody sees its output; you compare against the humans at the end. One cycle gives you the agreement numbers, the per-stratum breakdown, and the two-band disagreement rate without any student being affected — standard shadow mode.
• Then move the sample, not the marking. Keep total human effort flat and redirect it: less uniform double-marking, more targeted review of exactly the submissions where the agent and the first marker disagree, or where the agent's own criterion findings conflict. Disagreement is a better sampling key than randomness, and this is the step that actually buys capacity.
• Watch for drift at the cohort boundary, not weekly. Marking is seasonal: a new cohort, a reworded prompt, or a changed assignment resets the distribution. Re-run the agreement set at every cohort rather than on a calendar, and treat a model or prompt change as a new agreement run — the discipline in quality regression detection.
• Report the reviewer's time, because that is the business case. Minutes-per-submission and confirm-without-change rate are the two numbers that decide whether this saved anything; the hours themselves are not money until they relieve something, as fractional time savings spells out.