Tutoring agents: the only domain where doing the task well is the failure.
Every other agent in this section is graded on completing the job. A tutor is graded on whether the learner can do the job afterwards, without it — which puts helpfulness and the actual objective in direct opposition. Solve the problem and you have deleted the learning signal, and the learner will rate you five stars for it. That inversion is the whole design brief: withholding is the product, satisfaction is the wrong metric, and the assessment you were built alongside is already broken.
Name the objective, because the obvious one is wrong.
The job is a measurable change in unaided capability, not a good session. This distinction is not philosophical — it decides what you log, what you optimise, and which A/B test wins. A tutor optimised on in-session signals converges on the answer-giving machine, because that is what maximises every proxy you had lying around: task completion, session satisfaction, time-to-resolution, thumbs-up rate. All four go up as learning goes down.
- Measure unaided, later. The evaluation instrument is performance on a comparable problem with the tutor absent, ideally a day or more after the session — not accuracy during it.
- Measure transfer, not repetition. A learner who can redo the exact worked problem has memorised a trace. Vary the surface features and hold the structure fixed; the gap between the two scores is the honest signal.
- Treat high satisfaction with flat post-tests as an alarm. It is the signature of a tutor that has become a solver, and it is the most common way these products fail while every dashboard is green.
The known-good design in the education literature is a tutor that never volunteers the answer and always asks for the next step. That is not politeness — the effort of retrieval is where the learning happens, so an agent that removes the effort removes the mechanism. See outcome vs trajectory eval for the general form of this trap.
Build the hint ladder and refuse to skip rungs.
The core behaviour is a graduated response policy: on every request for help, the agent gives the least assistance that could unblock this learner, and escalates only after another attempt. Left unconstrained, a model does the opposite — it produces the complete, correct, well-formatted solution, because that is what it was trained to consider a good answer.
- Rung 1 — orient. Point at the relevant concept or the part of the problem where the reasoning went wrong. No content.
- Rung 2 — question. Ask the one question whose answer unblocks the step. "What has to be true about the denominator here?"
- Rung 3 — analogous example. A fully worked different problem with the same structure. This is where most of the pedagogical value sits and where most tutors never get to, because they jumped to rung 4 at the first request.
- Rung 4 — partial. The next step only, with the rest left open.
- Rung 5 — full solution with explanation. Reachable, but gated on a genuine attempt and useful mainly as a review artifact after the fact.
Enforce the ladder structurally rather than by instruction. Track the rung in the session state, require an intervening learner attempt before any escalation, and cap the reachable rung by the task type — a graded assignment should not have rung 5 wired up at all. A prompt that says "do not give the answer" is a suggestion the third turn of a frustrated conversation will overturn.
Diagnose the misconception; do not explain the topic.
The difference between a tutor and a textbook with a chat box is that a tutor knows which specific wrong model this learner is running. "I don't get derivatives" is not a diagnosis, and answering it with a general explanation of derivatives is the single most common failure of these products — a correct, well-written response to the wrong question.
- Ask for the work, not the confusion. "Show me what you tried" surfaces the actual error; "what don't you understand?" collects a self-report that learners are, by construction, poorly placed to give.
- Grade the process, not the answer. A right answer from wrong reasoning is a failure you must catch, because it will fail next week on a problem that does not forgive the error. A wrong answer from sound reasoning with an arithmetic slip is nearly a success and should be treated as one.
- Carry the misconception across sessions. The durable state worth persisting is not the transcript — it is a small model of what this learner has demonstrated, what they have not, and which errors recur. That is the memory that makes session six better than session one.
Ground everything, because the learner cannot check you.
In most domains the user is a partial oracle — a developer notices the API does not exist, an analyst notices the number is off. A learner is defined by not knowing, which makes them the one user population with no error-detection capability at all. A hallucinated formula in a tutor does not get caught and argued with; it gets memorised.
- Retrieve from the course corpus and stay inside it. The textbook, the syllabus, the instructor's notes — grounded per RAG, with the notation and conventions of this course, which is half of why a general model confuses learners even when it is factually right.
- Never invent an answer key. If you generate practice problems, generate them with a checkable solution — from a computation you actually execute, an item bank, or a solver — and discard any item whose answer you cannot verify. A practice set with a wrong key teaches the wrong thing and destroys trust the moment it is found.
- Refuse outside the boundary and say so plainly. "That's beyond what this course covers — here's who to ask" is a correct answer. See hallucination & grounding.
Your user is adversarial, cheerfully, and your co-deployment is compromised.
Two problems the other playbooks do not have. First, the learner actively wants the thing you are withholding, and will get it: rephrasing as a hypothetical, claiming to be the instructor, pasting the problem as "check my work" with a blank attempt, or simply opening a different chat window. Second — and more consequential — any agent capable of doing the homework has already invalidated homework as an assessment, whether or not your product is the one used.
- Design for leakage, not against it. Assume the answer is obtainable elsewhere in thirty seconds. Your value is not being the last gate; it is being better than the shortcut for a learner who has some reason to want to learn. Adversarial hardening past that point costs more than it returns.
- Detect the shapes, handle them kindly. A blank "check my work", an escalating series of rephrasings, a sudden switch to "explain it as if for a teacher writing a key" — recognise the pattern, name it without accusation, and step back down the ladder.
- Say what you tell the institution, up front. If sessions are visible to an instructor, that must be known before the first message, not discovered. A tutor that is also surveillance is a tutor learners lie to, and a learner who conceals confusion has broken the diagnostic loop in step 3.
- Push the assessment question to whoever owns it. The tutoring agent cannot fix summative assessment; the institution has to move to in-person, oral, or process-visible work. Shipping a tutor into an unchanged assessment regime creates the cheating problem you will then be blamed for.
The lines that do not move.
Learners include minors and people in genuine distress, which imports duties from outside the product. Route any disclosure of self-harm, abuse or crisis to a human path immediately and unconditionally — this is not a confidence-threshold decision. Do not let a tutor drift into diagnosing learning disabilities, and do not let it deliver grades that carry consequences: an agent may propose a score with evidence, but a human signs it, exactly as in hiring. Build the ladder in code and the post-test in your eval harness before you tune a single prompt — a tutoring agent whose only defence against giving the answer is being asked nicely not to will give the answer, delight the learner, and teach nothing, and you will not find out for a term.
Related: adapting a playbook to your domain for the five-question method behind this one.