Deep-Dives / Training Agentic Models
Training Agentic Models
Post-training for agentic ability — SFT, rejection sampling, distillation, RLHF/RLAIF, RL for tool use, reward design.
- Prompt, Fine-Tune, or RL?The decision tree for changing agent behavior: prompting asks, SFT imitates, RL optimizes — pick the cheapest lever that closes the gap.
- RLHF & RLAIFWalking the RLHF pipeline stage by stage — SFT, reward model, PPO/GRPO/DPO — and what swapping human labels for an AI judge actually fixes.
- RL for Tool Use & Multi-Step TasksWhy RL over tool trajectories is hard: sparse terminal reward, credit assignment across steps, and why a trustworthy verifier is the whole game.
- Reward Design & Reward HackingReward hacking is a quantity you can watch grow during training, not a tail risk you find in review — and the instrument is a second verifier that measures the same intent by a different construction and never enters the reward. A 2026 experiment (arXiv:2604.15149) made it causal: the extensionally trained variant develops a widening train-minus-audit gap that emerges mid-run, while fuzzing found exploits within two verifier queries in 94 of 100 math trials, SpecBench found frontier agents saturating the visible suite with hacking intact underneath, and METR measured hacking in 30.4% of RE-Bench runs against 0.7% of HCAST. Covers why the KL leash bounds drift and not gaming, and why the alarm is the gap's slope rather than its level.
- SFT, Rejection Sampling & DistillationThe supervised techniques that solve most agentic training problems before RL: rejection sampling, expert iteration, and distilling a strong agent into a cheap one.
- Process vs Outcome Reward ModelsPay for the answer or pay for the steps: when dense process reward beats sparse outcome reward, and the labeling-cost trade that decides it.
- RLVR & GRPO for AgentsThe 2026 recipe — SFT → DPO/SimPO → GRPO/DAPO with verifiable rewards; entropy collapse, KL drift, and the multi-turn algorithms (ARPO, StepPO, Turn-PPO).
- RL Fine-Tuning Open WeightsSageMaker RFT + TRL v1.0 + LlamaFactory + VeRL let teams GRPO on Qwen3 / Llama 4 / DeepSeek V4 with in-house verifiable rewards — the "custom reasoning model" playbook.
- Process Reward ModelsStep-level PRM vs outcome-only RLVR — dense credit assignment for long-horizon SWE agents, with SWE-TRACE, AgentPRM, SPARK as the current stack.
- DSPy 3 + GEPA for Agent OptimizationGEPA (ICLR 2026 oral) outperforms MIPROv2 by 13% and RL/GRPO by 20% at 35x fewer rollouts — when to use each optimizer.
- Environment Engineering for Agentic RLRollout time, not gradient time, sets the bill — one straggler trajectory idles a whole synchronous batch, so p99 episode latency and environment concurrency matter more than GPU count. And the verifier you wrote in an afternoon is the reward function, so buy precision over recall, pin the container digest not the tag, and note that the artifact doubles as the eval you needed anyway.
- Training on Your Own TracesThree filters nobody chose sit between production and your dataset — what users still attempt, what completed, what your retention window kept — and behaviour-cloning error compounds in the square of the horizon, so 99% per-step accuracy is not the 26% failure rate the independent-errors sum promises. Traces pay in this order: eval set, environment, prompt and tools, weights last; and a trace-tuned model encodes your tool catalogue, so renaming search_docs becomes a model problem.
- What Post-Training Does Not RemovePost-training moves a policy on the distribution it samples, so an association planted in pretraining and keyed on a string no alignment stage ever sees has no mechanism by which SFT, DPO or RL would find it. The dose is a document count, not a percentage: a 2025 UK AISI and Anthropic result reports ~250 poisoned documents backdooring models from 600M to 13B parameters, so a larger corpus lowers the fraction an attacker needs and not the count. The 2026 agentic version makes it a data-pipeline problem — one trigger is an ordinary path under /anthropic/ and declarations work as well as demonstrations — and its own repository warns that the headline figure is an any-of-32 string match, not a per-invocation rate. Ask about corpus provenance, not the alignment recipe, and buy controls that work without knowing the trigger.