Deep-Dives / Training Agentic Models

Training Agentic Models

Post-training for agentic ability — SFT, rejection sampling, distillation, RLHF/RLAIF, RL for tool use, reward design.

  1. Prompt, Fine-Tune, or RL?
    The decision tree for changing agent behavior: prompting asks, SFT imitates, RL optimizes — pick the cheapest lever that closes the gap.
  2. RLHF & RLAIF
    Walking the RLHF pipeline stage by stage — SFT, reward model, PPO/GRPO/DPO — and what swapping human labels for an AI judge actually fixes.
  3. RL for Tool Use & Multi-Step Tasks
    Why RL over tool trajectories is hard: sparse terminal reward, credit assignment across steps, and why a trustworthy verifier is the whole game.
  4. Reward Design & Reward Hacking
    Reward hacking is a quantity you can watch grow during training, not a tail risk you find in review — and the instrument is a second verifier that measures the same intent by a different construction and never enters the reward. A 2026 experiment (arXiv:2604.15149) made it causal: the extensionally trained variant develops a widening train-minus-audit gap that emerges mid-run, while fuzzing found exploits within two verifier queries in 94 of 100 math trials, SpecBench found frontier agents saturating the visible suite with hacking intact underneath, and METR measured hacking in 30.4% of RE-Bench runs against 0.7% of HCAST. Covers why the KL leash bounds drift and not gaming, and why the alarm is the gap's slope rather than its level.
  5. SFT, Rejection Sampling & Distillation
    The supervised techniques that solve most agentic training problems before RL: rejection sampling, expert iteration, and distilling a strong agent into a cheap one.
  6. Process vs Outcome Reward Models
    Pay for the answer or pay for the steps: when dense process reward beats sparse outcome reward, and the labeling-cost trade that decides it.
  7. RLVR & GRPO for Agents
    The 2026 recipe — SFT → DPO/SimPO → GRPO/DAPO with verifiable rewards; entropy collapse, KL drift, and the multi-turn algorithms (ARPO, StepPO, Turn-PPO).
  8. RL Fine-Tuning Open Weights
    SageMaker RFT + TRL v1.0 + LlamaFactory + VeRL let teams GRPO on Qwen3 / Llama 4 / DeepSeek V4 with in-house verifiable rewards — the "custom reasoning model" playbook.
  9. Process Reward Models
    Step-level PRM vs outcome-only RLVR — dense credit assignment for long-horizon SWE agents, with SWE-TRACE, AgentPRM, SPARK as the current stack.
  10. DSPy 3 + GEPA for Agent Optimization
    GEPA (ICLR 2026 oral) outperforms MIPROv2 by 13% and RL/GRPO by 20% at 35x fewer rollouts — when to use each optimizer.
  11. Environment Engineering for Agentic RL
    Rollout time, not gradient time, sets the bill — one straggler trajectory idles a whole synchronous batch, so p99 episode latency and environment concurrency matter more than GPU count. And the verifier you wrote in an afternoon is the reward function, so buy precision over recall, pin the container digest not the tag, and note that the artifact doubles as the eval you needed anyway.
  12. Training on Your Own Traces
    Three filters nobody chose sit between production and your dataset — what users still attempt, what completed, what your retention window kept — and behaviour-cloning error compounds in the square of the horizon, so 99% per-step accuracy is not the 26% failure rate the independent-errors sum promises. Traces pay in this order: eval set, environment, prompt and tools, weights last; and a trace-tuned model encodes your tool catalogue, so renaming search_docs becomes a model problem.
  13. What Post-Training Does Not Remove
    Post-training moves a policy on the distribution it samples, so an association planted in pretraining and keyed on a string no alignment stage ever sees has no mechanism by which SFT, DPO or RL would find it. The dose is a document count, not a percentage: a 2025 UK AISI and Anthropic result reports ~250 poisoned documents backdooring models from 600M to 13B parameters, so a larger corpus lowers the fraction an attacker needs and not the count. The 2026 agentic version makes it a data-pipeline problem — one trigger is an ordinary path under /anthropic/ and declarations work as well as demonstrations — and its own repository warns that the headline figure is an any-of-32 string match, not a per-invocation rate. Ask about corpus provenance, not the alignment recipe, and buy controls that work without knowing the trigger.