Agentic AI Wiki
Deep-Dives / Training Agentic Models

Training Agentic Models

Post-training for agentic ability — SFT, rejection sampling, distillation, RLHF/RLAIF, RL for tool use, reward design.

  1. Prompt, Fine-Tune, or RL?
    The decision tree for changing agent behavior: prompting asks, SFT imitates, RL optimizes — pick the cheapest lever that closes the gap.
  2. RLHF & RLAIF
    Walking the RLHF pipeline stage by stage — SFT, reward model, PPO/GRPO/DPO — and what swapping human labels for an AI judge actually fixes.
  3. RL for Tool Use & Multi-Step Tasks
    Why RL over tool trajectories is hard: sparse terminal reward, credit assignment across steps, and why a trustworthy verifier is the whole game.
  4. Reward Design & Reward Hacking
    The reward is always a proxy: concrete agent reward-hacking patterns, the KL leash to the base policy, and the discipline of auditing the top, not the mean.
  5. SFT, Rejection Sampling & Distillation
    The supervised techniques that solve most agentic training problems before RL: rejection sampling, expert iteration, and distilling a strong agent into a cheap one.
  6. Process vs Outcome Reward Models
    Pay for the answer or pay for the steps: when dense process reward beats sparse outcome reward, and the labeling-cost trade that decides it.
  7. RLVR & GRPO for Agents
    The 2026 recipe — SFT → DPO/SimPO → GRPO/DAPO with verifiable rewards; entropy collapse, KL drift, and the multi-turn algorithms (ARPO, StepPO, Turn-PPO).
  8. RL Fine-Tuning Open Weights
    SageMaker RFT + TRL v1.0 + LlamaFactory + VeRL let teams GRPO on Qwen3 / Llama 4 / DeepSeek V4 with in-house verifiable rewards — the "custom reasoning model" playbook.
  9. Process Reward Models
    Step-level PRM vs outcome-only RLVR — dense credit assignment for long-horizon SWE agents, with SWE-TRACE, AgentPRM, SPARK as the current stack.
  10. DSPy 3 + GEPA for Agent Optimization
    GEPA (ICLR 2026 oral) outperforms MIPROv2 by 13% and RL/GRPO by 20% at 35x fewer rollouts — when to use each optimizer.
  11. Environment Engineering for Agentic RL
    Rollout time, not gradient time, sets the bill — one straggler trajectory idles a whole synchronous batch, so p99 episode latency and environment concurrency matter more than GPU count. And the verifier you wrote in an afternoon is the reward function, so buy precision over recall, pin the container digest not the tag, and note that the artifact doubles as the eval you needed anyway.