Training Agentic Models

Prompt, Fine-Tune, or RL?

The decision tree for changing agent behavior: prompting asks, SFT imitates, RL optimizes — pick the cheapest lever that closes the gap.

RLHF & RLAIF

Walking the RLHF pipeline stage by stage — SFT, reward model, PPO/GRPO/DPO — and what swapping human labels for an AI judge actually fixes.

RL for Tool Use & Multi-Step Tasks

Why RL over tool trajectories is hard: sparse terminal reward, credit assignment across steps, and why a trustworthy verifier is the whole game.

Reward Design & Reward Hacking

The reward is always a proxy: concrete agent reward-hacking patterns, the KL leash to the base policy, and the discipline of auditing the top, not the mean.

SFT, Rejection Sampling & Distillation

The supervised techniques that solve most agentic training problems before RL: rejection sampling, expert iteration, and distilling a strong agent into a cheap one.

Process vs Outcome Reward Models

Pay for the answer or pay for the steps: when dense process reward beats sparse outcome reward, and the labeling-cost trade that decides it.

RLVR & GRPO for Agents

The 2026 recipe — SFT → DPO/SimPO → GRPO/DAPO with verifiable rewards; entropy collapse, KL drift, and the multi-turn algorithms (ARPO, StepPO, Turn-PPO).

RL Fine-Tuning Open Weights

SageMaker RFT + TRL v1.0 + LlamaFactory + VeRL let teams GRPO on Qwen3 / Llama 4 / DeepSeek V4 with in-house verifiable rewards — the "custom reasoning model" playbook.

Process Reward Models

Step-level PRM vs outcome-only RLVR — dense credit assignment for long-horizon SWE agents, with SWE-TRACE, AgentPRM, SPARK as the current stack.

DSPy 3 + GEPA for Agent Optimization

GEPA (ICLR 2026 oral) outperforms MIPROv2 by 13% and RL/GRPO by 20% at 35x fewer rollouts — when to use each optimizer.