A Stanford and UIUC team decomposed the variance in 22 agent benchmarks from the Holistic Agent Leaderboard and the Harbor Index, and the result splits along the claim you make. Ranking a fixed model-and-scaffold system is reliable: generalizability coefficients of 0.935 to 0.994. Ranking the underlying model from the same rollouts runs 0.148 to 0.841. And the standard remedy has a ceiling — where scaffold coverage dominates the uncertainty, even unlimited extra tasks of the same kind buy at most 0.097 more reliability. The number to ask a leaderboard for is how many scaffolds it ran, not how many tasks.
Kimi K3 Is Open Weights. That Is Not the Same as Cheap, Local, or Unrestricted
Moonshot released 2.8 trillion parameters as a free download on 27 July — and priced its own API above the model it replaced, while no single GPU on the market can hold the weights. Open weights buy agent builders exactly one thing that closed APIs cannot, and it is not cost.
Claude Mythos 5 vs GPT-5.6 vs Gemini 3.2 vs Qwen 3.7 vs DeepSeek V4.1: The June 2026 Frontier Refresh
Five frontier-tier models shipped inside a two-week window in June 2026. The differences are no longer about who tops MMLU — each lab is now betting on a different axis: agentic computer use, reasoning cost, multimodal latency, or pure price floor. Pick the axis before you pick the model.
Llama 4 vs DeepSeek V3 vs Qwen3 vs Mistral Large 3: Four Open-Weights Flagships, Four Different Bets
Every few months, four labs ship a similar-sounding open-weights flagship — MoE, long context, reasoning mode, multimodal. The benchmarks keep getting passed back and forth. The thing that actually decides which one you run in production is the axis each lab is betting on next: multimodal ecosystem, inference economics, agentic reasoning, or permissive-license frontier intelligence.