AI Blog

Tagged: model-comparison

← Back to AI Blog

8 min read

The leaderboard ranks the pair, not the model

A Stanford and UIUC team decomposed the variance in 22 agent benchmarks from the Holistic Agent Leaderboard and the Harbor Index, and the result splits along the claim you make. Ranking a fixed model-and-scaffold system is reliable: generalizability coefficients of 0.935 to 0.994. Ranking the underlying model from the same rollouts runs 0.148 to 0.841. And the standard remedy has a ceiling — where scaffold coverage dominates the uncertainty, even unlimited extra tasks of the same kind buy at most 0.097 more reliability. The number to ask a leaderboard for is how many scaffolds it ran, not how many tasks.

9 min read

Kimi K3 Is Open Weights. That Is Not the Same as Cheap, Local, or Unrestricted

Moonshot released 2.8 trillion parameters as a free download on 27 July — and priced its own API above the model it replaced, while no single GPU on the market can hold the weights. Open weights buy agent builders exactly one thing that closed APIs cannot, and it is not cost.

16 min read

Claude Mythos 5 vs GPT-5.6 vs Gemini 3.2 vs Qwen 3.7 vs DeepSeek V4.1: The June 2026 Frontier Refresh

Five frontier-tier models shipped inside a two-week window in June 2026. The differences are no longer about who tops MMLU — each lab is now betting on a different axis: agentic computer use, reasoning cost, multimodal latency, or pure price floor. Pick the axis before you pick the model.

17 min read

Llama 4 vs DeepSeek V3 vs Qwen3 vs Mistral Large 3: Four Open-Weights Flagships, Four Different Bets

Every few months, four labs ship a similar-sounding open-weights flagship — MoE, long context, reasoning mode, multimodal. The benchmarks keep getting passed back and forth. The thing that actually decides which one you run in production is the axis each lab is betting on next: multimodal ecosystem, inference economics, agentic reasoning, or permissive-license frontier intelligence.