Nate's core claim, repeated and sharpened across two years of releases, is that "which model is best" stopped being the right question once every frontier lab cleared the floor for easy tasks. What's left is a fragmenting capability surface, and the real skill is routing: matching a specific job to the model whose optimization target fits it, then verifying the output yourself.
His method evolved from ad hoc side-by-sides (Claude 3.7 vs. six rivals on a Monopoly-simulator prompt, o3 vs. Gemini 2.5 on invented "resonance" tests) into a repeatable stress-test doctrine: give models deliberately hostile, real-world-shaped work — poisoned CSVs with duplicates and SQL injection, a 465-file small-business "shoebox" migration seeded with canary records (Mickey Mouse, a fake $25,000 payment), a handwritten Christmas-tree tally sheet — because clean benchmarks reward answer-key memorization (Grok 4's #1 leaderboard score collapsed to real-world garbage on identical tasks) while messy tests expose a model's actual stance toward ambiguity. He names three recurring stances: Claude reconstructs faithfully and flags uncertainty; GPT imposes structure and will fabricate coherence when reality won't cooperate; Gemini interprets for insight but lets its own outputs contradict each other.
The throughline that hardened by 2026: capability gains no longer show up as "smarter," they show up as narrower and more literal. Newer models (GPT-5.x, Opus 4.6+) increasingly do exactly what you ask and nothing more — the inference that used to paper over sloppy prompts is gone, so prompts must become specs (role, phases, deliverables, constraints, checkability) rather than wishes. Self-grading is unreliable in both directions (Opus oversells itself, GPT undersells), so peer review — one model checking another's work — is the only pattern he trusts for anything with stakes. Compute availability and harness (Codex, Claude Code, Cowork) now matter as much as raw weights; a slightly weaker model you can actually run beats a stronger one throttled by caps. And with Chinese open-weight models (DeepSeek, Kimi, GLM) closing the gap on price and matching mid-tier benchmarks, the operative question shifted again: not "is it good enough" in the abstract, but "what does an accepted result cost end-to-end" — tokens plus retries plus review time plus what breaks if you have to switch models later.
Key moves
Stop asking "which model is best" — ask "which model belongs in this specific job," then test on your own real, messy work, not benchmarks.
Build (or borrow) a hostile stress test per domain: planted duplicates, fake/canary records, corrupted files, contradictory data — clean test sets hide exactly the failures that matter in production.
Learn each model's default stance toward ambiguity: Claude reconstructs and flags uncertainty; GPT/ChatGPT imposes clean structure and will fabricate confidence; Gemini interprets fast across huge context but lets outputs drift out of sync with each other.
Treat prompts as specs, not wishes, on newer/more-literal models: state role, phases, required deliverables, constraints, and what proves the work is done.
Never trust a model's self-review of its own work; use peer review (a second model, or a human, grading the first) since self-assessment is unreliable in both directions.
Route by job shape, not brand loyalty: unclear/messy/high-stakes work to your daily-driver frontier model, familiar/checkable artifacts to a cheap workhorse, and sensory/live-data/action-heavy work to a specialist tool or harness.
Weight compute availability and harness quality (Codex, Claude Code, Cowork, agent-team features) as part of "which model," not an afterthought — a model you can't reliably access isn't the best model.
Before adopting a cheaper or open-weight model, run a replay pack of known cases and compute total cost per accepted result (tokens + retries + human repair), not price-per-token.
Re-audit your prompt scaffolding and hardcoded workarounds every time a step-change model ships — yesterday's compensating complexity becomes today's constraint on a smarter model.
2025-12-15 — Here's How I Pick the Right AI for Jobs that Matter — Models now differ in fine-grained "capability surface" detail invisible to benchmarks; find it by testing on work you have real standards for.
2025-12-12 — NEW: ChatGPT 5.2 Complete Teardown — GPT-5.2 sustains 20-40 minute delegated work packets; "delegation craft" (checkable outputs, phased work, truth friction) replaces prompt engineering as the key skill.
2025-11-25 — I Tested Opus 4.5 Early — A Christmas-tree inventory test reveals three model "stances" toward messy data: Claude reconstructs, Gemini interprets, ChatGPT abstracts — each with a matching failure mode.
2025-11-19 — I Tested Gemini 3 And It's The Real Deal — Gemini 3's vision and huge context make "see or do" work (screens, video, messy blobs) newly automatable; writing/persuading work stays with GPT/Claude.
2025-11-19 — Gemini 3 is #1, day 1 impressions — First-day reaction to Google reclaiming the top model spot, and why the launch matters beyond the benchmark.
2025-11-14 — No One is Telling the REAL ChatGPT-5.1 Story — GPT-5.1's real upgrade is instruction-following, not personality; prompts become mini-specs and contradictory instructions now cause visible drift.
2025-08-14 — GPT-5 Pro: The First AI That's Smarter and Worse at Once — Parallel-reasoning architecture makes GPT-5 Pro brilliant at multi-perspective analysis and bad at personality/coding/conversation — the death of universal intelligence.
2025-07-14 — Grok 4 is "#1" But Real-World Users Ranked it #66 — A blind five-task mini-benchmark shows Grok 4's leaderboard dominance is overfitting; real work exposes formatting brittleness and ideological drift.
2025-06-12 — Finally, a way to choose the right model — An open-source, customizable dashboard that turns seven major model releases into one tunable recommendation.
2025-06-11 — o3 Pro is Out and It's Easily The Best Model in the World — o3 Pro passes a "resonance test" for strategic thinking that no prior model cleared, at a fraction of o1 Pro's cost — but demands heavy, careful prompting.
2025-04-17 — The Complete Guide to ChatGPT o3 — o3 wins a three-task creative/reasoning gauntlet against Gemini 2.5 Pro through better instruction fidelity and agentic tool judgment.
2025-02-01 — OpenAI o3-mini and o3-mini-high — Head-to-head coding and planning tests score o3-mini and o3-mini-high against rival models.
2025-01-30 — ChatGPT o3-mini is probably out tomorrow — Using the DeepSeek moment to explain what AI benchmarks actually measure and why the acronyms matter less than people think.
2025-01-26 — Battle Bots: six AI chatbots face off — ChatGPT, Gemini, DeepSeek, and Claude compared on helping a non-engineer escape a Bolt debugging loop.
2024-12-25 — OpenAI's o3 and ARC-AGI: an explainer — A deep dive into the ARC-AGI benchmark and what o3's score does and doesn't say about abstract reasoning versus human minds.