The AI Rankings Just Got Reshuffled

Context

This episode is mainly about Artificial Analysis rolling out version 5 of its Intelligence Index, the benchmark bundle many people use as a shorthand for which AI model is best right now. NLW argues that the update matters because it changes the rankings by changing what gets measured: more private datasets, more agentic work, and different category weights. For PMs choosing models, the real lesson is not which model is first this week. It is how benchmark design, cost per task, and workload fit should shape product decisions.

The Big Idea

Benchmarks matter, but a benchmark ranking is never neutral. Change the tests, the weighting, or the operating settings, and the leaderboard can reshuffle even when the underlying models barely changed.

NLW's point is that PMs should treat a composite score as a starting point, not a verdict. The right model choice depends on the work you need done, the price to get that work done, and whether the benchmark setup matches how your team will actually use the model.

Key Insights

Rankings changed because tests changed

• Artificial Analysis overhauled its index, and the new design materially moved models up and down the leaderboard.
• GPT-6 Astra jumped from sixth to second because the new index gives more weight to computer use and scientific tasks. GPT-5.1 Sol moved from ninth to third. Kimmy K3.1 Pro fell from fourth to ninth.
• The lesson for PMs is simple: when a ranking changes, first ask whether model capability changed or whether the evaluation changed.

Private datasets curb bench-maxing

• Version 5 is the first Artificial Analysis index where a majority of the component benchmarks use private datasets.
• The reason is to make benchmark maxing harder. If model builders cannot see the exact test set, it is harder to train directly to the benchmark and inflate the score.
• New additions reflect that goal. Frontier Code 2.0 uses Cognition's private real-world coding tasks and scores code based on whether a repository maintainer would merge it. Terminal Bench Science evaluates agentic scientific research work rather than textbook recall.
• Kyle Chan adds a useful nuance: the private share rose from 45% to 60%, which is meaningful, but it does not prove every open-model drop was pure gaming.

Agentic work now sets bar

• The redesign shows how fast the definition of a good model has changed.
• Earlier indices spent more attention on long-context reasoning because models struggled to work across large codebases without context rot.
• Artificial Analysis removed AA LCR, its long-context reasoning test, because newer memory and compaction improvements have made that less differentiating.
• Coding benchmarks in version 5 now measure agentic performance, not just isolated code generation. Science benchmarks also moved toward real research workflows instead of static knowledge checks.
• For PMs, this means benchmark reading should track how users actually invoke models. A model optimized for one-shot Q and A may look good on older tests and weak in real product workflows.

Composite scores hide priorities

• Artificial Analysis now weights four categories equally: knowledge work, coding, science, and general capability.
• Equal weighting makes the index easy to read, but it also bakes in a worldview about what matters.
• NLW explicitly questions whether 25% for science matches the average individual user's needs, even if OpenAI and Anthropic are pushing scientific discovery as a frontier use case.
• A PM choosing a model for support automation or internal search should not inherit those weights blindly. A composite index is only useful if its category mix resembles the product workload.

Open-weight gaps are task-specific

• The headline result is that the gap between proprietary US models and open-weight models widened from 3 points to 12 after the reweighting.
• Kimmy K3.1 Pro remains the top open-weight model at 46, but that is well behind the new leaders in the high 50s.
• The important nuance is that weakness is not uniform across tasks:
• On Terminal Bench Science, Kimmy scored 34% versus 63% for GPT-6 Astra.
• On Terminal Bench 4.0, Kimmy scored 67% and actually led all models on that coding-agent benchmark.
• Kimmy also beat Claude on the private Automation Bench AA evaluation.
• The right conclusion is not that open models are bad. The better conclusion is that open and closed models now separate more clearly on some advanced research and reasoning workflows than on every agentic task.

Cost changes leaderboard value

• Frontier models are not just separated by capability. They are separated by cost per completed task.
• Artificial Analysis measured GPT-6.1 Terra at $0.96 per task and Fable 5.1 at $13.04 per task, a very wide spread inside the top 10.
• Gemini 4 Carbon reached first place on the new index at 59, with a measured cost of $7.76 per task. NLW notes that this is 23% cheaper than Opus 5.5 and 20% more expensive than Astra.
• A PM shipping AI features should care about capability only after asking what the extra capability costs. A leaderboard win can be economically irrelevant if the product margin cannot absorb it.

Mental Models & Frameworks

Benchmark maxing check

• Use this when a model looks unusually strong on published rankings.
• Ask three questions:
• Is the benchmark based on public or private data?
• Does the benchmark test memorized answers or end-to-end task execution?
• Do real user reports match the score, or does the score feel disconnected from lived experience?
• This check helps separate genuine capability from training-to-the-test behavior.

Weighting reveals worldview

• Every composite index is a hidden prioritization framework.
• Artificial Analysis version 5 assigns 25% each to knowledge work, coding, science, and general capability. That makes sense if you want one broad frontier index.
• It becomes misleading when your product is not broad. A PM evaluating models for customer support, code generation, or research assistants should build a custom weighting that mirrors the actual workload.

Saturation forces benchmark resets

• Benchmarks stop being useful when many models cluster near the ceiling.
• The usual response is to add harder questions, swap in new tests, or move to a new benchmark version.
• That means time-series comparisons are tricky. A model can appear to drop mainly because the measuring stick got harder.
• Use this model whenever a vendor markets a rank change as pure progress. First check whether the underlying test changed.

Snapshot versus progress view

• Artificial Analysis added a Progress Index that places models released since November 2022 on one capability scale.
• A snapshot leaderboard tells you who is ahead today. A progress view tells you which lab is improving fastest and whether a competitor is closing the gap.
• These are different decisions. Procurement and feature launches care about the snapshot. Strategy, platform risk, and partnership bets care about the direction of travel.

Decision Principles

Principle: Choose by workload

• When: a team is tempted to standardize on whichever model tops a general leaderboard.
• Why: a model can trail overall and still lead on the specific work your product needs, as Kimmy K3.1 Pro did on Terminal Bench 4.0 while lagging badly on science tasks.

Principle: Re-read benchmark methodology

• When: a vendor or analyst says the rankings have changed dramatically.
• Why: a benchmark update can change dataset privacy, task types, or category weights. Those design choices can move rankings without any equally dramatic change in product fit.

Principle: Price completed tasks

• When: two models are close enough in quality that either could work in production.
• Why: Artificial Analysis showed a huge spread in cost per task across frontier models. Product economics depend on cost for a finished task, not prestige from using the top-ranked model.

Trade-offs & Nuance

Private tests versus transparency

• Private datasets reduce benchmark gaming, which is good.
• Private datasets also make the benchmark harder to audit independently, which is a real trade-off.
• A PM should treat private-benchmark gains as more trustworthy against overfitting, but less transparent in terms of external reproducibility.

Science weight versus relevance

• Giving science a quarter of the score may be sensible for judging the frontier.
• Giving science a quarter of the score may be misleading for a product where users mostly need drafting, support, search, or software tasks.
• The trade-off is between a future-facing leaderboard and a use-case-specific leaderboard.

Max settings versus reality

• NLW flags an important practical issue: many leaderboards are run at max settings, but most teams do not deploy models that way in everyday use.
• Higher effort settings may improve scores, but they also change latency and cost.
• A PM comparing models for production should care about performance at the exact effort tier users will experience, not only the best possible lab setup.

Leaderboard rank versus fit

• A single rank helps people talk about the market quickly.
• A single rank also hides category strengths, failure modes, and economics.
• This is why the same benchmark update can be both useful and insufficient. It improves the map, but it is still only a map.

Common Mistakes

Mistake: Buying headline scores

• Teams often treat the number one model as the automatic default.
• That misses the fact that version 5 changed the underlying mix of tasks. The best model is only best for the benchmark's definition of important work.
• The better approach is to use the leaderboard to shortlist candidates, then test them on your own workflow.

Mistake: Ignoring effort settings

• A benchmark score achieved at maximum effort can create false confidence about production behavior.
• If your product needs low latency or tight cost control, a model that looks weaker on a max-effort leaderboard may be better at the settings you can actually afford.
• This mistake becomes more expensive as AI features scale.

Mistake: Overgeneralizing one category

• Kimmy K3.1 Pro illustrates the risk. It was weak on advanced science tasks and strong on coding-agent tasks.
• Drawing a broad conclusion from one category would miss the real picture.
• The better move is to inspect the few benchmarks that match your product's critical path.

Practical Application

Build a workload scorecard

• List the 3 to 5 AI tasks your product actually depends on, such as support drafting, code generation, document extraction, or browser automation.
• Assign each task a weight based on business importance.
• Use public rankings only as input. Then run the same prompts or agent loops on each candidate model and score the results against your own weighted mix.
• The resulting scorecard turns a generic market leaderboard into a product-specific decision tool.

Re-test top models quarterly

• Set a quarterly review for any model that matters to product performance or cost.
• Include benchmark methodology changes in the review, not just new model releases.
• Gemini 4 Carbon reaching first place right after an index overhaul shows how quickly both models and measuring systems can move.

Measure price per outcome

• Track cost at the unit your business cares about: resolved support ticket, accepted pull request, completed research brief, or automated browser task.
• Artificial Analysis uses cost per task, which is closer to product reality than cost per token.
• Tracking cost per finished task makes it easier to justify paying more for a model that finishes the work, and easier to reject a prestigious model that does not.

Run effort-tier evaluations

• Test the same workflow at low, medium, and high effort settings when the model provider exposes them.
• Record three things for each tier: quality, latency, and cost.
• NLW's critique is that a max-effort leaderboard can be directionally useful while still hiding the setting that will actually win in production.

Separate frontier from sufficient

• Not every AI feature needs a frontier model.
• Use the top-end leaderboard to identify where the frontier is moving. Use a second pass to find the cheapest model that is good enough for your actual product bar.
• The 10x-plus spread between top-model costs makes this separation especially important.

Questions to Consider

• Which AI tasks in your product would be misrepresented by a benchmark that gives 25% of the score to scientific reasoning?
• Where is your team treating a number one leaderboard position as a buying signal instead of testing the model on the exact workflow users will run?
• If your production AI feature cannot afford max-effort settings, what would your model ranking look like at the latency and cost limits you actually ship?
• Which open-weight or smaller model might be good enough for one narrow task, even if it trails badly on a broad frontier benchmark?

Bottom Line

The benchmark update is useful precisely because it shows how unstable model rankings can be once the tests get closer to real work. For PMs, the winning habit is not memorizing who is first this week. It is understanding what the benchmark measures, what your product needs, and whether the extra capability is worth the extra cost.

Resources Mentioned

Case Studies Mentioned

Gemini 4 Carbon jump

Google released Gemini 4 Carbon only two weeks after Gemini 4 Argonne and it immediately took first place on the new Artificial Analysis index with a score of 59. NLW treats this as a reminder that frontier rankings can move for two reasons at once: the model improved, and the benchmark changed. Carbon also illustrates why PMs should wait for real usage data. Benchmark gains looked large, but Google had not broadly released the model yet.

Astra forced an update

Artificial Analysis pushed an earlier version 4.2 update because GPT-6 Astra's strong computer-use performance was not being reflected well enough in version 4.1. That is a useful benchmark governance example: if the index lags behind how people actually use models, the ranking becomes less helpful even if the math is internally consistent.

Tools & Products