Every Model Is State of the Art
Every lab ships "state of the art." They can't all be right. The leaderboard measures the leaderboard, and the labs are teaching to the test.
Read the launch posts. Every new model is "state of the art." Google's is the best. OpenAI's is the best. Anthropic's is the best. The open-weights model out of a lab you can't pronounce is also, somehow, the best. They cannot all be the best. What they can all do is find one chart where they win and lead the announcement with it.
The trick is picking your own ruler. There are enough benchmarks now that any competent lab can cherry-pick the three where its model looks like a prodigy and quietly leave out the twelve where it looks average. MMLU, GPQA, SWE-bench, MATH, HumanEval, the acronyms breed faster than anyone can audit them. You get a bar chart with your model's bar painted the brand color and standing one pixel taller than the rest. That's not a measurement. It's a billboard with error bars.
Then there's the part the launch post skips. The tests are public. The training data is the internet. The benchmark questions, and their answers, live on the internet. So the model that "aces" the exam has, in a huge number of cases, already read the exam. This has a name. Data contamination. And it isn't a freak accident that slips through once in a while. It's the default condition of a system trained on everything humans have ever written down, including the answer key.
A benchmark stops measuring anything the moment a company's valuation depends on the number.
You can watch it happen. When Apple's researchers took grade-school math problems the models supposedly crushed and rebuilt them with the names and numbers swapped, keeping the logic identical, scores dropped. Same difficulty, different surface, worse performance. A model that actually understood the math wouldn't care that you renamed a variable. A model that pattern-matched its way through a memorized test set cares a lot. That gap between the polished benchmark and the reworded version of it is the entire story, and the industry would very much prefer you not look at it.
Every teacher already knows how this ends. When a school's funding rides on a standardized score, teachers stop teaching the subject and start teaching the test. The kids get better at the test and no better at anything the test was supposed to stand in for. Labs do the exact same thing, just with more GPUs and better PR. There's a law for it. Goodhart's law: when a measure becomes a target, it stops being a good measure. The whole AI leaderboard economy is a live demonstration, running in public, funded to the tune of billions.
Watch the treadmill run and it gets funnier. A benchmark comes out. Within a year or two the models saturate it, everyone's scoring in the high nineties, and the number stops separating anyone from anyone. So a new, harder benchmark appears, conveniently, right about the time the next generation needs a fresh chart to win. GPT-4 buried the old tests, so we got harder ones. Those are getting buried too. The goalposts don't move because the science demands it. They move because a saturated benchmark can't sell a launch, and there's always a launch coming.
Here's why the con works on you and me. The number is legible and the reality isn't. "94.2 percent on SWE-bench" fits in a tweet and lands in a headline. "It's genuinely good at some coding tasks and randomly, confidently terrible at others in ways you'll only discover in production three weeks from now" does not fit in a tweet, and it definitely doesn't move a funding round. So the industry sells the number, and we buy the number, and then we deploy the thing and slowly realize the number was answering a question none of us actually asked.
The benchmark that can't be gamed is the boring one. Does it work on your job. Not the leaderboard's job. Your codebase, your customers, your specific pile of weird edge cases that no eval set has ever seen. That score is different for every single person reading this, which is precisely why no lab will ever put it on a slide. You can't raise a round on "results may vary." You can raise a round on a red bar that's taller than the blue one.
None of this means the models are fake or useless. Some of them are genuinely, uncannily good. That's the part that makes the theater work. Wrap a real capability in a rigged scoreboard and the scoreboard borrows the credibility of the capability. You end up trusting the chart because the tool impressed you once, and the chart was never the reason the tool was good.
So the next time a model launches and the graphic shows it clearing the bar, ask the only three questions that matter. Winning at what. Measured by whom. On a test written and scored by the company selling you the answer. Then close the launch post, run the thing against your own work, and trust that number instead. The leaderboard was never built for you. It was built for the raise.