For most of this year, if you asked me how to compare coding agents, I'd have pointed you at SWE-bench. The safer answer now is: don't.
SWE-bench Verified died in February. OpenAI audited it and found frontier models could reproduce the original human-written patches verbatim, which meant scores reflected training contamination, not capability. They pulled their own numbers from Verified and recommended SWE-bench Pro instead.
Then in July, OpenAI audited Pro and found roughly 30% of the 731 tasks are broken.
Not "hard." Not "noisy." Broken. Their automated pipeline flagged 200 tasks (27.4%). Human reviewers tagged 249 (34.1%). OpenAI retracted their Pro recommendation and called on the broader evaluation community to start over.
So the fallback for the fallback is gone. And models are still shipping press releases citing SWE-bench scores.
How We Got Here
SWE-bench Verified made sense when it launched. Real GitHub issues, real test suites, tasks that looked like actual software work. The problem is it became the target. Once models trained on code from the same GitHub ecosystem the benchmark was scraped from, contamination was inevitable.
Pro was supposed to be harder and cleaner. For a while it was. Models went from 23.3% on SWE-bench Pro to 80.3% in eight months. Claude Mythos 5 sits at the top of the leaderboard at 80.3% (as of August 12), with Fable 5 at 80% and Opus 5 at 79.2%. Those three are within 1.1 points of each other. When the top frontier models are separated by barely one percentage point, you're not measuring capability. You're measuring noise.
The speed of that progression is the tell. The tasks didn't start easy. They got easier as models absorbed more code from the same sources the benchmark was scraped from.
What the Audit Actually Found
The three failure categories say a lot about how SWE-bench Pro was constructed.
Overly strict tests. A task asks you to fix a bug. Your fix is correct. Tests fail because the test hardcodes a specific implementation approach the prompt never mentioned. You're penalized for finding a valid but different solution.
Underspecified prompts. The task description omits requirements that only show up in the hidden test suite. You can't solve what you can't see.
Low-coverage tests. You fix the bug being tested. The test passes. But the test only checks one case, and the real fix requires handling three more.
These aren't random defects. They're structural artifacts of building a benchmark from scraped GitHub issue threads and existing test suites rather than designing tasks specifically for evaluation. That mirrors how actual developers write tests: sometimes too strict, sometimes incomplete, often tied to implementation choices. Fine for production code. Fatal for a benchmark that's supposed to measure raw capability.
What This Changes for Your Team
If you're using SWE-bench scores to pick models for coding-heavy work, treat the ranking as tier guidance rather than a selection criterion. The broad ordering is real: models at the top of the leaderboard are genuinely more capable on average. The point-by-point differences aren't reliable enough to act on.
The more important shift is what you measure instead. Generic coding benchmarks tell you which tier a model belongs to. They can't tell you whether that model handles your actual codebase, your test conventions, or the specific edge cases in your stack. Supabase ran their own task-specific evals recently and found a 22-point pass rate swing just from loading context files into the agent's session, on a model already running in production. The variance on your own codebase will swamp the variance between frontier model scores on a benchmark they may have partially encountered in training.
Build a domain-specific eval suite from your own work. Pick 20 to 30 real tasks you've already solved. Run models against them. Measure pass rate, but also behavioral signal: tool calls made, context consumed, retry count. That's what actually predicts performance in your CI pipeline.
Where the Field Goes Next
OpenAI's statement called for benchmarks "built by experienced software developers specifically to test model capabilities," not repurposed from scraped issue threads. That's the right direction. But it won't fully close the loop. Any public benchmark becomes a target once models train on adjacent code at scale.
The evals that will hold up are private, task-pool-rotated, and domain-specific. Scored in tasks completed per dollar or engineering hours saved, not percentage of GitHub issues patched. SWE-bench's collapse is accelerating that shift.
Which is probably a good thing for teams who care about what a model does in their actual deployment rather than what it scores on a benchmark it's encountered before.