Supabase open-sourced their evals framework last week (supabase/evals, Apache-2.0), and I think it's the most useful thing published about AI coding agent evaluation in months. Not because of which model topped the leaderboard. Because of how they designed the measurement itself.
What They're Testing and Why It's Hard
Supabase built their evals around a three-axis grid: products (database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api), topics (RLS, security, migrations, SQL, SDK, observability, self-hosting, declarative-schema), and stages (build, deploy, investigate, resolve).
That last axis is where it gets real. "Build" is the easy part. Any capable coding agent can scaffold a schema. "Deploy" and "Investigate" are where agents start to diverge. "Resolve" is where you find out if an agent can fix a broken RLS policy without silently breaking three others it didn't know existed.
They're running Claude Code, OpenAI Codex, and OpenCode through these scenarios against a real Supabase backend, not exam questions with pre-known answers. That distinction is the whole point.
The Numbers That Matter
Opus 5 and Kimi K3 both hit 100% pass rates unaided in the Build stage. That's expected. They're the biggest frontier models available right now.
Here's what I found more interesting. Sonnet 5 scored 78% unaided, then jumped to 100% after Supabase loaded their context files into the agent's session. They call these "skills" in the framework. GPT-5.6 Sol went from 89% to 100% with skills loaded. GPT-5.4 mini went from 78% to 89%.
Different model sizes, different baselines, but every single one improved with context injection. The floor lifted across the board. That's a meaningful signal. Even the smallest tested model can clear most build-stage tasks if you give it the right context. You're not necessarily locked into running the biggest model for every Supabase operation in your agent loop.
The Context Injection Effect Is the Real Finding
Loading relevant documentation before the agent runs isn't a new idea. But seeing a 22-point jump for Sonnet 5 on a domain-specific eval is not a rounding error. That's the difference between "this agent sometimes works on our stack" and "this agent passes every build scenario we throw at it."
The thing most teams are missing is measurement. You probably inject some context into your agents already: docs, API references, a README. But do you know how much it actually helps? Supabase's framework makes that testable. Run the eval with skills, then without. Get the delta.
One finding I haven't seen discussed much: Codex read around 8 pages of documentation per evaluation scenario, while Claude Code averaged roughly 2. Neither is obviously better. More doc pages might mean better coverage, or it might mean the agent is fishing for context it should already have internalized. That kind of behavioral difference would never surface on SWE-bench or HumanEval. It only shows up when you test on your actual domain with your actual tooling.
Two Tracks, Two Purposes
Supabase splits their scenarios into two buckets. Published benchmark scenarios are fixed. They move scores on the public leaderboard and stay stable so comparisons stay valid. Regression scenarios refresh daily, cover known failure modes, and don't feed the public numbers. Those are purely for internal signal.
That split matters more than it looks. A public benchmark needs stability or the comparison becomes meaningless. An internal regression needs freshness or it stops catching real failures. Most teams I've talked to conflate these, and end up with a stale benchmark they trust too much and a regression suite they check too rarely.
How to Borrow This Design
If you're shipping something that relies on AI agents doing domain-specific work, this is the eval structure worth copying:
- Pick the axes that describe your problem. For Supabase it's products x topics x stages. For your team it might be services x intent types x error conditions.
- Separate your stable benchmark from your regression suite. Don't conflate them.
- Test agents with and without your context files. Measure the actual lift.
- Add at least one behavioral metric beyond pass/fail. Doc pages read per scenario, tool calls made, retry count. These reveal things that pass rates hide.
I haven't built something this structured for my own agent work yet. But looking at that 78%-to-100% gain on a model I'm already running in production, I'm going to. The repo is at github.com/supabase/evals and the public leaderboard lives at supabase.com/evals. Worth studying even if you're not a Supabase shop.