Benchmarking the Agents That Write Our Code

A follow-up to "Why green tests stopped meaning shipped." That post argued that LLM-powered features need evals. This one applies the same discipline to the LLM-powered developers.

We keep asking ourselves the same question: which AI coding setup is best for developers working on our repository? Which model, with which skills loaded, driven by which agent harness? Every new model release reopens the debate, and the debate always ends the same way, with opinions.

Public benchmarks can't settle it. SWE-bench and its cousins measure generic code development on public repositories; they say nothing about our codebase, our conventions, or our safety constraints. They also age badly. Their tasks are public, so solutions leak into training data over time, and this July OpenAI audited SWE-Bench Pro, found 30% of its tasks broken, and retracted its recommendation that the research community use it. We wanted a benchmark about our repo, graded by our tests.

So we built one. This post walks through the whole process: building a benchmark dataset from our own repository, then running three controlled comparisons on top of it. Model against model. Skills against no skills. Harness against harness. We aren't alone in thinking this way: Databricks recently published a similar internal benchmark built on their multi-million-line codebase, and Cursor built CursorBench from their own engineering sessions for the same reason. Their conclusions rhyme with ours.

The goal

The system under test is our clinical lab-interpretation service. It turns patient intake data and biomarker results into practitioner-style health interpretations, and it feeds a production Rails consumer. The stakes are specific: a plausible-looking but wrong output here can weaken clinical safety or silently break a response schema a downstream system depends on.

A public leaderboard can't tell us whether a model preserves those invariants. The question we needed answered was not "which model is smartest?" but:

Which coding-agent configuration is most useful on this repository, under our safety and contract constraints, per dollar?

Two principles shaped everything downstream.

First, measure the tool as deployed, not the bare model. Engineers don't paste tasks into a raw completion endpoint; they work through an agentic CLI that carries its own loop, tools, and context management. So the benchmark drives each candidate through a production coding agent and grades what it actually produces.

Second, grade deterministically wherever possible. Every code task carries hidden tests with a validated fail-before / pass-after gate, so "resolved" means the change genuinely works, not that a judge liked the look of the diff.

Step 1: Building the dataset

We built a SWE-bench-style dataset from our own git history: 36 tasks across eight families, each one a real piece of developer work replayed from the repository.

Mine the repo, don't invent tasks

We started by measuring what our engineers actually do. We classified six months of merged commits by the surface they touched and weighted the eight task families to match: safety-rule calibration and recommendation logic heaviest, schema and contract work weighted up for risk, prompt and eval quality in between, infrastructure light, plus three non-code families (code review, repo-navigation Q&A, and design planning), because a meaningful share of developer work never shows up as a diff. The final mix is 23 code-writing (patch) tasks and 13 non-code tasks.

Within the code tasks, two shapes matter for everything that follows. Coverage tasks are the routine small changes that make up most real work. Boundary tasks each carry a subtle trap: a cross-cutting invariant, a contract that unit tests alone won't catch, an interface a plausible patch can miss. Coverage tasks keep the dataset honest about what the job looks like. Boundary tasks are where models actually differ.

Split what the model sees from what the grader knows

Each task has two halves:

• model_visible, the brief: problem statement, deliverable, allowed context, and the same repository instructions (CLAUDE.md) a human would have.
• grader_only, the answer key the model never sees: the reference fix, hidden tests, contract checks, and expected findings.

A strict leakage policy keeps solution tells out of the brief, so the model solves the problem instead of pattern-matching a known answer.

Validate every gate

Every code task must pass a red-to-green gate before it enters the dataset: the hidden test fails at the task's base commit, proving the fix is genuinely absent, and passes once the reference fix is applied. We enforce this end to end through the grading pipeline. The reference fix scores 100 and resolves; an empty patch doesn't. If a task can't demonstrate that, it doesn't ship.

Score two independent signals

Grading produces two numbers per task, and we're careful not to conflate them:

The second signal exists because a failed attempt is not worthless. A patch that finds the right file and passes half the hidden tests leaves an engineer twenty minutes of work; a patch that touches nothing relevant leaves them starting over. The score measures that difference.

For code tasks, resolved means the hidden tests pass and no regression or contract check fails. Non-code tasks use a coverage gate instead: the output has to follow the required format, match enough of a hidden answer checklist, and catch every item the task marks critical.

The score's dominant component is a partial-credit ladder: a patch that applies, edits the right file, and passes half the hidden tests outscores a non-attempt. Contract checks, coverage of expected outputs, and closeness to the reference fix make up the rest. One deliberate exception couples the two signals: a clinical-safety or contract violation zeros the entire score, even if the tests pass. In this codebase, an unsafe-but-green change is the worst possible outcome, and the rubric says so.

Step 2: Comparing models

With the dataset stable, we ran the first real comparison: three Claude models, each through the Claude Code CLI, each task in an isolated git worktree, one run per (model, task).

The headline isn't the ranking. It's how close the ranking is: all three models scored within 4.5 points of each other, close enough that the order changes depending on which statistic you pick. The differences that do hold up are about character, not rank. Sonnet was the most consistent, Opus was the most bimodal and Haiku cleared the easy tasks and missed more of the hard ones.

Digging into which families discriminate exposed a structural truth about repo-native benchmarks: routine work saturates. All three models aced the recommendation-logic tasks. The separation lived where the boundary tasks live, concentrated in schema/contract evolution (Opus 2/4, Haiku 0/4) and safety-rule calibration (Sonnet 6/8 vs Haiku 3/8). A benchmark weighted to be realistic is, by construction, mostly tasks every frontier model can do. This is the same saturation problem Cursor describes with public benchmarks at frontier level, just one layer down: even a private benchmark saturates on its routine majority, and the discriminating power has to come from deliberately hard tasks.

Which led to the practical conclusion: when quality is tied, cost decides. Sonnet led resolution and mean score while running at less than half of Opus's cost. On this repository's actual work, a quality-only argument does not justify the most expensive model.

Extending across vendors

The obvious follow-up: how do open-weight models do on the same suite? We added GLM-5 (Z.ai) and Qwen3 Coder Next (Alibaba), both served from AWS Bedrock.

For this separate cross-vendor comparison, we changed the question: rather than comparing each setup exactly as deployed, we held the harness and agent context constant to isolate model performance. Because the gateway-routed models could not load ambient skills, we re-ran the Anthropic models with those skills disabled as well.

This took plumbing. Claude Code speaks the Anthropic Messages protocol and can't call GLM or Qwen on Bedrock natively, so we stood up a local LiteLLM gateway that exposes them behind an Anthropic-compatible endpoint. Fairness cut both ways: since the gateway-routed models load no ambient skills, we re-ran the Anthropic models in the same stripped-down baseline configuration.

(Sharp-eyed readers will notice the Anthropic counts differ from the first table. Different run, different configuration. With one run per task, individual counts move by a few tasks between runs. That variance is itself a finding; more below.)

Three takeaways:

• Anthropic's frontier models lead clearly, and they are the most consistent across task families.
• GLM-5 is a credible challenger. It finished six tasks behind Opus, comfortably ahead of Haiku, and matched the top models on the two largest task families (recommendation logic 6/6, safety rules 6/8). It was also noticeably slower, 10 to 17 minutes per task, but completed reliably. Databricks reached a similar verdict on the newer GLM 5.2, which landed in their top capability tier, statistically tied with Opus on quality at a lower cost per task.
• Code review is where open-weight models fall behind. Only Sonnet swept the review family (5/5); Qwen managed 0/5. If review judgment matters to your workload, weigh this heavily.

One caveat we won't bury: the harness was built and tuned for Claude, and the free-text judge is itself an Anthropic model. Some of the gap may be harness affinity and grader bias rather than raw capability. We didn't correct for either, but we did name them.

Step 3: Comparing skills

Skills are add-on instruction and tool packs you load into the coding agent, and they get pitched as a general quality dial. Before rolling them out across our tooling, we wanted evidence.

We ran a 3x4 grid: three Claude models by four skill configurations, over the 23 code-writing tasks. The configurations: baseline built-ins only, superpowers (obra's open-source skill pack), thermo-nuclear (a strict code-quality review skill), and both together. Twelve cells, everything else held constant.

What the grid says:

• Skills help more often than they hurt, but modestly. Ten of twelve setups matched or beat their baseline. The best single gain was two tasks (Opus + superpowers, 15 to 17).
• The effect concentrates in specific task families. Skills reliably moved the safety-rule tasks (Opus 4 to 6 resolved, Sonnet 5 to 7) and did nothing, or slightly hurt, on schema-contract tasks. A skill is not a general quality dial. It's a targeted one.
• Stacking both packs never won. For every model, both-together tied or trailed the better single pack. More skill mostly meant more overhead.
• The pack matters more than the pitch. Superpowers was reliable across all three models; thermo-nuclear was mixed and actually cost Haiku a task.

The meta-lesson mirrors the model comparison: don't adopt a skill on reputation. Measure it on the model and the kind of work you actually plan to use it on.

Step 4: Comparing harnesses

The final axis is the one almost nobody isolates. Coding-agent comparisons routinely confound two variables: the model and the harness driving it (the agent loop, tools, system prompt, and context management). We held the model fixed, the same Bedrock Sonnet inference profile, and varied only the harness. Claude Code brings a large system prompt, a rich purpose-built toolset, and skills machinery. Pi is deliberately minimal: four tools, a small prompt, no sub-agents or plan mode. Same 36 tasks, same worktrees, same grading.

Three findings:

1. The model dominates; the harness moves the margins. The resolved counts differ (18 vs 22), but look underneath: on 30 of 36 tasks the two harnesses reached the same resolved-or-not verdict, and on 24 of 36 the 0-100 quality score came out identical to the decimal, down to making the identical wrong guess on one task (both misread the same environment-variable name). The four-task gap all comes from the handful of tasks where they diverged. Harness choice shifted the resolved rate by about 11 points without changing which tasks were fundamentally hard.
2. The minimal harness resolved more; the rich one finished faster. Pi's wins concentrated in planning and review tasks that reward long, deliberate exploration. Claude Code's richer toolset found things in fewer, bigger steps and finished about 40% faster. Databricks saw the same shape on their workloads: pi sent about a third of the context per turn and simple harnesses often performed best.
3. Pi's leaner context made it cheaper. Repriced on a single price table, pi's 43% token advantage translated into a cost advantage of about a third. One methodological note for anyone comparing CLIs: each tool counts tokens and prices its own usage differently, so normalize both to one counting rule and one price table before trusting any cross-tool number.

Recommendations

Pulling the three comparisons together, here's where we landed for our own tooling.

1. Default to Sonnet. Its quality is on par with Opus (it even resolved one more task in our first run), it delivers that quality more consistently than any other model we tested, and it costs less than half as much. Reserve Opus for the hard-task families where it separates (schema and contract work); use Haiku for routine, low-stakes tasks where its cost advantage compounds.
2. Turn skills on selectively, not globally. Superpowers earns its keep on safety-rule work on the larger models. Don't stack packs, and don't enable a skill for work it hasn't been measured on.
3. Pick the harness by what you're optimizing. Claude Code for wall-clock throughput; a slower, more deliberate agent like pi if resolution rate per run matters more. On a single price table, pi's leaner context also made it about a third cheaper per suite.
4. Watch GLM-5. It isn't a frontier replacement, especially for code review, but at 56% resolved it already beats our smallest first-party option. A workable alternative is worth having on the bench even if we never switch.
5. Distrust any cross-tool token or cost number that doesn't state its counting contract. Including ours.

And one recommendation about the method itself: the benchmark is worth more than any single result it produced. Every one of these questions (a new model drop, a new skill pack, a new agent CLI) used to be a hallway debate. Now it's a command and an afternoon of compute. That's the same shift we described for prompt changes in the evals post: from "does this look better to you?" to "v2 beats v1 by 8%."

Next steps

We're explicit about what this benchmark is: a proof of concept, one run per cell. Here's the list before we'd call any ranking definitive.

• Multi-sample everything. Single-run noise still exceeds the model gaps. The same task can flip resolved/unresolved between rounds at temperature 0, and our own Anthropic counts moved by a few tasks between runs. Rank on medians with variance bands.
• Grow the boundary-task minority. Most of the ranking signal lives in a handful of hard tasks; the routine-coverage majority inflates every model's mean. We'll keep mining incidents and near misses for tasks that separate.
• Instrument cost per resolved task. With quality this close, dollars per useful result is the decision metric, and it deserves first-class measurement rather than back-of-envelope pricing.
• De-bias the judging. Validate the free-text judge against human labels and test for same-vendor grader bias before any cross-vendor claim carries weight.
• Re-mine periodically. A task distribution measured from six months of commits ages. Like the eval sets in our last post, this dataset needs maintenance or it quietly stops representing the work.

The through-line from the last post holds: green tests stopped meaning shipped, and leaderboard rank never meant "right for your repo." The only benchmark that answers your question is one built from your own work, and it's smaller to build than you think. Databricks put the same point well: any team with a backlog of merged PRs is already sitting on a benchmark no model has trained on, graded by the tests the team wrote. Ours started as a manifest, a grading rubric, and a runner that diffs a worktree.