The Benchmark

Real feature-implementation tasks pulled from public GitHub repos, replayed by an agent and graded by each repo's own test suite. Same model, same scaffold, same tools in both arms — the only difference is SourceIndex.

64→87% Pass rate, hard-repo suite

−36% Agent time in session

246 Runs across both studies

11 Public repos

Study 1 · Capability

On hard repos, more tasks actually land

Nine tasks from nine different repos (aws-cli, faker, dspy, jinja, requests, poetry, pre-commit, hermes-agent, agent-reach), Claude Opus, 5 runs per configuration — 45 runs per arm. Pass = the repo's tests for that feature all pass.

Claude Code Claude Code + SourceIndex

requests

0/5

5/5

hermes-agent

0/5

4/5

pre-commit

4/5

5/5

aws-cli

5/5

dspy

5/5

faker

5/5

jinja

5/5

agent-reach

5/5

poetry

0/5

Totals: Claude Code 29/45 (64%) · with SourceIndex 39/45 (87%). Wall-clock time was at parity (≈5 min per task in both arms, roadmap search included). We show the misses too: the poetry task fails in both arms — better context doesn't rescue a task the model can't implement.

Study 2 · Efficiency

On tasks both can do: fewer turns, less time, slightly cheaper

Seven tasks from two repos (beets, RAG-Anything), run with both Claude Opus and Claude Sonnet, 4 runs per cell — 112 runs. Both arms pass nearly everything here (98% vs 100%), so the differences show up in agent time, turns, and cost.

Model	Metric	Claude Code	+ SourceIndex	Δ
opus	Pass rate	96%	100%	+4pts
opus	Cost / task	$2.54	$2.46	−3%
opus	Turns	18	17	−4%
opus	Agent time	154s	110s	−28%
sonnet	Pass rate	100%	100%	—
sonnet	Cost / task	$0.46	$0.40	−13%
sonnet	Turns	13	12	−7%
sonnet	Agent time	187s	108s	−43%

Per-task detail (avg over 4 runs)

Cost and in-session agent time per task. Not every task wins — broad cross-file tasks like rag-anything-3bee08d8 cost slightly more with an index. We publish those rows anyway.

Task · Claude Opus	Cost	+ SI	Δ Cost	Time	+ SI	Δ Time
beets-0d8d3bfa	$2.83	$2.06	−27%	144s	107s	−26%
beets-3bcc539b	$1.40	$1.24	−11%	99s	50s	−49%
beets-da7714ae	$1.87	$1.70	−9%	155s	86s	−44%
beets-f203bc52	$2.74	$3.16	+16%	227s	154s	−32%
beets-f9f4af87	$1.14	$1.12	−2%	77s	56s	−28%
rag-anything-3bee08d8	$6.82	$7.06	+3%	328s	282s	−14%
rag-anything-3e2f6429	$0.96	$0.85	−11%	48s	37s	−23%

Task · Claude Sonnet	Cost	+ SI	Δ Cost	Time	+ SI	Δ Time
beets-0d8d3bfa	$0.62	$0.34	−45%	277s	99s	−64%
beets-3bcc539b	$0.32	$0.27	−16%	141s	71s	−50%
beets-da7714ae	$0.43	$0.30	−30%	181s	92s	−49%
beets-f203bc52	$0.36	$0.36	−1%	173s	109s	−37%
beets-f9f4af87	$0.23	$0.20	−13%	107s	46s	−58%
rag-anything-3bee08d8	$1.03	$1.08	+5%	314s	294s	−6%
rag-anything-3e2f6429	$0.26	$0.29	+11%	120s	44s	−63%

Methodology & honest caveats

Tasks are real commits. Each task is a feature actually merged into the repo. We roll the repo back to just before that commit, hand the agent the feature description, and grade with the tests the real commit had to pass.
Both arms are identical — same model, same agent scaffold, same tools — except that one has SourceIndex set up in the repo.
Multiple runs per cell (N=4–5). Agent runs are high-variance; single-run comparisons are noise.
SourceIndex's own search cost is included in its cost figures. The one-time index build (typically under $1 per repo; $2.30 for the largest we tested) is excluded — it amortizes across every future task.
Agent time vs wall clock: "agent time" is the time your coding agent spends in-session. In Study 2's setup the roadmap was prepared before the session (~75s), putting total wall clock between −1% (Sonnet) and +21% (Opus); in the current release the search runs inside the session, and Study 1 measured wall-clock parity.
Cost telemetry for Study 1 was partial (a logging-proxy outage), so cost figures come from Study 2.
It doesn't always win. Tasks needing broad cross-file context can cost a few percent more, and no amount of context rescues a task the model can't implement (see poetry, 0/5 in both arms).

Get Early Access