100× efficiency on the SWE-Atlas QA benchmark

A 108-question codebase exam: Opus 4.7 + Claude Code scores 40.3% at about $3.11 a task. DeepSeek V4 Pro with a SourceIndex roadmap scores 41.7% at 5 cents, in two-thirds the time — cheaper × faster ≈ 100×.

Same score, two very different bills

SWE-Atlas asks an agent deep questions about real codebases. Mini-SWE-Agent is a simple and naive harness — about 100 lines of Python whose only tool is the shell.

Configuration	Score	Cost / task	Minutes / task
Opus 4.7 · Claude Code published score	40.3%	~$3.11	11.2
DeepSeek V4 Pro · Mini-SWE-Agent + SourceIndex	41.7%	$0.05	7.2

~60× cheaper and 1.5× faster at the same score — about 100× less cost×time. The 40.3% is Opus 4.7's score on Scale's public leaderboard; the cost and time next to it are from our own Opus 4.8 · Claude Code run. On our 108-question set that same harness scored higher — 48.6%, the figure in the five-configuration table below.

Same model, add the roadmap

The same roadmap works for both DeepSeek V4 Pro and Sonnet 4.6.

Mini-SWE-Agent Mini-SWE-Agent + SourceIndex

DeepSeek V4 Pro

23.3%

41.7% · +18 pts

Sonnet 4.6

36.4%

51.5% · +15 pts

Sonnet 4.6 + SourceIndex, at 51.5%, is the best of the five configurations we ran — above the Claude Code Opus baseline.

Same exam, all five configurations

Configuration	Score	Cost / task	Minutes / task
Opus 4.8 · Claude Code	48.6%	$3.106	11.2
Sonnet 4.6 · Mini-SWE-Agent	36.4%	$2.557	13.8
Sonnet 4.6 · Mini-SWE-Agent + SourceIndex	51.5%	$2.261	13.4
DeepSeek V4 Pro · Mini-SWE-Agent	23.3%	$0.045	5.2
DeepSeek V4 Pro · Mini-SWE-Agent + SourceIndex	41.7%	$0.050	7.2

On frontier models the roadmap buys you a smaller bill. On cheap models it buys you accuracy instead — DeepSeek V4 Pro goes 23.3% → 41.7% at essentially the same cost per task ($0.045 → $0.050). Different trade, same mechanism: the roadmap removes exploration the model would otherwise pay for in turns, and gives a weaker model information it wasn't going to find on its own.

Up to 5.2× cheaper per task in real execution

Real feature commits from public repos, replayed by an agent and graded by the repo's own tests. The only difference between the two runs is SourceIndex.

Claude Code · best measured

3.9×

cheaper per task

Cost: $4.61→$1.19
Turns: 44→13.5

OpenCode · best measured

5.2×

cheaper per task

Cost: $2.386→$0.461
Turns: 74→18

Every one of the 16 task × model pairs — all frontier models — came out cheaper. Here they all are.

Claude Code Claude Code + SourceIndex

faker

$1.23

$0.59 · 2.1×

beets

$2.03

$1.05 · 1.9×

aisuite

$0.57

$0.36 · 1.6×

starlette

$0.79

$0.69 · 1.1×

Claude Code, Opus 4.7.

OpenCode OpenCode + SourceIndex

faker

$1.16

$0.77 · 1.5×

beets

$2.39

$0.46 · 5.2×

agent-reach

$0.92

$0.38 · 2.4×

starlette

$0.66

$0.38 · 1.7×

OpenCode, Opus 4.6.

Claude Code

Claude Opus 4.7, four repos, three runs each; every run in both setups passed. Turn counts and wall clock per task:

Task · Opus 4.7	Turns	Fewer	Wall clock
faker-740812bd	21.3→7.3	2.9×	127s→108s
beets-0d8d3bfa	33.0→22.0	1.5×	243s→190s
aisuite-7eecd6d4	10.3→8.3	1.2×	93s→77s
starlette-49d4de92	19.3→13.5	1.4×	135s→125s

Across models and effort

One task (beets), four configurations. The headline cell is the last row.

Model · effort	Cost	+ SI	Cheaper	Turns	Wall
Opus 4.7 · xhigh	$2.03	$0.91	2.2×	33→19	243s→201s
Opus 4.8 · xhigh	$3.35	$1.32	2.5×	52→23	461s→379s
Fable 5 · xhigh	$4.13	$2.29	1.8×	28→15	504s→382s
Opus 4.8 · max	$4.61	$1.19	3.9×	44→13.5	755s→450s

Every SourceIndex run used fewer turns than every baseline run of the same task, and turn ranges never overlapped between arms in any configuration. starlette is the weakest case at 1.1× — we show it.

OpenCode

Same tasks, same roadmap, run through OpenCode 1.18.2 on AWS Bedrock. SourceIndex is a context layer, not a plugin for one editor.

Task	Model	Cost	+ SI	Cheaper	Turns
beets-0d8d3bfa	Opus 4.6	$2.386	$0.461	5.2×	74→18
agent-reach-66883060	Opus 4.6	$0.923	$0.379	2.4×	24→16
starlette-49d4de92	Opus 4.6	$0.659	$0.381	1.7×	19→17
faker-740812bd	Opus 4.6	$1.158	$0.769	1.5×	30→17
faker-740812bd	Sonnet 4.6	$0.887	$0.268	3.3×	30→18
starlette-49d4de92	Sonnet 4.6	$0.365	$0.221	1.7×	21→17
agent-reach-66883060	Sonnet 4.6	$0.374	$0.229	1.6×	20→16
beets-0d8d3bfa	Sonnet 4.6	$0.868	$0.584	1.5×	38→30

All eight pairs are wins. SourceIndex turn counts stay flat at 16–18 whatever the task, while baseline turns swing from 19 to 74 with exploration luck — the dollar win tracks how far the unguided agent wanders. One run per cell, so read these as directional.