Partial RuneBench results: DeepSeek dominates cost-effectiveness as testers run out of free quota

On November 12, developers mariofilhoml and maxbittker gradually posted partial real-world results from the AI coding benchmark RuneBench on X. With OpenRouter's free quota exhausted, this round only covered some models such as Inkling 2 Small; a fuller comparison will have to wait until the quota resets.

Confirmed

Inkling 2 Small scored 5.314 (log-average, i.e. geometric mean) across the 9 skills tested so far, far ahead of the previous generation.
Hy4 High (Pi internal) scored 6.23 at roughly $0.59 per run, placing it on the cost-performance Pareto frontier plotted by API cost versus score.
The cost-score scatter plot shows the cost-performance quadrant dominated by DeepSeek models.
mariofilhoml's tests ran on Pi (xhigh config). His observations: OpenCode fell short of Pi in his tests; OpenCode 2.x is clearly better than 1.19; Hy4 underperformed Pi. OpenCode currently has no paid tier on OpenRouter.

Not Yet Confirmed

Full RuneBench scores and final rankings for all models are not out yet, pending retests once the OpenRouter quota resets.

Why It Matters

RuneBench plots API cost and scores on the same chart, making the line between "expensive high scorers" and "cheap good-enough models" immediately visible. This partial round shows the low-cost quadrant being taken over by DeepSeek — directly useful reference for developers choosing models by cost-performance.

2026-11-12 ~ 2026-11-12 · 5 related posts

Primary sources

Partial RuneBench results: Inkling 2 Small scores 5.314, far above original — mariofilhoml · 2026-11-12 02:34 (UTC+08:00)
RuneBench Cost-Efficiency Data: Hy4 High Hits 6.23 at $0.59 per Run, a Pareto Frontier Point — mariofilhoml · 2026-11-12 05:44 (UTC+08:00)

[source] Partial RuneBench results: Inkling 2 Small scores 5.314, far above original — mariofilhoml · 2026-11-12
RuneBench cost-vs-score chart shows DeepSeek dominating the budget quadrant — maxbittker · 2026-11-12
Dev benchmarks: OpenCode trails Pi as DeepSeek dominates agentic coding quadrant — mariofilhoml · 2026-11-12
OpenCode 2.x notably better than 1.19, Hy4 trails Pi in tester's runs — mariofilhoml · 2026-11-12
[source] RuneBench Cost-Efficiency Data: Hy4 High Hits 6.23 at $0.59 per Run, a Pareto Frontier Point — mariofilhoml · 2026-11-12