Dev evals 20+ third-party models vs Gemini for spec quality: all fail, "wasteland of mediocrity"
julianharris · x · 2026-09-30
Julian Harris ran a large-scale eval hunting for a materially cheaper alternative to Gemini 3.6 Flash on spec quality, testing essentially everything DeepInfra offers. All models failed. He bet Claude on the outcome beforehand and won — most third-party models were simply too old to compete with today's SOTA. He expected free Chinese open models to give third-party infra an edge, but they didn't; despite a 10x price hike over the past year (doubling again in January), Gemini "absolutely cleans up" for his use case. He eval'd 20+ models in one day and calls the third-party landscape a "wasteland of mediocrity."
More from Models
- Stealth Model 'silicon-2' Surfaces in Devin, Suspected to Be Anthropic's Next Model — kimmonismus · 2026-11-16
- Google hides Gemini model label in AI Mode, rolls Gemini 4 Carbon to Pro subscribers — gaganghotra_ · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- Nathan Lambert calls out Ethan Mollick for 'selectively ignoring' Kimi in model comparisons — natolambert · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- Unsloth Q8 crushes default Q3 in a self-hosted test over 331 financial PDFs — No_Cartographer3953 · 2026-11-16