Gemini 4 Neon's SWE-sweep Score Challenged as Harness-Dependent
Google DeepMind released Gemini 4 Neon, claiming up to 39% on SWE-sweep. Researchers questioned the result, noting the same weights scored only 12.8% with a different harness, fueling concerns that benchmark scores depend heavily on the evaluation harness rather than model capability.
2026-11-19 ~ 2026-11-19 · 2 related posts
- Episode 1: Unreleased Gemini 4 Neon Appears in Google Cloud Billing, Priced $3/$15(2026-11-18, 7 posts)
- Episode 2: Google Launches Gemini 4 Neon: Fast-Reasoning Model Tops AA Coding Agent Index(2026-11-18, 95 posts)
- Episode 3: Gemini 4 Neon rolls out quietly with three-tier model lineup(2026-11-19, 4 posts)
- Episode 4: Gemini 4 Neon launches to top coding agent rankings at half the price(2026-11-19, 22 posts)
- Episode 5: Gemini 4 Neon's SWE-sweep Score Challenged as Harness-Dependent(2026-11-19, 2 posts)
- Episode 6: Gemini 4 Neon Tops CursorBench Pareto Frontier(2026-11-19, 2 posts)
- Episode 7: Gemini 4 Neon Wins Praise Across Dev Circles for Strength and Value(2026-11-19, 9 posts)
- Episode 8: Google's Gemini 4 Neon Tops Vending-Bench 2 Agent Benchmark(2026-11-19, 2 posts)
- Episode 9: Alleged Gemini 4 Neon Benchmarks Leak, Leading Opus 5.5 and GPT-6.2(2026-11-19, 2 posts)
- Gemini 4 Neon's 39% SWE-sweep questioned: results hinge on the harness — OfirPress · 2026-11-19
- Same weights, 3x score: Gemini 4 Neon jumps 12.8%→39% on SWE-sweep by swapping the harness — Delicious_Character6 · 2026-11-19