Gemini 3.8 Flash eval: strong at numerical methods, mogged by Qwen 3.8 27B on checkpoints
AnuranBuilds · x · 2026-09-03
PhyseraAI benchmarked Gemini-3.8-Flash on its TB bench across 5 random tasks: it excels at deriving and implementing coherent numerical methods, but is inconsistent on interacting edge-case semantics, overbuilds static-analysis solutions while missing the hardest coverage cases, and long trajectories often mean speculative scope expansion rather than better outcomes. Another developer confirms Gemini's random-quality checkpoints make it lose badly to smaller models like Qwen 3.8 27B on checkpoint/sequence-based work.
More from Models
- AxiomProver tops LeanEval, the last unsaturated math formalization benchmark — BenBlaiszik · 2026-09-03
- Muse Spark 1.3 calls user 'Judah' then denies it, users report odd behavior — fragment_me · 2026-09-03
- Google AI Mode shows zero citations on high-level TOFU queries, SEO tests find — gaganghotra_ · 2026-09-03
- Fable 5.1 halves agent failure rate to 7% with 0.7% hallucinations, at 1.8x the cost — ryanshrout · 2026-09-03
- Seroter Daily #859: Gemini 3.8 Flash, agent telemetry, and 7 agent skill patterns — rseroter · 2026-09-03
- Insider leak: OpenAI's Astra tested as 'ultima-alpha' and 'vega-alpha' checkpoints — Ok_Display_3159 · 2026-09-03