MathArena flags Fireworks inference issues as DeepSeek-V4.1-Flash lands at 49% for $1.29
teortaxesTex · x · 2026-10-27
MathArena added DeepSeek-V4.1-Flash (10/17) and found it a solid improvement over its predecessor — though still below Kimi K3.1 Pro at 3x lower cost. Lead Jasper Dekoninck revealed the first run looked worse until they traced it to inference provider Fireworks misbehaving; rerunning via Together was more reliable, though 21 traces still failed simultaneously with tool-call formatting errors. His takeaway: inference providers aren't yet reliable enough to trust blindly — always verify evals before publishing. The leaderboard shows GPT-6.1 Sol (max) at 91.51%, Gemini 4 Argon 87.74%, Claude-Opus-5.5 (high) 82.08%, and V4.1-Flash-1017 (Max) at 49.06% for just $1.29 with a hefty 676K tokens.
More from Infra
- Dragonfruit runs its ventures on Apple Silicon with fixed-fee open-weight inference — akumar · 2026-10-28
- Token pricing has no ceiling: agents burn compute without breaks, reshaping enterprise AI billing — akumar · 2026-10-28
- Vjeux's AI agents hit 2.6x speedups on Open Kernel Challenge kernels, then stalled — Vjeux · 2026-10-28
- "My job is to make tokens as cheap as humanly possible" — every chip, every watt, every acre — GregKamradt · 2026-10-28
- General Compute raises $100M Series A to bet on fast inference demand — _akhaliq · 2026-10-28
- Microsoft's custom Maia chips to make up only ~10% of its AI chip spend by early 2027, says Evercore — Beth_Kindig · 2026-10-28