ISTA says Unsloth's 32K reasoning budget, not hidden tweaks, explains DeepSeek-V4.1-Flash benchmark gap
Loginhe · reddit · 2026-10-31
ISTA responded to Unsloth's recent quant comparison of DeepSeek-V4.1-Flash-1017, attributing the discrepancy to benchmark constraints rather than hidden adjustments.
- ISTA says it used standard settings: 128K context window, reasoning effort 50, no modified hyperparameters for the 763B-parameter, 1M-context model
- Key difference: Unsloth capped reasoning at 32K tokens, while ISTA's benchmarks allow up to 128K output tokens — that compute gap explains the variance
- Code is on GitHub and papers on arXiv; ISTA invites independent reproduction with exact settings and is re-running comparisons itself
- Prior models were benchmarked by third parties like Benjamin Marie (The Kaitchup)
More from Models
- Anthropic files for Nasdaq IPO: $16.3B H1 revenue, compute spend down to 0.5x revenue — DeepLearningAI · 2026-10-31
- ThursdAI crew debates: with Opus 7 coming, who still cares about better benchmarks? — thursdai_pod · 2026-10-31
- Swift 1.5 Flash Next Heretic quants add IQ4_XS and IQ3_S tuned for reverse engineering — jjusko20 · 2026-10-31
- 14 models tested on 960 analyst tasks: AI scores 97/100 as analyst, picks winners no better than random — HeyAmit_ · 2026-10-31
- Anthropic adds invisible text watermark to Claude outputs starting Nov 3 — petesena · 2026-10-31
- GPT-6 Luna vs GPT-6.1 Terra vs GPT-6.1 Sol vs GPT-6 Astra in Pixel Art — Angaisb_ · 2026-10-31