Unsloth's 35B-token benchmarks challenge GSQ-RCO's DeepSeek quant recovery claims
danielhanchen · reddit · 2026-11-02
Unsloth (danielhanchen) published large-scale benchmarks of DeepSeek-V4.1 GGUF quants, claiming its Dynamic v3.1 is 10-20% better than GSQ-RCO at 512K context on agentic tasks—and statistically challenging GSQ-RCO's recovery-rate claims.
Method upgrades: context raised from 32K to 512K (agentic) / 128K (general); unsaturated agentic benchmarks (Terminal Bench 2.1, DeepSWE v1.1, Automation Bench); 35.1B benchmark tokens vs 1.3B before (27x), with 64x AIME2025 runs and 3x DeepSWE/TB; better graders including Math Verify and official Harbor scoring for Terminal Bench.
Key findings (charts show most GSQ-RCO quant points well below Unsloth and the 8-bit baseline on agentic benchmarks):
- GSQ-RCO IQ1M claims 98.7% recovery, measured 97.24% [96.23, 98.21], p=0.0013—significantly overstated
- IQ2XXS claims 99.80%, measured 98.38%, p=0.0001—"nearly impossible"
- IQ2XS at 99.54% measured vs 100% claimed is statistically plausible
- Paired tests show IQ1M significantly below the official baseline on AIME25, GPQA-D and LCB v6
All methods and data are public on unsloth.ai docs; the new Dynamic 3.1 GGUFs are on Hugging Face (unsloth/DeepSeek-V4.1-Flash-1017-GGUF), and Unsloth invites GSQ-RCO to share its run details for re-testing.
More from Models
- dbreunig: stop treating OpenRouter usage data as macro AI trends — dbreunig · 2026-11-03
- Frontier models flunk 1,000-variant DNA classification: refusals, token blowouts, encrypted reasoning — _lewtun · 2026-11-03
- GPT-6 Astra tops robot index, but finishes dual-arm sim task only twice in 800 trials — minchoi · 2026-11-03
- Qwen Head Dispels Rumors: Qwen 4.0 and 5.0 Will Stay Open Source — ResearchCrafty1804 · 2026-11-03
- Plus Subscriber Complaints About OpenAI's Opaque Usage Limits to UK's CMA — YouHadTooMuchToThink · 2026-11-03
- OpenAI's Cyber Guardrails Softlock a User's Legit Security Audit Mid-Task — Linuxxsxx · 2026-11-03