2.25bpw DeepSeek-V4.1 quant nearly matches official API on 100-question test
PhysicalIncrease3 · reddit · 2026-10-27
Testing ISTA-DASLab's 2.25bpw quant of DeepSeek-V4.1-Flash against the official API on a 100-question text subset of Humanities Last Exam: local scored 13 strict / 20 semantic correct vs API's 13 / 21; of 74 API-completed questions, only 2 local-only vs 4 API-only misses, with 30+ identically wrong answers suggesting similar behavior. The author candidly details methodology flaws: different token-limit handling (API times out with no output), thinking 'high' not 'max', small sample with high variance, 12 hours at 40.4 tok/s. Conclusion: the extreme quant isn't lobotomized—anything the API can answer, local roughly can too.
More from Infra
- Anthropic's S-1 suggests ~70% of its compute goes to internal R&D, not customers — scaling01 · 2026-10-27
- Cloudflare: GLM 5.3 on Workers AI leaked other customers' data into 155 responses — djcroman · 2026-10-27
- Stop sizing LLMs by static leaderboards; compute the API-vs-self-host breakeven instead — zackku · 2026-10-27
- InferNEON runs MoE LLMs on dirt-cheap boards with a bare-metal Rust kernel and ternary experts — paraschopra · 2026-10-27
- 125 tokens/s on an RTX 4070 Ti SUPER: running a local coding agent on 16GB VRAM — ievkz · 2026-10-27
- ApexW: a Triton PyTorch optimizer that halves optimizer-state memory, 1.5x faster steps — Ready_Parking_5435 · 2026-10-27