Redditor flags Unsloth's LiveCodeBench jump: all 5 models up 2-3 points, odds ~1 in 4 billion
Positive-Resource922 · reddit · 2026-11-03
A Redditor compared two published LiveCodeBench v6 charts for DeepSeek-V4.1-Flash-1017 and found all four Unsloth quantizations plus the Q8 baseline simultaneously improved 2-3 points (e.g. IQ1S 88.95%→91.0%), while GSQ-RCO results moved inconsistently (-0.4 to +1.9). The first chart's percentages correspond exactly to 525 outcomes (175 problems × 3 generations), while Unsloth's updated methodology reports 12 generations per problem (2,100 outcomes). Under a one-sided hypergeometric test assuming the original outcomes are included, per-model probabilities of such a low initial result range 0.26%-4.14%; assuming independence across models, the joint probability is 1.24×10⁻¹⁰ to 2.59×10⁻¹⁰ — roughly 1 in 3.9-8.0 billion, though the author cautions the models are correlated so this shouldn't be read literally. Unsloth responded that the first chart was a single run followed by additional evaluations with unchanged settings.
More from Models
- How LLM inference actually works: from 140GB weights to GPUs and model serving, explained as a story — _jaydeepkarale · 2026-11-04
- $100 of GPU time edits model weights to cut thinking tokens by 40% — jarrodwatts · 2026-11-04
- Sarvam open-sources Prism, a 44ms decision model that skips text entirely — heyshrutimishra · 2026-11-04
- Frontier models' teaching ability is 'off the charts', says dev: your 'I know kung fu' moment — shauseth · 2026-11-04
- kalomaze: crude diffusion autoencoders may lose low-frequency info at the latent stage — kalomaze · 2026-11-04
- Dev complains frontier models can't do light-touch English editing — A_K_Nain · 2026-11-04