2.3bpw quant update: 27B model on 12GB cards ties Qwen3.6 BF16 on LiveCode Bench at 84.3%
Brief-Tap-6616 · reddit · 2026-11-02
The author released an updated 2.3bpw quant of a Qwen 27B model for 12GB Ampere cards, now tying Qwen3.6 BF16 on LiveCode Bench (84.3% strict vs 83.9% BF16, three runs, xhigh reasoning), up from 75% in the previous checkpoint — a 60% reduction in LCB degradation.
Technical details
- Enabled by the new AmperInteger+ format in LlamAmpere 0.6: fractional 2-3bpw quantization with trellis-based encoding plus an RCO-like optimization, with minimal post-training (<20M tokens) to avoid overfitting.
- KLD dropped from 0.096 to 0.059 (37-57% error reduction); verbosity also fell (median answer 4.5K vs 6K tokens).
- 7.4GiB on-card footprint including MTP head; upgraded embedding table stays in system RAM. SJKVaRN 3/2 config enables 200K+ context on one 12GB card.
- Over 90 tps for agentic workloads on 3090/80ti; MIT licensed, requires latest llamAmpere.
More from Infra
- Crusoe CEO says neocloud debt fears overblown amid 'tidal wave' of compute demand — nathanbenaich · 2026-11-02
- Kapitsa hits 346 tok/s on a 35B MoE and runs a 180B model on 48GB Macs via SSD offloading — DotaMate · 2026-11-02
- Texas Mega-Project Pipeline Tops $500B, Led by $100B Stargate AI Hub — MatthewChang · 2026-11-02
- SemiAnalysis: 30.3GW of planned 2028 US data center capacity at risk of pause after midterms — Eliv_nurotic · 2026-11-02
- B300 GPU rental prices plunge from $7.9/hr to $4.6/hr in a month as providers beg for customers — SumitGup · 2026-11-02
- FT chart: Big Tech's surging CapEx is a rational bet, not reckless spending — luisdans · 2026-11-02