LlamAmpere v0.7: Qwen 27B at Full Context on 12GB Cards, 70 TPS

Brief-Tap-6616 · reddit · 2026-11-18

LlamAmpere v0.7 targets small-VRAM cards: a new 12GB 27B quant runs full context with 70 TPS and MTP, cutting error 70% vs the Oct 9 2.3bpw file (96% top-1 match vs BF16), and is 25% lower-error yet 8% smaller than updated unsloth quants while supporting 200K ctx. Key tricks: dynamic microbatching (start with the largest fitting microbatch, periodically shrink as context grows, freeing hundreds of MB) and Latent Sparse Attention (project tokens into a 64-dim index space, cluster, exact attention only on best clusters; keep 8K recent tail + first 128 tokens + 20% mid tokens for 10-40x less attention error, cache moves most context off-card). 10GB cards now fit what needed 12GB (213K headless ctx); an 8GB distill is in progress. MIT-licensed on GitHub.

Original post →

More from Infra

Infra channel →