LlamAmpere v0.7: Qwen 27B at Full Context on 12GB Cards, 70 TPS
Brief-Tap-6616 · reddit · 2026-11-18
LlamAmpere v0.7 targets small-VRAM cards: a new 12GB 27B quant runs full context with 70 TPS and MTP, cutting error 70% vs the Oct 9 2.3bpw file (96% top-1 match vs BF16), and is 25% lower-error yet 8% smaller than updated unsloth quants while supporting 200K ctx. Key tricks: dynamic microbatching (start with the largest fitting microbatch, periodically shrink as context grows, freeing hundreds of MB) and Latent Sparse Attention (project tokens into a 64-dim index space, cluster, exact attention only on best clusters; keep 8K recent tail + first 128 tokens + 20% mid tokens for 10-40x less attention error, cache moves most context off-card). 10GB cards now fit what needed 12GB (213K headless ctx); an 8GB distill is in progress. MIT-licensed on GitHub.
More from Infra
- Bloomberg: New US Chip Rule Freezes $480M Nvidia GPU Loan for ByteDance in Malaysia — rohanpaul_ai · 2026-11-18
- AI raises the cost of weak cloud architecture, says resilience expert — DavidLinthicum · 2026-11-18
- Harvard paper: 94.8% of coding agent tokens are cached re-reads, 25x energy gap — rohanpaul_ai · 2026-11-18
- French team drops vLLM/PyTorch for Rust+CUDA and FP4/FP8, huge single-GPU inference gains — IgorCarron · 2026-11-18
- Qwen4-Max lands on DeepInfra with 1M context at $1.413/M input tokens — gharik · 2026-11-18
- TensorScale Visualizes Every Inference Stage and Benchmarks Against a Leading Hosting Provider — TerryTangYuan · 2026-11-18