112 TPS on a single RTX 3090: LlamAmpere v0.6 pushes 27B models with 400K context
Brief-Tap-6616 · reddit · 2026-10-28
LlamAmpere v0.6 ships with major local inference gains on a single RTX 3090:
- 13% decode speedup, averaging 112 TPS across the 100-105K token interval; prompt processing up 20%, starting around 1800 TPS and holding 1000+ through the first 100K
- New Ampere Integer+ quant format: roughly EXL3-level KLD at significantly faster speeds; the recommended model has 10% better KLD and is 1.2 GB smaller. EXL3 kernels also upgraded for 8% speedup
- SJKVaRN enables over 400K total context at 23.5 GB usage
- Recommends the 3090/ti GPU unlock; MOE speedups and smaller quants for 12 GB cards coming this week
More from Infra
- 7.25M Gitee Repos Scanned, 26,400 Live Secrets Found Using Modal, TruffleHog and Kingfisher — charles_irl · 2026-10-28
- DeepSeek rerun through Fireworks reproduces results; original dip was likely a fluke — teortaxesTex · 2026-10-28
- NIST Releases Official Post-Quantum Cryptography Migration Guidance as Harvest-Now-Decrypt-Later Risk Grows — entrup · 2026-10-28
- Decision Models cut inference to ~100ms, making the retriever RAG's new latency bottleneck — CShorten30 · 2026-10-28
- Infinity: an open-source Rust agent runtime that treats concurrency as a scheduling problem — ShadajL · 2026-10-28
- vLLM PR Adds Native Nemotron-3.5-Transcribe Support: Transcription+Chat, FP8/NVFP4 — vllm_project · 2026-10-28