95+ TPS and 262K context for Qwen 27B on a single RTX 3090 with LlamAmpere v0.4
Brief-Tap-6616 · reddit · 2026-09-29
Developer JakeATX released LlamAmpere v0.4, a Llama.cpp fork with Ampere-specific optimizations, hitting 95+ TPS and 262K context running Qwen3.8 27B (4.6bpw) on a single RTX 3090 — 10% faster than v0.3 with 10%+ more context. Closest rival vLLM stays within 10% but caps context lower. The post includes full build/run commands, notes EXL3 speedups (80% of XS-M quant speed), a Swift-qwen distill base with 1% performance loss, and benchmarks at temp=1 to avoid vanity temp=0 numbers. Open source on GitHub/HF.
More from Infra
- Sarvam AI Launches Token Factory: India-Hosted Inference for Open Models — itsOmSarraf_ · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Stratechery's Fictional Obituary: Imagining Jensen Huang's 2026 Exit From Nvidia — Gregory_C_Allen · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16