Kangwook Lee details inference setup: W4A16 quant, ~120 tok/s single-stream without drafter
Kangwook_Lee · x · 2026-10-22
Answering follow-up questions about test conditions, Kangwook Lee confirms the speed figures were measured with W4A16 quantization, short inputs, and concurrency of 1, at around 120 tok/s without a drafter. Details on TTFT, peak memory, and high-concurrency performance will be shared by @ugonfor — useful for judging whether the setup suits local interaction or high-concurrency serving.
More from Infra
- Deutsche Bank Deploys Farsight On-Prem as Financial AI Question Shifts to Where Models Run — huggingface · 2026-10-22
- Gradio distills FLUX.3 Klein's 9B prompt upsampler into 0.8B and 2B models that run on CPU — Gradio · 2026-10-22
- Securing coding agents with Docker Sandboxes and edge LLM inference at ATO — HowDevelop · 2026-10-22
- Revolut's massive outage blamed on GCP, but ex-sales contact says it rejected multi-zone redundancy — thedealdirector · 2026-10-22
- 0.9B Model Transcribes a 4-Person Meeting on a MacBook With 0.08% Speaker Confusion — Binary_orchid · 2026-10-22
- Qdrant 1.20 ships BM25 hybrid search without sparse vectors — qdrant_engine · 2026-10-22