Enabling embeddings on Qwen3 27B halves token generation speed in llama.cpp
No_Advance3911 · reddit · 2026-09-15
A user testing Qwen3 27B's built-in embeddings support in llama.cpp reports decent embedding quality, but with a major cost: enabling embeddings cuts token generation speed roughly in half versus running the model normally.
They're asking whether generation performance can be preserved with embeddings on, or whether llama.cpp can switch between tasks without unloading and reloading the model each time — a practical problem for running one local model for multiple workloads.
More from Infra
- LFM2.5-24B Hermes adapter ships as GGUF: local agents with 20% fewer tool calls — yb2698 · 2026-11-02
- BofA: B200 and H100 GPU rental prices hit new highs, up 47% and 31% YTD — TheZachMueller · 2026-11-02
- vikhyat predicts alternating compute oversupply and undersupply on the road to AGI — TheZachMueller · 2026-11-02
- PyTorch Mac downloads hit 3.29M in October, up 20.5% MoM — LysandreJik · 2026-11-02
- CVSS 9.8 ingress-nginx flaw lets unauthenticated attackers get RCE and steal cluster Secrets — evilsocket · 2026-11-02
- RAM prices tripled, so Tesla cut a third of memory from Optimus's brain — aakashgupta · 2026-11-02