llama.cpp adds tensor split for LFM2/MoE, boosting inference performance significantly
pmttyji · reddit · 2026-08-21
A PR submitted to the llama.cpp repository adds --split-mode tensor support for the LFM2 and LFM2MoE model families, accompanied by detailed benchmarks on 2x RTX A5000 (NVLink).
Feature Update:
- Enables tensor splitting for LFM2/LFM2MoE models, allowing the model to be distributed across multiple GPUs.
Accuracy & Performance Benchmarks:
- Accuracy: Tests passed with Meta NMSE around 1e-14. Perplexity (PPL) remained stable across quantization levels, with high top-p prediction consistency (>90%).
- Performance Gains:
- LFM2.5-2.6B: Token generation speed improved by 8-35%, prompt processing by 24-34%.
- LFM2.5-VL-3B: Prompt processing improved by 23-31%, token generation by 14-23%.
- LFM2.5-8B-A1B: Prompt processing improved by 28-31%.
- One configuration (8B-A1B BF16) showed a regression in prompt processing (-35.1%), but most scenarios showed significant gains.
More from Infra
- Wake: macOS app unifies chat history for 13 code agents — aigclink · 2026-08-21
- Kubernetes CPU Limits Make Apps Slow and Costly: Proof and Experiments — JeremyCMorgan · 2026-08-21
- Productionizing AI Apps: OpenTelemetry, On-Call Agents, and Full Observability Workflow — Al_Grigor · 2026-08-21
- LLMRouter 2.0: Unified Infrastructure for LLM Routing Dev and Eval — youjiaxuan · 2026-08-21
- Run MiniMax H3 locally on 12GB GPUs: 15-second multi-shot ComfyUI template — vortis23 · 2026-08-21
- Used RTX 3090 purchase review: Local AI performance crushes 3060, Qwen 35B 20x faster — Yanzihko · 2026-08-21