laya.cpp: standalone C++ inference makes open-source Laya 2.5x faster at 366 q/s
lkarlslund · reddit · 2026-09-21
A community developer built laya.cpp, a standalone ggml-based C++ inference implementation of the open-source decision model Laya, with custom CUDA kernels, native tokenization, and a JEV-compatible HTTP endpoint—no Python or PyTorch required. On an RTX PRO 6000 Blackwell capped at 450 W, BF16 hits 366 questions/s at batch 1 (2.5x the Python implementation) and 810 vs 663 at batch 8. Gains come from removing redundant conversions/copies, fusing ops while preserving rounding, and better attention memory access. MIT-licensed; BF16 needs CUDA 13.0/cuBLAS 13.1.0. Built using Codex Astra.
More from Infra
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12
- Megakernel is multiprogramming for GPUs: 'GPUs are entering the CPU age' — pranjalssh · 2026-11-12