laya.cpp: standalone C++ inference makes open-source Laya 2.5x faster at 366 q/s
lkarlslund · reddit · 2026-09-21
A community developer built laya.cpp, a standalone ggml-based C++ inference implementation of the open-source decision model Laya, with custom CUDA kernels, native tokenization, and a JEV-compatible HTTP endpoint—no Python or PyTorch required. On an RTX PRO 6000 Blackwell capped at 450 W, BF16 hits 366 questions/s at batch 1 (2.5x the Python implementation) and 810 vs 663 at batch 8. Gains come from removing redundant conversions/copies, fusing ops while preserving rounding, and better attention memory access. MIT-licensed; BF16 needs CUDA 13.0/cuBLAS 13.1.0. Built using Codex Astra.
More from Infra
- Four Intel B70s vs a 128GB MacBook for Local LLMs: A Buyer's Dilemma — Rokett · 2026-09-21
- Jeff Dean says RL plus new EDA tooling could compress chip design from 2 years to 3 months — ycombinator · 2026-09-21
- fal tests Serverless cluster mode for on-demand 1K H200 RDMA clusters — isidentical · 2026-09-21
- Meta data center floods Louisiana school district with $50,000 teacher bonus checks — victor_explore · 2026-09-21
- Agent Substrate open-sourced: agent runtime with 10x container density, sub-500ms resume — rakyll · 2026-09-21
- Huawei's Guo Ping: data centers next need 'book nerds' to route tokens, not coders — teortaxesTex · 2026-09-21