Sebastian Raschka: open-source LLMs need a building-block API for swapping components
rasbt · x · 2026-11-16
Sebastian Raschka argues the open-source LLM ecosystem lacks a unified building-block API: swapping GQA/MLA, dense FFN/MoE, attention/Mamba2 layers carries high switching costs even with LLM-assisted coding. Existing efforts (torchtitan, torchtune, Megatron Core, OLMo-core) fall short on abstraction, and such an API needs serious engineering and production-grade efficiency to matter — plus it faces a chicken-and-egg adoption problem. An attached mockup shows a PyTorch-style declarative nn.LLM API for mixing component choices.
More from Infra
- Sen. Cantwell blasts Commerce Dept over AI chip export control failures in letter to Lutnick — ShakeelHashim · 2026-11-16
- AI stock rout intensifies for a second day: Nebius -9%, Marvell -7%, CoreWeave -6% — GaryMarcus · 2026-11-16
- Baidu Q3: AI Cloud Infra Revenue Up 94%, GPU Cloud Surges 191% — Baidu_Inc · 2026-11-16
- Llama Windows app hits 100 tok/s running Liquid AI's LFM Omni on an RTX Spark laptop — LysandreJik · 2026-11-16
- llama.cpp team launches Llama desktop app: local AI that never leaves your computer — mervenoyann · 2026-11-16
- Alibaba, Tencent, ByteDance all run overseas data centers, but not every Chinese hyperscaler buys Ascend — teortaxesTex · 2026-11-16