Sebastian Raschka: open-source LLMs need a building-block API for swapping components

rasbt · x · 2026-11-16

Sebastian Raschka argues the open-source LLM ecosystem lacks a unified building-block API: swapping GQA/MLA, dense FFN/MoE, attention/Mamba2 layers carries high switching costs even with LLM-assisted coding. Existing efforts (torchtitan, torchtune, Megatron Core, OLMo-core) fall short on abstraction, and such an API needs serious engineering and production-grade efficiency to matter — plus it faces a chicken-and-egg adoption problem. An attached mockup shows a PyTorch-style declarative nn.LLM API for mixing component choices.

Original post →

More from Infra

Infra channel →