tiny-llm: an open-source course where you build a vLLM-style inference engine for Qwen3 on a Mac
techNmak · x · 2026-10-30
tiny-llm is an open-source hands-on course for systems engineers, modeled as an LLM-serving counterpart to CMU's Needle project. Built around Qwen3 and runnable entirely on a Mac, it has you construct a miniature vLLM-style inference engine step by step.
The four-week path covers:
- Week 1: attention, RoPE, grouped query attention, RMSNorm/MLP, model loading, and the decoding loop in plain matrix ops;
- Week 2: real inference engineering — KV cache, packed W4 quantized matvec, SIMD matrix multiply, fused primitives (rmsnorm/rope/swiglu), tiled dense prefill attention;
- Week 3: continuous batching, chunked prefill, paged KV cache, paged FlashAttention, plus optional MoE and speculative decoding;
- Week 4: building a validated agent loop, workspace inspection, checkpoint/resume, and agent evaluation.
Each chapter ships with code, tests, and docs; the core idea is learning inference end-to-end by fixing the performance problems yourself.
More from coding & agent
- exe.dev launches durable dev sandboxes with SSH root access and built-in coding agent Shelley — davidcrawshaw · 2026-10-30
- Solo dev building a game entirely with AI ships update 5: 494 medals, 28 vehicle types, Steam submission — Apprehensive-Law77 · 2026-10-30
- Dev ditches unanimous-vote stopping rule for a judge in multi-AI debate project — lluisantoni · 2026-10-30
- Replit can now run any GitHub repo from a link and auto-open PRs — amasad · 2026-10-30
- GitHub Octoverse: 34.3B commits in a year as agents start 40% of PRs — lemire · 2026-10-30
- Google Antigravity launches plugins marketplace with Jira, Figma, Stripe connectors — testingcatalog · 2026-10-30