tiny-llm: an open-source 4-week course to build your own mini vLLM from scratch
HowDevelop · x · 2026-10-31
tiny-llm (5.4k stars, Apache-2.0) is a hands-on course for systems engineers to understand LLM inference end to end — think of it as an LLM-serving counterpart to CMU's Needle: load a Qwen3 model, turn tokens into logits, and generate text, all implemented yourself.
Built on MLX arrays without high-level NN layers, each chapter has you implement operators in Python/C++/Metal, with MLX serving as the correctness oracle and performance baseline. The 4-week path:
- Week 1: build Qwen3 from mlx.core ops — attention, RoPE, GQA, RMSNorm, MLP, sampling, autoregressive loop
- Week 2: faster single request — KV-cache, W4 quantized matvec, SIMD matmul prefill, tiled prefill
- Week 3: build a mini vLLM — continuous batching, chunked admission, paged KV layout with FlashAttention reading pages directly
- Week 4: build a coding agent with validated loops, checkpoint-and-resume, and context compaction
Related event: Open-source course tiny-llm goes viral for building a mini vLLM(2 posts)→
More from Infra
- Opera proposes WebAI standard to let websites run local models in the browser — JChataigne · 2026-10-31
- India should build GPU-class accelerators, not yet another LLM, argues dev — PrasoonPratham · 2026-10-31
- India Should Build AI Chips, Not Another LLM, Argues Tech Commentator — PrasoonPratham · 2026-10-31
- Big Tech returns smallest share of cash to shareholders in 15 years amid AI spending — nordicinst · 2026-10-31
- At SF Compute's "no more compute" townhall, someone pitched me cheap B300 capacity — daniel_mac8 · 2026-10-31
- FastSDCPU Halloween release adds Fibo 2.0 Edit Turbo image editing via OpenVINO on CPU — simpleuserhere · 2026-10-31