tiny-llm: Open Course Builds a Mini vLLM Inference Engine on Apple Silicon
alexcovo_eth · x · 2026-11-05
Someone turned the hardest parts of LLM inference engineering into a course where you fix the performance problems yourself: tiny-llm, an open-source course for building a small vLLM-style inference engine around Qwen3 on Apple Silicon.
The four-week hands-on curriculum covers implementing attention and text generation, adding a KV cache to reuse computation, working with packed 4-bit weights, writing faster ops in C++ and Metal, and handling concurrent requests with batching and paged memory. It also covers a small coding agent with tool calls, checkpoints, and context management, with exercises and reference solutions at each stage.
More from coding & agent
- Box taps Nvidia's Nemotron: 4x faster queries, 90% fewer tokens for agents — gaganghotra_ · 2026-11-05
- Vercel's agent-desktop reads accessibility trees to automate desktop apps, 16.4k stars — victor_explore · 2026-11-05
- IOTA Decide 4B open-weight decision model hits ~11 decisions/sec on a Mac mini — const_reborn · 2026-11-05
- AI agent audits mom's APOE E3/E4 genetic report, with a second AI trying to break the work — kevrussell · 2026-11-05
- Public benchmarks won't tell you which AI model works for your stack, Microsoft advocate warns — HamelHusain · 2026-11-05
- 12 AI coding agents reviewed: every notable October changelog in one roundup — Ok-Brush8377 · 2026-11-05