$21 of DeepSeek API wrote Molten: a Rust+Metal engine serving 300+ tok/s on M5 Macs
GabGarrett · x · 2026-10-26
Developer zhangshuibai used DeepSeek V4.1 Flash as a coding agent and, for about $21 of API spend, built Molten in roughly a week — an open-source (Apache-2.0) inference engine purpose-built for LiquidAI's LFM2.5-24B-A2B on Apple's M5 Max.
Key idea: general frameworks like mlx-lm and llama.cpp must run any model on any hardware; a specialized engine serves one model on one chip at batch size 1, leaving far more optimization headroom.
Numbers (M5 Max, 128GB):
- 305 tps at 256-token context vs 190 (mlx-lm) and 199 (llama.cpp)
- Still 248 tps at 16K context; 1.5x decode speedup, 1.28x end-to-end over mlx-lm
- Timeline: first correct engine at 0.56x after 3 hours, surpassed mlx-lm on day 2.5, hit 1.28x by day 5.7
Form: native Rust + Metal, a single 6 MB binary, no Python/PyTorch/MLX; ships a chat CLI and an OpenAI-compatible server with streaming and tool calls for coding agents. Requires an M5-series Mac with 24GB+ RAM. The author frames it as recursive self-improvement — an AI writing its own inference engine.
More from coding & agent
- Collage Homage to Maggie Appleton's 'Planning with Agents' Talk — GrantCuster · 2026-10-26
- Feed Your Daily Tools to an Agent: A Recipe for a Workspace Built Around Your Business — EXM7777 · 2026-10-26
- Why AI filmmakers should build small tools for their slowest workflow step — lmoroney · 2026-10-26
- Kitebook: self-hosted open-source MCP server lets AI assistants hand off work to each other — ChainEnough5654 · 2026-10-26
- Watch an AI agent build a Minecraft scene block-by-block in a 6-hour timelapse — flowersslop · 2026-10-26
- Kornia author: AI agents skip library docs and read code/docstrings instead — ducha_aiki · 2026-10-26