From-scratch 13MB Rust engine runs 107GB model on two £300 Snapdragon boxes
WorldlyAd7946 · reddit · 2026-11-18
- ToRc is a from-scratch Rust inference engine, 13MB, unzip-and-run on Windows/Linux x64+ARM with an experimental Mac build.
- On two £300 Snapdragon X laptops pooling 64GB LPDDR5x, it co-schedules CPU+NPU+iGPU to run the 107GB Qwen3.8 Flash Next (Q4) at 67 tok/s read / 17.5 write on a 16k prompt, streaming experts from SSD.
- Other numbers: Qwen3.6 35B A3B at 406/54 tok/s (4k prompt, two boxes); on an i9-13900K it hits 243/44 vs llama.cpp's 143/24, writing 2–10x faster on 32k prompts — though not ahead everywhere.
- Supports multi-machine model splitting (USB4 tested), OpenAI/Anthropic APIs with tool calling and MCP, GGUF one-click conversion, and Auto-tune.
More from coding & agent
- Replacing a $150K/yr media buyer's weekly ad review with 10 Claude skills across 4 ad platforms — aryanXmahajan · 2026-11-19
- GPT-6.2 tops browser-use benchmark: 82% success, 35% cheaper, 2x faster — gdb · 2026-11-18
- Vibe-coded in 4 days: a 150-million-star galaxy in one 299 kB HTML file — BigTutor6739 · 2026-11-18
- Stateless Streamable HTTP MCP server on Cloudflare Workers (19ms) to shield FX agents from news whipsaws — No_Comedian_6913 · 2026-11-18
- After 8,000+ Claude sessions, he rebuilt Notion, Maps and Finder into one open-source agent OS — Conchylicultor · 2026-11-18
- Pydantic CEO to demo Agent Sandboxes: Rust-based sandbox for running agent code — samuelcolvin · 2026-11-18