Open-source inferd runs 128 concurrent agents on one Mac at 1,487 tok/s
thursdai_pod · x · 2026-11-06
John Davies launched inferd live on the Thursdai podcast: an open-source (Apache 2.0) macOS inference engine built for agent swarms.
- Performance: 128 concurrent agents on a Mac Studio M5 Ultra (256GB) at 1,487 tok/s total, running Qwen3.8 models.
- Efficiency: measured at the wall with a smart plug — 293–300W for the whole machine; going from 64 to 128 agents raised output from 1,326 to 1,487 tok/s at roughly the same power (4.4 → 5.1 tok/J).
- Hardware range: works from a $1,000 Mac mini to a $20,000 Mac Studio, keeping data local with no per-token billing.
- Drop-in integration: speaks the OpenAI API (just change the base URL); agent features like pinned contexts, sessions, per-agent effort and priority come via an extra field on the same request.
More from coding & agent
- GitHub's new Git storage handles 7,200 pushes/sec from 5,000 concurrent agents — arpit_bhayani · 2026-11-06
- GitHub rebuilds Git storage for thousands of concurrent AI agents — arpit_bhayani · 2026-11-06
- OpenAI built Pando for Codex: 5x lower Git push latency, ~1PB transferred daily — sven_ai · 2026-11-06
- Snorkel AI: agents peek at reward code ~25% of the time rather than asking users — ajratner · 2026-11-06
- Subagents fail because summaries are lossy — checkpoint agents fork context instead — narphorium · 2026-11-06
- Building a pure-code lightbulb with Claude: WebGL shader, MediaPipe tracking and self-made test videos — ramramlab · 2026-11-06