ANEMLL Forge 0.2.1 runs Qwen3.8-27B at up to 192K context on M6 Mac Mini's Neural Engine
Competitive-Bake4602 · reddit · 2026-10-25
ANEMLL Forge 0.2.1 brings Qwen3.8-27B to the M6 Mac Mini (32GB) on the Apple Neural Engine with up to 192K context. New: INT8 K/V cache (half of FP16) plus a smaller drafter sharing the target's output head — 0.86 GB less ANE memory at the same speed. Decode varies by content: JSON 35–62 tok/s, code 18–32, prose 10–17; prefill is 260 tok/s short-context, 62 at 192K. Full 192K compile takes 2h once (cached), but --ctx exports cut this — 16K runs 10 min from download to server. The author calls 96K the sweet spot on 32GB: 123 tok/s prefill, 27 tok/s decode, 21.4 GB memory. Release notes, a paper on Three Bit MLP Vector Quantization in ANE, and a Tetris-over-150K-tokens demo video are linked.
More from Infra
- Jensen Huang unveils Vera Rubin DSX POD at GTC Berlin, pitches Europe's AI economy — nvidia · 2026-10-25
- TensorSharp ships Day-1 MiniMax-M3.1-Flash support: 9.8× faster 64K prefill than llama.cpp on 8× A40 — fuzhongkai · 2026-10-25
- Compute contracts vs reality: Nscale has only 5% of contracted GPUs live, TeraWulf 12% — AccBalanced · 2026-10-25
- Live MCP rewrite from Node to Rust cuts memory usage by roughly 8x — draginol · 2026-10-25
- 305M-Parameter Sparse MoE Runs Computer-Use Agent Fully Offline on a MacBook Air — lmoroney · 2026-10-25
- SGLang reruns DeepInfer on matching hardware, matches or beats its prefill performance — hsu_byron · 2026-10-25