Reverse-engineered Splash format ports Qwen3.8-27B to Mac, 88 tok/s at 64k on M5 Max

SeveralViolins · reddit · 2026-09-22

• A developer ported Swift-Qwen3.8-27B as a 4bit model for the Splash inference engine (macOS only). Splash documents the package shape but ships no converter, so the byte-level layout (section offsets, packing order, quantization rule) had to be reverse-engineered from the engine source and verified against the single reference package. Qwen 27B did the heavy lifting, with Opus tidying and spot-checking.
• Performance: 79–129 tok/s decode on M5 Max 128GB; at 64k context, 87.8 tok/s vs 36.6 tok/s for the same weights in oMLX (oQ4e-mtp, mixed 4/5-bit), with the gap widening at longer context.
• Gotchas: Splash's /v1 defaults to greedy decoding, but Swift wants temp 1.0 / topk 20 / topp 0.95, and the format has nowhere to carry generationconfig. — pass them explicitly (write a shim if using DSH).
• Licensing follows upstream projects; Swift Open License v1.0 is free commercially under $1M revenue, and that condition travels with the weights.

Original post →

More from Infra

• Whittle distills on HF: 27B-A3B MoE quant claimed to run on 8GB VRAM laptops — depressedclassical · 2026-09-22
• Qdrant benchmark: post-upload latency spikes are optimizers, tuned configs yield up to 100x faster search — qdrant_engine · 2026-09-22
• Inference-free SPLADE: retrieval at BM25-like query cost without per-query inference — qdrant_engine · 2026-09-22
• What Cloudflare can't do: D1 caps at 10GB, is single-threaded, and no real Postgres — Paimaamu · 2026-09-22
• Rackspace joins NVIDIA Cloud Partner Program with Blackwell pods for regulated enterprises — DavidLinthicum · 2026-09-22
• Bridging LLM Decision Readouts into DuckDB: Zero-Token Probabilistic Classification via LuaJIT UDFs — Shoddy_Telephone9702 · 2026-09-22

Infra channel →