Running Qwen3.5 9B/27B INT4 on cheap ex-mining FPGA boards
I_am_purrfect · reddit · 2026-10-05
Background
The author wanted FPGA LLM inference for a while; Qwen3.5 finally made the 9B-27B scale compelling. Cheap hardware: SQRL FK33 ($280, 8GB HBM2, 400GB/s, eBay ex-mining card), later the dual-VU35P Jungle Cat ($375). The RTL was largely implemented with Claude Opus 4.8/5.5 and Kimi K3.
Measured results
- Qwen3.5-9B INT4, 2x FK33 @75MHz pipeline split: 6 tok/s prefill (256-token prompt), 3.2 tok/s generation early, 2.4 tok/s at 2-3k context; outputs verified layer-by-layer against llama.cpp
- Qwen3.8-27B INT4 (modelled from 9B per-op profile, not yet run): 1.1 tok/s generation with current design on two dies; resized RTL at 200MHz 8 tok/s; 4x VU35P tensor-parallel at 200MHz estimated 25 tok/s short-context, 10 tok/s at 16k, 1 tok/s at 262k. Two dies cap out around 45k context (KV cache won't fit beside 14.5GB of weights); full 262k needs four dies.
ASIC estimate
An Opus 5.5 estimate puts this RTL on TSMC 2023 N3 with 6 stacks of HBM3 (4.9TB/s) at 2GHz at roughly 294 tok/s short-context, 106 tok/s at 16k, 10 tok/s at 262k for the 27B, at 125-340W. Two BC-250s bought for $60/$75 proved an incredible value.
MIT-licensed repo: llm.vhdl
More from Infra
- Ex-Waymo exec Abe Murray: CUDA and ISAs won't stay moats, huge chip innovation incoming — IgorCarron · 2026-11-20
- MIT and NVIDIA's QDA uses agents to auto-design LLM quantization recipes, beating top public ones by up to 2.2 points — omarsar0 · 2026-11-20
- Caixin: US remote-access cutoff and Nvidia CEO exit leave China's tech giants halting overseas compute deals — Gregory_C_Allen · 2026-11-20
- QuixiInfer: Rust Engine Runs 125B MoE Model on a Single RTX 5090 — QuixiAI · 2026-11-20
- QuixiInfer: from-scratch Rust engine runs DeepSeek with 256k context on a single RTX 5090 at ~30 tok/s — QuixiAI · 2026-11-20
- AMD's Lisa Su: data center to hit ~80% of revenue by 2027, sees $220B server CPU market — AccBalanced · 2026-11-20