Dev ports Inkling-2-Mini to a single AMD R9700: 196K context at ~110 tok/s
abhinand05 · reddit · 2026-11-22
A developer ported Inkling-2-Mini's official NVFP4/MXFP8 checkpoint to vLLM on an AMD R9700 (32GB, RDNA4) — upstream only supports NVIDIA and MI355X — releasing it as the open-source vllm-redshift project.
Key work and results:
- RDNA4 has no FP4 hardware, so custom kernels (written with AI help) decode 4-bit weights exactly on the fly, no requantization
- Custom attention kernels plus an FP8 KV cache fit 196K of the model's 1M context with 2 concurrent requests in 32GB
- Worked around ROCm launch overhead, a segfaulting system ROCm, and an idle CPU core pegged at 100%
- 80 tok/s decode, 80-115 tok/s with built-in speculative decoding (llama.cpp Q8: 31), 4,600 tok/s prefill; tool calling, reasoning and images all work
The author plans day-one ports for upcoming releases like Qwen4, plus support for 24GB cards (7900 XTX) with custom quants.
More from coding & agent
- Auto Ontology MCP: open-source semantic layer lets Claude and ChatGPT query your DB safely — solyarisoftware · 2026-11-22
- 10 skills to fix the cookie-cutter look of AI-generated websites, drop into Claude Code or Cursor — sven_ai · 2026-11-22
- A catalog of 2,256 real agent use cases with copyable prompts, from bill-negotiating bots to 11-agent trading teams — sven_ai · 2026-11-22
- Disney Parks MCP Server offers structured queries for attractions, dining and Lightning Lane status — modelcontextprotocol · 2026-11-22
- Trade Intel MCP chains Comtrade, Census, and Treasury data for agents — modelcontextprotocol · 2026-11-22
- DeepSeek Harness v0.2.3-alpha.1 ships 14 changes: plugin market, PR review, multi-agent UI — teortaxesTex · 2026-11-22