AMD and Cerebras Split LLM Inference: KV Cache Bridges the Hardware Gap
TheTuringPost · x · 2026-08-02
AMD and Cerebras are building a novel AI infrastructure that splits LLM inference into two phases across different hardware architectures:
- AMD Helios handles the prefill phase, reading the prompt and building the KV cache.
- The KV cache and request metadata are then transferred to a Cerebras CS-3 wafer-scale engine.
- Cerebras uses this cached state to generate response tokens at extremely high speeds.
This heterogeneous combination is transparent to the user but significantly boosts data center efficiency. The companies claim up to a 5x improvement in tokens per second per watt. However, the latency of transferring the KV cache across systems will be the biggest potential bottleneck for this architecture.
More from Infra
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- GYRO quant of Qwen3.8-Flash-Next tested: near Q3 quality at much smaller size — PlasticRevenue4601 · 2026-10-16
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16