FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference
Xinyu Wang, Yalong Xue, Xiaotian Sun, Xiaoyu Zhang, Xinjiang Zhang, Chunmeng Dou, Xueqi Li, Xiaoming Chen
cs.AR
2026-07-11
FlashAccel co-places HBF with HBM on a GPU, hiding 4μs NAND reads via prefetch and layouts. Six-stack CSI hits 2.49x throughput and 1.93x energy efficiency vs H200 at 100ms SLO.
HBM capacity is no longer keeping up with LLM inference. Weights already take hundreds of gigabytes, and long context plus multi-turn agents push KV cache past what a single GPU can hold. On an H100 running Qwen3-32B, about 64GB goes to weights and 16GB remains for KV. That is eight 8K requests. Decode is memory-bound, so a tiny batch leaves compute idle. Evicted KV then forces recompute on later turns. Adding GPUs restores capacity, and also adds cross-node communication and cost.
High-Bandwidth Flash (HBF) is a NAND stack whose peak bandwidth is meant to match HBM, with higher density and non-volatility. Putting HBF next to the GPU looks like an easy capacity win. NAND read latency is about 4μs against roughly 100ns for HBM, a 40× gap. Peak bandwidth only appears when thousands of planes read at once. HBF, HBM, and SRAM also have no shared management stack. Extra capacity does not turn into throughput on its own.
FlashAccel is a hardware-software design from ICT, Chinese Academy of Sciences, that binds HBF, HBM, and GPU cores into one heterogeneous-memory accelerator. Two packagings are modeled. Cascaded integration (CSI) adds six HBF stacks beside the six HBM3e stacks of an H200, so both capacity and bandwidth grow. Co-located integration (CLI) swaps five of those HBM stacks for HBF and keeps one HBM stack: cheaper, and 16.7% less bandwidth. Each HBF stack is eight SLC Flash dies plus a base die, 192GB NAND and 32MB SRAM, rated 768GB/s. A die is split into 96 planes, then grouped 32-wide into a megaplane that shares command logic. The software-visible read unit is a hyperpage: one command that hits the same offset on every megaplane, so all planes fire together. Each plane gets 32KB of SRAM on the circuit die, and the base die holds another cache, used to prefetch and cover the 4μs array-read time tR.
Layout decides whether that peak is reachable. Weights are sliced into megapages in execution order, striped round-robin across megaplanes, and packed into consecutive hyperpages. A 12MB MoE expert by itself occupies only about 3072 of 4916 planes. Ordering by access, not by source-code declaration, keeps consecutive reads from stalling on a straggler plane that still holds an unrelated matrix.
KV cache is harder because the live request set changes every decode step. FlashAccel keeps PagedAttention KV blocks, aligns each per-layer slice of N tokens to a megapage, and gives each Flash block to exactly one request so eviction is an erase, not a migrate. Before a step, overloaded megaplanes copy excess megapages into HBM and redirect those reads. HBF can then issue hyperpage reads at the average load. Newly written KV sits in HBM until K megablocks are ready (K is the megaplane count), then all megaplanes program in parallel at hyperpage granularity. M=32 is the chosen tradeoff: larger megaplanes ease stragglers but force CLI to park more KV in the remaining HBM.
The storage layer drops a conventional FTL. Weights are immutable and KV is append-only, so the object index stores physical addresses and the GPU issues them directly. A small API maps HBF, HBM, and SRAM into one virtual address space: NandMmap for weights, GroupCreate / GroupMmap / GroupWrite for the live KV set, SramPrefetch / SramRelease for explicit SRAM staging. Prefetches are pulled out of the compute graph into a queue and run asynchronously. Page-table entries start on HBF and flip to SRAM when the copy finishes; kernels keep using the same virtual address.
The study is an event-driven simulator on top of LLMCompass. The baseline is an 8×H200 DGX node (141GB HBM3e and 4.8TB/s per GPU). Models: Qwen3-235B, Qwen3-Coder-480B, LLaMA3.1-405B, DeepSeekV3-671B. Two sequence mixes: long-context 8.11K/2.53K and agentic 15K/6K. Decode SLOs are 50ms and 100ms; each bar uses the largest batch that fits both the SLO and capacity. For LLaMA3.1-405B at 15K/6K, each request needs 10GB of KV, so 8×H200 tops out at batch 30; 8×CSI can theoretically hold 110 under a 50ms SLO after weights.
| Config | Per-GPU throughput vs 8×H200 at 100ms SLO | Note |
| 8×CSI | 2.49× | six HBF stacks beside HBM, headline number |
| 8×CLI | 2.00× | five HBF stacks replace HBM, 16.7% less bandwidth |
| 8×CSI, all settings | 2.10× | includes the 50ms SLO |
On Qwen3-235B at batch 256, full FlashAccel sits within about 4% of HBM-only latency. Turning off prefetch, weight layout, or KV layout cuts throughput by 55%, 7%, and 15%. A plain HBF GPU with none of the three loses 65% and falls below the HBM baseline. Prefetch is the largest of the three knobs.
A 50ms SLO is tighter. CLI slightly loses to the baseline on DeepSeek-V3 and Qwen3-235B because bandwidth is short. Four-GPU CSI/CLI cannot serve LLaMA-405B or Qwen3-480B at 50ms: loading weights alone exceeds the budget. Scaling to 16 GPUs pushes the HBM system off NVLink onto RDMA, about 9× less interconnect, and expert-parallel all-to-all eats the extra capacity. An 8-GPU HBF node still beats a 16-GPU HBM node on throughput per GPU.
For multi-turn KV reuse, 8×CSI and 8×CLI keep every session and hit the ideal hit rate; 16×H200 remains 50% lower. Tokens that still need compute drop by up to 89%. On the write path, CSI reads at 4.6TB/s but programs at 245.8GB/s. The heaviest case writes 988MB of KV per GPU per second, 3.9ms, a 0.4% tax. Over five years that is 148,570TB of writes. With 1152GB and a conservative 1M P/E (10× endurance from shorter retention, not the 50× some device papers claim), CSI offers 1,125,000 TBW. Read energy is 8pJ/bit, 2.68× HBM3e; TDP rises 1.31×/1.23× for CSI/CLI, yet tokens/J still reach 1.93×/1.66×. One stack occupies 149mm² versus 121mm² for HBM3e (1.23× area, 6.50× density).
This is not another "spill weights to SSD" offload. Commodity SSD bandwidth is an order of magnitude too low for datacenter decode. In-flash designs such as AiF, Lincoln, and Cambricon-LLM put compute inside NAND for the edge: hundreds of GB/s, a few TFLOPS, and no KV in Flash. FlashAccel takes the opposite cut: keep the GPU doing the math, and use Flash only to add capacity and bandwidth.
What transfers to serving software is the layout and the API: stripe weights in execution order, isolate Flash blocks per request, replicate straggler slices into HBM, and write back only when a full set of megablocks is ready. The hardware is still a paper stack. If the goal is to hold a larger model inside one NVLink domain with fewer GPUs, CSI is a coherent story in simulation.
Every number comes from an extended LLMCompass model. There is no silicon and no production serving stack on real HBF. Timing reuses XL-Flash tR=4μs and tProg=75μs, kept unchanged even though each plane is modeled 4× smaller. Prefetch implies kernel, page-table, and scheduler changes that the paper sketches rather than costs out.
The endurance claim holds only if KV write volume stays near the 988MB/s derived from a DeepSeek cluster report and an agent input/output ratio, and if retention can be shortened enough to justify 10× more P/E cycles. MHA, longer contexts, or higher QPS change that ledger. CLI already hits a bandwidth wall at 50ms. Area at 96 wordline layers is 23% above HBM; 300-layer density is a projection, not this design.
The paper is explicit that NAND write bandwidth and endurance remain far below DRAM. KV in HBF is viable here because GQA and MLA already cut the write volume.