Inference-First Architecture Sparks Debate: FP4 KV Cache and Pure CSA2 Compression in Focus

On September 11, tech blogger stochasticchasm posted a series of detailed threads dissecting a newly published model architecture blog, reaching a core conclusion: the architecture changes almost solely target inference efficiency—a textbook case of "inference-informed design"—with cheap prefill, an extremely small KV cache, and friendliness to prefill-decode disaggregated (PD disagg) deployment, achieving decent utilization under large-batch prefill.

Confirmed

• The KV cache is stored directly in FP4 precision, with the blogger observing no quality degradation; combined with CSA compression (CSA/HCA dual-frequency alternation in v4, changed to pure CSA2 in the new design), the model barely uses any KV cache space—inference engineers will have plenty to play with.
• The decoder does not attend to encoder KV across many layers; instead, it periodically reads the encoder's final-layer state. The blogger corrected his initial assumption, arguing this makes sense: the last-layer state carries the richest information, stays close to the original Transformer design, and is not standard cross-attention.
• The sparse attention version no longer requires a dense-attention warmup phase. The blogger sees this as a good sign: lower prefill FLOPs can support long-context pretraining and enable introducing agent-like trajectory data earlier; the team also already has mature recipes and batch invariant kernels.
• The new architecture introduces the engram mechanism and simplifies mHC—changes the blogger considers expected, with mega-mHC worth deeper study.

Not yet confirmed

• In one 20-layer model, only the first two layers use sliding-window attention (SWA): the blogger is puzzled that only 2 of 20 layers use SWA, as the prefill FLOPs savings seem limited; while less global attention is indeed cheaper, there may be a story behind it—no conclusion yet.
• Netizens found the CSA/HCA dual-compression-frequency alternation in v4 "odd," arguing that without explanation it's less reasonable than a pure CSA2 scheme—this is a subjective community assessment.

Why it matters

• KV cache memory footprint is one of the biggest bottlenecks in inference deployment; FP4 output with no degradation plus compression means a single request still needs to hold the KV cache in HBM/memory to complete inference, but memory pressure across requests drops sharply, potentially reshaping the cost structure of inference services.
• Sparse attention shedding dense warmup, combined with low prefill compute, hints at a possible shift in the long-context pretraining paradigm; at roughly 500K ultra-long context, KV cache trade-offs remain a problem that must be solved within a single request.

2026-09-11 ~ 2026-09-11 · 11 related posts

• [source] New attention design called inference-informed: cheap prefill, small KV cache — stochasticchasm · 2026-09-11
• Inference efficiency is now the sole driver of architecture changes, observer notes — stochasticchasm · 2026-09-11
• Pure CSA2 beats v4's alternating CSA/HCA compression, argues arch observer — stochasticchasm · 2026-09-11
• [source] Straight-up FP4 KV cache with no degradation: inference engineers take note — stochasticchasm · 2026-09-11
• FP4 KV cache with no degradation sparks debate on aggressive inference compression — stochasticchasm · 2026-09-11
• New model adds engram and simplified mHC; community digs into KV cache tradeoffs — stochasticchasm · 2026-09-11
• [source] Sparse attention without dense warmup signals midtraining absorption, analyst argues — stochasticchasm · 2026-09-11
• Sparse Attention Without Dense Warmup Signals Long-Context Training Breakthrough, Analyst Argues — stochasticchasm · 2026-09-11
• Decoder doesn't cross-attend to encoder KVs — just periodically reads final encoder states — stochasticchasm · 2026-09-11
• Architecture deep-dive: decoder periodically reads final encoder states instead of cross-attention — stochasticchasm · 2026-09-11
• Why Only Two Layers Use Sliding Window Attention? Architecture Detail Sparks Debate — stochasticchasm · 2026-09-11