Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
Amanda Bertsch, Luca Soldaini, Matthew R. Gormley, Graham Neubig, Hannaneh Hajishirzi, Kyle Lo, Dirk Groeneveld
COLM 2026
cs.CL
2026-08-11
Across 26 identical-data 7B models, four common architectural choices compound to cut long-context HELMET by up to 47%, and no short-context metric predicts the gap.
Long context is now expected, but extending a pretrained model to 64K and beyond is unreliable. Llama 3 extends cleanly; Olmo struggles. Whether that gap comes from architecture or from Llama 3's undisclosed pretraining data has stayed open, and so has a practical question: during pretraining, can you tell early whether a model will extend well? The answer here is no for every standard metric, and the deciding factor turns out to be a set of architectural choices everyone treats as minor.
The core is a tightly controlled pretraining sweep, released as OlmPool: 26 models of 7-8B parameters. Each is pretrained for 140B tokens (the Chinchilla-optimal amount), then context-extended with 10B tokens at 64K on the Longmino mix, with RoPE theta adjusted. Data selection and ordering, optimizer, learning-rate schedule, and tokenizer are identical across all 26. Only four axes vary, and only at values Llama 2/3, Qwen 3, or Olmo 3 actually ship:
These four were chosen because they genuinely differ across recent major releases and each touches attention or context length directly. Holding everything else fixed is what lets the study peel architecture away from data.
One coupling is easy to miss: QK-norm exists for training stability. The authors confirm that runs with post-sublayer-norm and no QK-norm diverge before 140B tokens, so "removing QK-norm" in practice means removing it and switching back to prenorm. The tradeoff is real, not a free win.
Across the 26 models, HELMET at 32K runs from 29.9 to 56.4 and RULER at 32K from 44.7 to 67.7. Same data, and architecture alone opens a 26.5-point, roughly 47% gap.
Compounding is the whole story. Each feature is mild alone: 4K versus 8K pretraining costs 1-2 points; sliding window alone costs 1.1; GQA scales with KV-head count, where more heads than Llama 3's 8 helps and fewer hurts; QK-norm is the largest single lever, worth +6 HELMET when removed from Olmo and -3.8 when added to Llama 3. Stacked, they go super-linear: sliding window plus GQA averages -9, against -1.1 alone, and the single worst configuration is GQA plus sliding windows plus headwise QK-norm.
The best predictor is the crudest: count how many of the four features a model carries. That one number hits R²=0.67 in-sample and 0.61 leave-one-out, beating a linear regression over the four axes. Carrying all four is not always worse than three, because some overlap: a 4K sliding window during pretraining looks much like pretraining at 4K.
Standard pretraining metrics are nearly blind to all of it:
| Metric | R² vs. 32K HELMET |
| Pretraining loss | 0.29 |
| Extension-phase loss | 0.06 |
| WikiText perplexity (best intrinsic) | 0.39 |
| Mean of 16 short-context benchmarks | 0.17 |
| Pre-extension HELMET 8K | 0.32 |
| Four-feature count | 0.67 (LOO 0.61) |
Even the closest proxy, pre-extension HELMET at 8K, reaches only 0.32 and misses the double-digit swings. Llama 3's ease of extension is an architectural fact, not a data dividend: its architecture ranks near the top of the design space, and several OlmPool configurations beat it.
Mechanistically, the authors pull apart the attention distributions. Models without QK-norm spontaneously form strong attention sinks (dumping attention mass on the first few tokens) with higher attention entropy, and that sink behavior correlates with better long-context scores (R²=0.38). The sink appears to be the default strategy a QK-norm-less transformer learns to absorb "excess" attention, and QK-norm suppresses exactly that. Sliding windows weaken sink behavior on the full-attention layers.
If long context matters to your model, screen architectures before pretraining: by extension time the result is largely settled, and nothing on the pretraining dashboard will warn you. The cheapest screen is literally counting four features. The sharpest finding is QK-norm: a feature added for stability is also the single biggest tax on extensibility, and you cannot just remove it without also reworking norm ordering. GQA is the next lever: more KV heads buy long-context quality at a known inference cost.
Scale is the main caveat. Only 7B models, only 140B pretraining tokens, against trillions for production models. Extending the worst architecture to 50B tokens still leaves it behind Llama, but the authors cannot rule out the gap closing at extreme scale. The four-feature counter is fit on 26 models in a narrow design space and is untested outside it. The retrieval-head analysis is inconclusive, with low scores across all models that may just mean the models are too early in training to identify retrieval heads reliably. The sweep cost about 170,000 H100-hours and 42.6 tons of CO₂-equivalent, so this kind of controlled study is expensive to reproduce. And the mechanistic line, that sinks compensate for excess attention in models without QK-norm, is correlational (sink versus HELMET R²=0.38), not causal.