Three COLM Papers Reexamine Transformer Defaults: 47% Long-Context Swings and 99% Gradient Loss

量子位 · wechat · 2026-08-17

QbitAI reports that multiple papers at COLM 2026 target rarely-questioned "standard" Transformer designs, with strikingly consistent conclusions: every default choice carries real costs on dimensions existing benchmarks miss.

Cracks in the Foundation (Ai2 et al.): Training 26 7-8B models ("OlmPool", 170k+ GPU hours) from permutations of Llama2/3, Qwen3, Olmo3 settings, HELMET 32K scores span 26.5 points (47% relative). Individual choices matter little, but combining GQA with sliding-window attention drops 9 points—more than the sum of parts. Removing Olmo3's QK-norm gains 6 points yet loses 3.8 on Llama3. Counterintuitively, stronger attention sink correlates with better long-context performance.

Lost in Backpropagation (Cornell): The output projection is an information bottleneck—95%-99% of gradient norm is destroyed passing through it in GPT-2, Pythia, Llama3, OLMo2, Qwen3, with residual signal cosine similarity of only 0.1-0.2 to the original gradient.

When Fewer Layers Break More Chains (Tübingen): Layer pruning looks safe on knowledge benchmarks, but on AIME24, s1.1-7B drops sharply after pruning one layer and nears zero after two; test-time scaling breaks, and neither LoRA nor full fine-tuning fully recovers performance.

Original post →

Three COLM Papers Reexamine Transformer Defaults: 47% Long-Context Swings and 99% Gradient Loss

More from Models