COLM papers reveal common Transformer designs hurt long-context scaling
Multiple COLM papers challenge default Transformer design choices. One study comparing 26 7B models found that compute-saving conventions like qk-norm, GQA, and SWA significantly hurt long-context scaling, revealing hidden costs of standard architectural decisions.
2026-08-17 ~ 2026-08-18 · 4 related posts
- Study: Minor architecture choices cripple long context capabilities — rohanpaul_ai · 2026-08-17
- Paper tests 26 comparable 7B models: cheaper architectures hurt long context — eyishazyer · 2026-08-17
- Three COLM Papers Reexamine Transformer Defaults: 47% Long-Context Swings and 99% Gradient Loss — 量子位 · 2026-08-17
- Studies Reveal Transformer Architecture Choices Hinder Long Context Extension — yoavartzi · 2026-08-18