Q Labs: LLMs are depth-bottlenecked, loss keeps improving to 128 layers
rickasaurus · x · 2026-09-24
Q Labs Research argues depth stopped scaling: frontier models still sit near 100 layers, same as GPT-3 six years ago. Their language models keep improving to 128 layers on just 1B tokens, past frontier depths with no saturation. They reproduced contrastive RL results showing new capabilities emerging up to 256 layers (and reportedly 1,024). Scaling computational depth yields compute-efficiency gains: 1.6x at 10^20 FLOPs, projected 3.1x at 10^26. They claim current architectures cap expressive power and depth is the next scaling frontier.
Related event: Q Labs: Computational Depth Is the Missing Scaling Axis for LLMs(4 posts)→
More from Models
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12
- Gemini re-renders images as text when vision is removed, dropping only 6% WSR — zacharynado · 2026-11-12
- Bare-harness coding benchmark: Fable 5.2 best overall, Gemini 4 Carbon best value — Semantic_meaning · 2026-11-12
- Google's Physical-Engineering's Last Exam: all 14 frontier models score below 50% — dl_weekly · 2026-11-12
- Reddit's 2026 LLM hall of fame: Opus 5.5 still unbeaten after 9 rival releases — G0dZylla · 2026-11-12
- Commit leak reveals unreleased Gemini 4 variants: Argon, Hydrogen and Helix (unconfirmed) — BlackHC · 2026-11-11