Depth may be the next scaling axis — if you pick the right residual connections
FinanceYF5 · x · 2026-09-30
Wrapping up a DepthBench thread, the author argues that depth could become the next major scaling axis — provided the right residual connection scheme is used. HC and Full AttnRes have both shown models can gain real performance from depth. The bottleneck, he predicts, now shifts from architecture design to infrastructure and kernel optimization.
Related event: DepthBench: Residual Connection Design Determines Whether Depth Can Scale(4 posts)→
More from Infra
- DeepSeek open-sources Huawei Ascend toolkit with TileLang support, challenging Nvidia's CUDA — kimmonismus · 2026-09-30
- Orbital compute is coming: SpaceX and Tesla target 200GW solar per year, says AI KOL — tetsuoai · 2026-09-30
- On-prem LLM stack: the leaks you miss are OCR parsing and hosted eval judges — lucasbennett_1 · 2026-09-30
- Batching vLLM calls with shared prefixes: a warm-up-then-fan-out KV cache strategy — Theboyscampus · 2026-09-30
- Memory price surge threatens on-device AI hardware: BOM tops $300, device may cost $1,500 — stephen280up · 2026-09-30
- Is local hardware worth it for AI agents? Reddit math: 5-10 years to break even vs cloud — pixpuz · 2026-09-30