Hosting a 3T Model on Cerebras Requires 68 Chips and 1.5MW
zephyr_z9 · x · 2026-08-14
Addressing the issue of attention dominating at longer contexts, the author discusses scaling attention and FFN independently. They speculate that future systems will aggressively use context compaction systems and add more Blackwell chips on the attention side.
Furthermore, a rough cost estimate is provided for deploying a 3-trillion parameter model (like Sol) on Cerebras architecture. Since a single WSE chip has about 44GB of SRAM, hosting this model would require around 68 chips. This would cost nearly $200M and consume around 1.5MW of power for a single copy.
Related event: Cerebras and Blackwell Launch Commercial AFD Architecture(2 posts)→
More from Infra
- Three CoreWeave co-founders have sold $3.3B in shares since IPO amid credit worries — SumitGup · 2026-10-16
- Cherry pi on a Raspberry Pi: the tiny-model rumor developers are watching — sull · 2026-10-16
- CoreWeave's debt costs falling: DDTL spreads tightened from S+9.62% to S+2.25% — BenBajarin · 2026-10-16
- SoftBank lines up US IPOs for SB Energy and AI robotics spinoff Roze, valuations top $50B — nordicinst · 2026-10-16
- TRL reproduces Modal's mixed-precision async RL: plain GRPO collapses, SC+MIS holds 0.72 — QGallouedec · 2026-10-16
- Hugging Face scales Indic pretraining with 2.8T tokens and an open data pipeline — Hugging Face Blog · 2026-10-16