Zhipu chief scientist reveals GLM-5.5 training pipeline: 1T web pages, six data engines
algo_diver · x · 2026-11-05
Zhipu chief scientist Jie Tang revealed the pretraining pipeline behind the upcoming GLM-5.5 (32B/122B/256B) in a Tsinghua University lecture. Instead of public datasets, it draws on six internal engines: 1T web pages crawled at 10B pages/day with entropy-scored link discovery and PB-scale template clustering; 2B scientific PDFs parsed via GLM-OCR (4x faster with speculative decoding); 50M books with layout reordering and garbled-text filtering; 500M code repos turned into executable tasks with real tests and issue descriptions; 10B videos for long-sequence world knowledge; and 300M hours of audio rewritten via ASR into trillions of tokens. Each source poses its own scaling problems: crawl, filter, dedup, quality — plus a data scaling law L(N,D) for effective tokens.
More from Models
- StepFun's Liu Licheng teases new work, community hopes Step 5 skips the Preview label — teortaxesTex · 2026-11-05
- HF dev questions Thinking Machines' SWE-2 Flash weight-sharing program over simple HF gating — QGallouedec · 2026-11-05
- RAG 101: One Untuned Setting Drops Answer Accuracy from 100% to 33% — dima806_dima · 2026-11-05
- Dev Praises DeepSeek's Eval Harness: 'They Really Cooked' — mariofilhoml · 2026-11-05
- Lebrel launches uncensored frontier-model inference with signed, Solana-verified receipts — HeyAmit_ · 2026-11-05
- IOTA Decide 4B open-weight decision model hits ~11 decisions/sec on a Mac mini — const_reborn · 2026-11-05