Zhipu chief scientist reveals GLM pretraining data roadmap: GLM-5 at 28.5T tokens, 100T next
rickasaurus · x · 2026-11-06
Zhipu chief scientist Jie Tang shared GLM's per-generation pretraining data progression: GLM-130B (2022) 0.4T tokens, GLM-4 (2024) 15T across 26 languages, GLM-4.5 (2025) 23T with quality buckets (>3.2 epochs) and repo code/agent mid-training, GLM-5 (2026) 28.5T with +28% unique code and 200K context, targeting 100T cleaned native public-text tokens next (excluding synthetic data).
Since open-source web parsers cap high-quality public text below 30T tokens, Zhipu built an internal HTML/WARC extractor, haruka-parser, recovering 50% more total and 150% more high-quality tokens. Training sets grow 2.4x/year; small models select the best 10% of data, trillion-scale models maximize high-quality effective tokens.
Related event: Zhipu's Tang Jie Reveals GLM-5.5 Training Pipeline and Data Roadmap(4 posts)→
More from Models
- Artificial Analysis publishes full Inkling 2 Small results on Intelligence Index v5.0.1 — ArtificialAnlys · 2026-11-06
- Thinking Machines' Inkling 2 Small scores 31 on AA Intelligence Index, eyeing top open-weights spot outside China — ArtificialAnlys · 2026-11-06
- NVIDIA ships NIM deployment recipes for Mistral Large 4 on Blackwell FP8/NVFP4 — NVIDIAAI · 2026-11-06
- Inkling 2 Small launches on Tinker, benchmarked for post-trainability — simonguozirui · 2026-11-06
- Sommeliers replaced before truck drivers: Inkling 2 IDs wines in 19/20 trials — simonguozirui · 2026-11-06
- DeepSeek-V4.1-Flash tops coding benchmarks: 28.5% on FrontierCode v2.0, beats Kimi K3 — soumithchintala · 2026-11-06