Prediction: an open model rated for 10M context within 3 months
teortaxesTex · x · 2026-11-03
AI researcher teortaxesTex predicts a workable open model rated for 10M-token context will arrive within 3 months.
The attached image gives his revised take on DeepSeek V4.1's long-context architecture: don't bolt ZHA onto V4.1 — redesign the indexer (as DeepSeek does every generation):
- Swap ReLU scoring for linear logits.
- Build ZHA trees over the 4 shared indexer keys, distilled from the flat indexer.
- Put the first one in the encoder, where the quadratic term lives.
- Keep fixed-size selection budgets per level so kernels stay regular.
The cost is a less expressive indexer, in exchange for removing the last term growing linearly per query — roughly flat attention reads per step, but storage stays linear in context, and no training data exists yet to make 100M-token contexts useful.
More from Research
- LocalFold update brings BioEmu2, Mol-1, AF3x crosslinks to browser-based protein folding — CatAstro_Piyush · 2026-11-03
- kalomaze conjectures deep sequence models could mimic restricted training families like rank-1 attention LoRAs — kalomaze · 2026-11-03
- Zyphra's ZHA brings subquadratic attention via hierarchical tree-structured indexer — yacineMTB · 2026-11-03
- kalomaze: In-context learning is an incidental byproduct of next-token prediction, not deliberate design — kalomaze · 2026-11-03
- RSNA 2026 lab walks through fine-tuning a VLM for radiology end to end — PRouzrokh · 2026-11-03
- RSNA 2026 lab walks through fine-tuning a VLM for radiology end to end — PRouzrokh · 2026-11-03