Unverified tip: V4.1's KV-compressing encoder allegedly copied from Astra; prior art abounds
teortaxesTex · x · 2026-10-07
teortaxesTex discusses architecture homogenization: Western open labs stick to boring GQA+SWA designs while Chinese labs flex custom architectures, since discovering scalable designs takes compute and OAI/Ant can punish copiers like Thinky. He relays an unverified DM tip that the "reintroduction of the encoder for compressing KV" in V4.1 was copied from Astra, though he finds it dubious given ample prior art, and credits Zyphra and Korea's Motif as genuinely original.
More from Models
- HuatuoGPT-3: 27B open medical LLM hits 70.1 on HealthBench, beating GPT-6 Astra — CUHKSZ · 2026-10-07
- Dev Complains Overnight Long-Running Tasks Keep Hitting Usage Limits — willdepue · 2026-10-07
- Gary Marcus on whether frontier LLMs can solve open math problems without symbolic harnesses — GaryMarcus · 2026-10-07
- From botching 9.9 vs 9.11 to tackling the hardest math problems in two years — Yuchenj_UW · 2026-10-07
- OpenAI claims 372 unsolved problems cracked, averaging about 3 hours each — i_dg23 · 2026-10-07
- Benchmark: OpenAI Decisions API costs 2x more, 5-10% worse than Jev — xeophon · 2026-10-07