Lizard converts pretrained Transformers to subquadratic architectures with near-lossless accuracy
burkov · x · 2026-10-13
Standard attention scales quadratically with context length, while training linear models from scratch is costly and often hurts reasoning and recall. Lizard is a linearization framework that converts pretrained transformer LLMs into subquadratic architectures, combining a learnable gated linear attention component to achieve near-lossless performance recovery and robust length extrapolation beyond training lengths — avoiding expensive retraining.
More from Infra
- Google's EmbeddingGemma 2: 740M on-device model for cross-modal vector search — TheTuringPost · 2026-10-13
- The AI race is shifting to agent infrastructure: search, evals and reliability become the moat — FinanceYF5 · 2026-10-13
- swc minifier can hit exponential memory: 40 modules ballooned past 40GB in Next.js build — Vjeux · 2026-10-13
- Over 1m gaming and workstation chips repurposed for AI as China's Nvidia gap persists to 2030 — zijing_wu · 2026-10-13
- When can local models optimize their own runtime? r/LocalLLaMA debates — Plastic-Stress-6468 · 2026-10-13
- NVIDIA unveils first 800 VDC AI Factory design with 16 MW deployment units — firstadopter · 2026-10-13