Removing embedding tables: ViT-style projections for tokens
kalomaze · x · 2026-08-19
A technical discussion suggests that models like Gemma 4 12b might be moving away from tokens towards processing original bits/data via linear projections (similar to ViT), potentially eliminating the need for 'embedding tables'. A reply criticizes papers like BLT for being noncommittal, resembling 'entropy patched heuristics' rather than a serious fundamental shift to byte-level tokenization.
Related event: Debate: Direct Projections May Replace Embedding Tables(3 posts)→
More from Research
- AI Agent Closes Design-to-Test Loop in Drug Discovery, Generating 3 Sub-nanomolar Binders — AllThingsApx · 2026-08-19
- Protein design harness coming in Sept: Claude/GPT propose antibody binders verified by SPR — chaitjo · 2026-08-19
- ByteDance Releases StartupBench: Top Models Complete Only 30% of Tasks — ByteDance-Seed · 2026-08-19
- Agentic ESOpt: Fine-Tuning Long-Horizon Agents with Minimal GPU Requirements — NationalUniversityofSingapore · 2026-08-19
- RUPA: Improving Agent Reliability via Relational Uncertainty Propagation — ICIP · 2026-08-19
- Dynamic Multi-Byte Prediction Accelerates Hierarchical Language Models — ohiostate · 2026-08-19