Discussion: Over-determined projections may outperform embedding tables in representation learning
kalomaze · x · 2026-08-19
A technical discussion suggests that projecting from low (e.g., 8) to high (e.g., 4096) dimensions is algebraically over-determined, which might be fundamentally better for learning representations than arbitrary per-token values. This challenges the necessity of the 'embedding table' concept, proposing linear projections of raw data bits similar to ViT or audio projections.
Related event: Debate: Direct Projections May Replace Embedding Tables(3 posts)→
More from Research
- Classic Paper: The Surprising Creativity of Digital Evolution — Ghost_Pilot_MD · 2026-08-19
- HarmProfile Benchmark: Harmfulness and Diversity Rise with Model Capability — Zhouyuan Ma · 2026-08-19
- Why Removing the Vision Encoder Can Be Better — From an Infra Perspective — liuziwei7 · 2026-08-19
- Matmul Optimization Bottleneck: Data Movement, Not Multiplications — yaroslavvb · 2026-08-19
- Hot take: "Skill optimization" is just repackaged prompt optimization — IanArawjo · 2026-08-19
- NBER study: Pangram detector achieves near-zero error rates — soumitrashukla9 · 2026-08-19