Transformer Byte-Level Embeddings Clash with Kernel Factorization
kalomaze · x · 2026-08-19
Discusses embedding tokens at the byte level in Transformers: by concatenating and projecting the final hidden states of byte-toks over spans defined by BPE, one can achieve mixed-bidirectional length and block-causal exact autoregression. However, this 'meme factorization' is fundamentally incompatible with modern PyTorch and most kernels built around the assumed input/output factorization symmetry in causal 1D sequence models.
Related event: Byte-Level Embeddings Enable Hybrid Autoregression in Transformers(2 posts)→
More from Research
- NBER study: Pangram detector achieves near-zero error rates — soumitrashukla9 · 2026-08-19
- Real AI risks lie outside the model: permissions, data, and presentation — bigdata · 2026-08-19
- TraceLab: 4,300 real coding-agent sessions reveal high cache hit rates but quadratic cost growth — CShorten30 · 2026-08-19
- Bayesian view won AI debate but no one noticed — AdaptiveAgents · 2026-08-19
- Discussion: Over-determined projections may outperform embedding tables in representation learning — kalomaze · 2026-08-19
- Claude autonomously designs proteins with 35% success rate in wet-lab tests — ResultBackground2450 · 2026-08-19