Cartesia research blog argues byte-level hierarchical networks are the next scaling step past tokenizers
SonglinYang4 · x · 2026-10-29
Albert Gu published Cartesia's research blog From Tokens to Bytes I: Scaling Hierarchical Networks (voiceover fully generated by Cartesia's Sonic TTS).
- Core claim: LLMs only partially embrace the bitter lesson — hand-crafted tokenizers block true end-to-end learning from raw data; byte-level language models are foundational for the next scaling step and a litmus test for general learning methods.
- Approach: hierarchical networks that learn directly from characters/bytes, with scaling experiments detailed across a two-part blog series.
- Motivation: in science, robotics and biology where data is scarce and hard to engineer for, learning abstractions from low-level inputs with minimal human priors matters more than ever.
More from Models
- You.com: agentic search failures are behavior problems, fixed in Nemotron 3.5 Lightning — RichardSocher · 2026-10-29
- How I AI agent bench: Grok Bot wins big at 93%, Claude Duet trails at 79% — lennysan · 2026-10-29
- DeepMind slammed for announcing models weeks before availability amid Gemini Xenon chatter — adonis_singh · 2026-10-29
- Alphabet Q3 call: Gemini Carbon frontier model, MCP apps in Search, AI Mode ads expand — lilyraynyc · 2026-10-29
- gpt-6.1-terra runs 4-hour browser session at ~1 action/second on 10% of quota — prd_008 · 2026-10-29
- AA Index revamp right before Gemini Carbon release raises benchmark timing eyebrows — stablequan · 2026-10-29