Dev squeezes a 7M-param byte-level LLM entirely into an RTX 4090's on-chip memory
dejanseo · x · 2026-10-24
Developer dejanseo built a 7M-parameter language model designed to fit entirely in an RTX 4090's on-chip memory. Key details:
- Byte-level input with a 256-token vocab, no tokenizer needed
- Linear attention with no growing KV cache; tiny d256, 8-layer, 4-head architecture
- Custom Pallas kernels for the recurrence, hitting 643K tokens/sec — 3.6x faster training than baseline
Training dashboards show val loss around 1.60 bits/byte and still improving. He's now testing whether a model this small can hold a conversation — a proof of concept for ultra-low-resource always-on chat models.
More from Infra
- Lafa: 8B flow language model hits 2.54x AR throughput with under 30B training tokens — LucaAmb · 2026-10-24
- King Slide, the 'king of AI server slide rails,' is up 2,385% in five years — jwt0625 · 2026-10-24
- Semis roar back into VC focus as CASPA panel calls for 1000x faster, greener AI chips — hardimanjames · 2026-10-24
- Same model, 2.3x cost gap: cache hit rates make per-token pricing a poor guide — nicolechirps · 2026-10-24
- Local LLM storage pain: an 8TB Gen5 SSD now costs $2,719.99 — TheZachMueller · 2026-10-24
- DeepSeek nears ~$15B round at $74B pre-money valuation, double its original target — 2C_ornot2C · 2026-10-24