Dev trains a 2B Apache-licensed model on 24GB VRAM using OLMo embeddings and a 1B Engram table

NineThreeTilNow · reddit · 2026-09-17

A developer with 20 years in ML (briefly an Anthropic red-teamer on the original Opus 4) shared an update on r/LocalLLaMA:

• Dropped his earlier 9B plan over license issues (Llama licensing blocked Apache 2.0) and rebuilt on the OLMo tokenizer with dmodel=2048, creating a 2B dense base plus a 1B-parameter Engram table (n-gram vocab pushes it to 50% of size, above DeepSeek's suggested 10-20%)
• Uses 40 SWA/Global blocks with attention-based residual streams in the Kimi style; training data is run through the 7B OLMo model to produce 32-token probability distributions instead of one-hot cross-entropy
• Borrowed down-projected (5k→2048) embeddings and LMhead, preserving 65% of the big model's embedding information per spectrum analysis
• Surprisingly coherent output after only 15M tokens, without the classic early-training token-repetition loop
• Claims 100% trainable on 24GB VRAM; code, model and data to be released on GitHub/HF

Original post →

More from Infra

• Cohere's CUDA Megakernel Serving Hits 292 tok/s at Batch 1 on a 30B Model — dl_weekly · 2026-09-17
• dotAI talk on llama.cpp speculative decoding: MTP, dflash, dspark — ngxson · 2026-09-17
• First M5 Ultra benchmarks show 50 tok/s running Qwen3 27B q4 locally — Ashefromapex · 2026-09-17
• 6 teams, one opaque LLM bill: a postmortem on tagging, proxies and gateway budget caps — Fit_Program7076 · 2026-09-17
• Graphsignal open-sources a GPU profiler designed for AI agents, not humans reading traces — l0g1cs · 2026-09-17
• How a decentralized inference network catches cheating GPU hosts: spot checks, reputation, open weights — autoimago · 2026-09-17

Infra channel →