Indie dev offers to train a fully open 9.4B dense model, tuned for a single GPU

NineThreeTilNow · reddit · 2026-09-13

A LocalLlama user is offering to train and open-source a 9.4B parameter dense model, asking the community what to target:

• Architecture: 1/2/3-layer Engram table injection, Moonshot-style AtRes modeling, 3:1 RoPE/NoPE layering; starts from the Llama 3 tokenizer and LM head, with logit-level data extracted from a Llama 3 teacher model.
• He produced all training data on a single 4090 plus rented hardware; the code is deeply optimized for one RTX 6000 Pro card, aiming to beat existing 9B models and add open-standard 'thinking'.
• Qwen's recent release showed a single Engram table captures most of the benefit, so he cut the design from two tables to one.
• Code and data are already in public repos; along the way he found and reported vLLM prompt-loading code that could be 10-100x faster.

Original post →

More from Models

• Leak: Google Reportedly Has Another 'Hella Chonky' Model After Carbon — iruletheworldmo · 2026-10-30
• Users report across-the-board GPT quality drop: weaker writing and broken instruction following — Kyrannio · 2026-10-30
• Leak claims Gemini Argon hits Pro trial from November 2nd (unverified) — Emotional-Company847 · 2026-10-30
• OpenAI retiring waves of legacy models: GPT-5 family dies Dec 11, GPT-4.1 by Jan 2027 — ThePeterMick · 2026-10-30
• DeepMind CEO Demis Hassabis tells Musk: testing with trusted partners, coming soon — demishassabis · 2026-10-30
• Adobe stands firm on not training on customer data, using licensed stock instead — omooretweets · 2026-10-30

Models channel →