Dev Trains Continual-Learning Model From Scratch on 8GB VRAM Laptop, Now at 530M Params
Another__one · reddit · 2026-09-21
A developer shared the mini-AGI project on Reddit: a continually learning model trained from scratch on an 8GB VRAM laptop using a batch-1 stream of data, now at 530M parameters and still growing.
Two key ideas:
- Dynamic MoE: experts are added and pruned during training, with only a small subset loaded at any time, so parameter count is bounded only by disk space;
- Batch-1 stream training: no need to store large randomized batches and gradients, drastically cutting VRAM needs.
The motivation is dissatisfaction with being unable to train 1B+ models on consumer hardware; the author wants full control over what the model sees. The model reads the 7.8B-character corpus as one continuous stream in 32K-character passages, "just like you or I would," and is still training. The author says the project was built by brainstorming with Claude and would have been impossible otherwise. It's open source — you can git clone and run it — and includes a promising scaling curve.
More from Research
- NOHARM: an 1,100-task medical safety benchmark with an RCT of physician-AI teaming — davidjhwu · 2026-09-21
- Unified 3D generative model for synthesizable structure-based drug design sparks synthesizability debate — _onionesque · 2026-09-21
- AI safety oncology work wins Best Poster at ANCO, but physician consortium sign-ups hit zero — davidjhwu · 2026-09-21
- Active Learning and Causal AI Challenge the Claim That Only Humans Can Run the Theory-Experiment Loop — maier_ak · 2026-09-21
- Counter to 'Theory Is All You Need': RL and Causal AI Already Design Experiments — maier_ak · 2026-09-21
- New Paper Achieves Sub-Second Interactive Diffusion on Consumer GPUs — chaumian · 2026-09-21