Training a 441M-Param Text-to-Image Diffusion Model From Scratch on a Single Local GPU
ostrisai · x · 2026-09-08
ostrisai spent a weekend training a purely experimental image diffusion model from scratch on a single local GPU; two days in, the unusual architecture is working surprisingly well for its size.
- No text encoder: Qwen3 tokenizer token embeddings are fed straight in alongside image tokens (time conditioned as 0), self-attention only
- Pixel space, no projection layers: patch 16 goes directly into the transformer; hidden size 768 blocks operate on raw pixels, not projected image embeddings
- A shuffler fixes visible patch seams: alternating unpatching, mirror padding 8 on all sides, re-patching so cross seams land at patch centers, running through a block, then depatch/crop/repatch every other block — cleaned up immediately, only possible without projections
- Tiny and fast: 441M total params (116M is the token embedding table), vs SD 1.5's 983M (TE + UNet); no VAE, no text encoder
It's unclear if it will produce coherent images, but it's already grasping basic prompt concepts and vaguely resembles AITK samples with the same prompts. Training continues.
More from Research
- FactoSR: RL with Factorized 4D Objectives Boosts VLM Spatial Reasoning — HKUST-GZ2 · 2026-09-08
- Stanford paper: general coding agents beat hand-built data agents by up to 37 points — CShorten30 · 2026-09-08
- Dhenu Vision 1.0 hits 94.4% precision, beating every frontier model tested — DevDminGod · 2026-09-08
- NBER paper: automation erodes the meaning of work before it eliminates jobs — ArtificialOther · 2026-09-08
- 417k-param RNN generates all 6,573 frames of Bad Apple from a single initial state — SEBADA321 · 2026-09-08
- Can current LLM architecture reach AGI? An engineer lays out his doubts — mostly_deterministic · 2026-09-08