Training a 1B-Parameter LLM from Scratch for Just $200: Full Details Open-Sourced

SevereTilt · reddit · 2026-08-11

A developer shared the complete process of training a 1.1B parameter LLM (Gemmeh) from scratch, with a total cost of only about $200.

Model & Data

• The architecture is based on Gemma3, with a context length of 4096 and a reduced 32k vocabulary.
• Pre-training data used fineweb-edu, while LoRA instruction tuning utilized the OpenHermes dataset.

Training Process

• Renting compute from vast.ai, the author initially tested three scales (185M, 500M, and 1.1B) on 2B tokens.
• The final 1.1B model was fully pre-trained on 20B tokens (taking about 130 hours on an H100), achieving a final validation perplexity of 10.93.
• The author logged generations for fixed prompts via W&B, clearly demonstrating the model's evolution from outputting gibberish to generating coherent text.

The project has been open-sourced, including the base model, instruction-tuned model, and GGUF quantized versions.

Original post →

More from Infra

• Self-Hosted Coding Agent in MicroVM Sandboxes with Local Inference and iOS App — tom_doerr · 2026-08-11
• SanDisk CEO Says Mid-80s Gross Margin Is a Fair Return for Storage Products — Beth_Kindig · 2026-08-11
• Jensen Huang says Nvidia is making AI compute an investable asset class — Polymarket · 2026-08-11
• Apple Approves tinygrad eGPU Driver: Macs Can Finally Use AMD/NVIDIA GPUs — ns123abc · 2026-08-11
• Tech Giants' Compute Budgets Eclipse US Federal Spending—We Live in Cyberpunk Now — tszzl · 2026-08-11
• ZEISS Creates the Smoothest Mirrors Ever: Hardcore Specs of EUV Lithography — DynamicWebPaige · 2026-08-11

Infra channel →