Developer live-runs continual pretraining of a Luganda LLM on Colab, 41% done with loss dropping fast
weskambale · x · 2026-10-16
Developer weskambale is publicly live-training a small model (pearl-chat-v2) via continual pretraining on Luganda, reaching 41% after 8 hours on Colab.
- Phased setup: Phase 1 is embedding-alignment warmup (300 steps, 84M trainable params); Phase 2 is continual pretraining (2,600 steps × 8 accum, 508M trainable params)
- Loss falls steadily: valppl drops from 815 after warmup to 34 at 41%, with valbpb 1.07
- The model is mostly absorbing Luganda prose, grammar, and vocabulary; Phase 3 will add instruction tuning for multi-turn conversation and reasoning
A transparent, reproducible recipe for low-resource language adaptation on a single Colab instance.
More from Infra
- Nscale AI datacenter plagued by coolant leaks, transformer fire, 10 safety violations — saranormous · 2026-10-16
- Are 22 GB modded RTX 2080 Ti cards a viable budget local LLM option? — Dave86ch · 2026-10-16
- Intel's neural-compressor: SOTA low-bit LLM quantization with FP4/NVFP4 support — HaihaoShen · 2026-10-16
- Tencent squeezes 770B-parameter Hunyuan Hy4 Preview into 214 GiB at ~2.38 bits per weight — coolmum18 · 2026-10-16
- TokenSpeed MLA 2 hits NVIDIA Vera Rubin: up to 3.44x faster decode vs B200 — zhyncs42 · 2026-10-16
- US vs Chinese hyperscalers: Americans fund and schmooze, Chinese demand your BoM and cap you at 7% margin — jwt0625 · 2026-10-16