Pearl-Chat: an 8.5M-param Luganda LLM built from scratch in JAX, with 2x tokenizer efficiency over LLaMA-3
weskambale · x · 2026-10-19
Wesley Kambale and Roland Sankara released an arXiv paper on Pearl-Chat, a 8.5M-parameter causal decoder-only transformer built from first principles in JAX and Flax NNX for Luganda, a Bantu language spoken by 10M+ people in East Africa.
Key points:
- Custom byte-level BPE tokenizer (V=8,192) achieves 4.88 chars/token compression and 1.60 tokens/word fertility — roughly 2x better compression than LLaMA-3 (128K) and Gemma (256K) multilingual tokenizers; the benchmark chart shows GPT-4o BPE at 3.20 fertility;
- Implementation uses explicit functional state decomposition, resumable data streaming via Google Grain, and SPMD tensor parallelism;
- Two-stage curriculum: pre-training on 59,501 sequences (parallel, biblical, encyclopedic, journalistic corpora) then instruction fine-tuning on 510 dialogues; validation perplexity converged from 12,959 to 1.20;
- Three-tier diagnostic evaluation: memorization 1.0 BLEU-4/ROUGE-L, in-domain 0.942/0.965, held-out generalization 0.941/0.979.
A nice case study in building a full native stack for low-resource languages instead of relying on generic multilingual models.
More from Research
- From a 2014 MSR bet to Nature: how deep learning conquered weather forecasting — LesterMackey · 2026-10-19
- Amateurs Are Using GPT-6 to Crack Decades-Old Ciphers, Overwhelming Cryptographers — ChuckDBrooks · 2026-10-19
- 27B anti-slop model trained via RLAIF hits #4 on EQ-Bench 4, beats several flagships — l4rz · 2026-10-19
- A literature-review workflow with Claude, Althea MCP, Lacuna and Overleaf — hugo_larochelle · 2026-10-19
- Study finds speech domain explains 4x more variance than model choice in TTS cloning — kastnerkyle · 2026-10-19
- Six ELLIS members win 2026 AI2050 Fellowships from Schmidt Sciences — maksym_andr · 2026-10-19