kalomaze runs matched OOD finetuning test: big MoE vs dense models, 1:1 comparable

kalomaze · x · 2026-11-05

kalomaze threw an out-of-distribution finetuning task (highly structured spatial symbolic language with audio conditioning) at big MoE vs dense models. For a fair comparison, both used the same custom tokenizer — making cross-entropy 1:1 comparable — with matched data, batch size, learning rate and LoRA rank (16). Results shared in a chart; ongoing discussion in the thread.

Original post →

kalomaze runs matched OOD finetuning test: big MoE vs dense models, 1:1 comparable

More from Models