Training combines chunk-level KL loss with document-level contrastive loss on pooled scores
antoine_chaffin · x · 2026-11-03
The thread explains the training objective: a chunk-level KL objective combined with a document-level contrastive loss using max-pooled chunk scores. This teaches the 9B model both which chunks contain the answer or supporting evidence, and which document is relevant among distractors.
More from Research
- Victor Taelin floats labs replacing backprop with models editing their own weights at scale — arthurcolle · 2026-11-03
- LLM paper claims last ~15 months vs ~75 years for general science, researcher estimates — patpat_mit · 2026-11-03
- LocalFold update brings BioEmu2, Mol-1, AF3x crosslinks to browser-based protein folding — CatAstro_Piyush · 2026-11-03
- kalomaze conjectures deep sequence models could mimic restricted training families like rank-1 attention LoRAs — kalomaze · 2026-11-03
- Zyphra's ZHA brings subquadratic attention via hierarchical tree-structured indexer — yacineMTB · 2026-11-03
- kalomaze: In-context learning is an incidental byproduct of next-token prediction, not deliberate design — kalomaze · 2026-11-03