NAVER paper: word-level unigram cross-entropy tracks zero-shot benchmark accuracy
naver-hyperclovax · hf · 2026-10-28
A NAVER Cloud & KAIST paper, "The Lexical Imprint of Pre-training Data," asks how much of zero-shot benchmark accuracy is captured by word-level alignment between the pre-training corpus and the benchmark. The authors propose word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free lexical alignment measure, and show it consistently tracks zero-shot accuracy across benchmarks, document selection, and prompt routing. Using controlled from-scratch pre-training runs that vary corpus identity while holding the recipe fixed, they trace how this lexical imprint scales with corpus-benchmark alignment.
More from Research
- figwig benchmark: random 2k-window reads up to ~40x faster than bigwig for ML training — jmschreiber91 · 2026-10-28
- Could cross-stream mixing only be needed in the first half of the network? A look at mHC — teortaxesTex · 2026-10-28
- DeepSeek v4 insider: mHC residual streams may converge — why iHC is the safer pick — teortaxesTex · 2026-10-28
- Qwenputer: a 32-bit RISC-V CPU compiled into Qwen3.5's weights — LatentSpacer · 2026-10-28
- Meta's Loop Scaling Laws: looped MoE matches 2x larger models on reasoning — yacinelearning · 2026-10-28
- Anthropic trial: 77% of analysts echoed their agent's arguments, only 15% credited it — paulnovosad · 2026-10-28