Context distillation fixes: relevance gating and 2x strength cut value spillover to 7%
Sauers_ · x · 2026-11-15
- Experiment on Qwen3-8B: train a student without a written value prompt (animal welfare) to match a teacher that has it, tested on 68 hard prompts where the value could apply but animals are never mentioned.
- Plain context distillation spills the value onto unrelated prompts in 33% of replies.
- Two fixes beat it: distill only on prompts where the value applies (copy the plain model elsewhere), and push the target 2× past the prompted model, like classifier-free guidance. Result: +13% over plain distillation, spillover down to 7%. Single seed so far; replications running.
More from Research
- Epoch AI: post-training enhancements yield gains equal to 5-20x compute at under 1% the cost — CFGeek · 2026-11-15
- Qwen3 at 2.7 bits/weight with Leech lattice quantization beats IQ2_S on quality, loses on speed — Fuzzy-Layer9967 · 2026-11-15
- Hybrid Decision Models: Only 30% of Queries Need Reasoning, Latency Cut 72.5% — hbouammar · 2026-11-15
- Steering with Natural-Scale Gaussian Noise in Random 4-Dim Space Sparks Follow-up Research — Sauers_ · 2026-11-15
- What 'Compute Multipliers' Mean: The Under-discussed Algorithmic Efficiency Metric in AI — CFGeek · 2026-11-15
- Researchers push back on Dwarkesh: defining AGI metrics is the hard part — dhadfieldmenell · 2026-11-15