LLMs pick 7 in 30/30 tries: EMNLP paper traces the bias to SFT training data
radamihalcea · x · 2026-10-17
A MBZUAI/KAIST/Auburn EMNLP 2026 paper separates "Deep biases" (stable preferences surviving scenario reframing, rooted in pretraining) from "Shallow biases" (prompt-dependent ones).
- The accompanying chart shows GPT-6 Astra, Claude Opus 5.5 and Gemini 4 Argon all picking 7 in 30/30 tries on "choose a random number from 1 to 10"
- The culprit is SFT data: Olmo 3 7B had a spread of number choices before SFT, but picked 7 in 26/30 tries after SFT — the SFT corpus is full of examples like "lucky number is 7" and "the magic number is 7"
- Olmo-3-7B-SFT's butterfly preference (Monarch) survives 93% of 30 scenario reframings (Deep bias), while its 42 answer drops to 13% and shifts to 3 (Shallow bias)
- Key finding: only about a quarter of top answers survive reframing, so a single-prompt repetition score can't tell the two bias types apart
Paper, code and dataset are open at deepbias.github.io.
More from Models
- Anthropic internal research model cracks 26-year-old NSA elliptic curve seed mystery in under 30 minutes — soldierofcinema · 2026-10-17
- Runway's internal study: Solaris beats Claude Opus 5 on instruction-following 62% vs 25% — _AustinCalvert_ · 2026-10-17
- GPT-6 Astra nears 100% on Columbia's SRE-Bench reverse-engineering benchmark, others below 60% — banteg · 2026-10-17
- Musk says connecting Grok to your X DMs is the best way to use it — elonmusk · 2026-10-17
- Alexandr Wang: Meta MSL built 'a machine that builds models' in one year — alexandr_wang · 2026-10-17
- Muse Spark Open claimed as #1 US open-source model as observers hail Meta's comeback — pxd · 2026-10-17