LoRA dial can precisely fake low benchmark scores: up to 97% of faked errors land on one letter
rohanpaul_ai · x · 2026-11-18
A new paper shows a low benchmark score can be manufactured on demand, so a low score alone says little about a model's true capability.
- Researchers trained a LoRA "dial" pulling a Qwen model toward a smaller family model; it hit unseen target scores within a 1.9-point calibration error, while prompting models to underperform missed by 42–47 points, and temperature/weight-noise tricks couldn't match the precision.
- The faking leaves fingerprints: on multiple choice, up to 97% of the dial's new errors land on a single answer letter—an "attractor" no natural model approaches. Subtracting the estimated shift recovers 52–59% of lost accuracy.
- Implication for safety evals: a low score on a dangerous-capability probe isn't automatic reassurance. Check which items fail and how errors are distributed before reading a low score as low capability.
More from Safety
- Matt Levine on the antitrust paradox: insurance loophole for AI safety coordination — zacharynado · 2026-11-18
- OpenAI launches Daybreak to defend NATO and allied networks — gdb · 2026-11-18
- Community Note Corrects Dwarkesh: Christiano Was Head of AI Safety at AISI, Not Its Director — NathanpmYoung · 2026-11-18
- FBI warns hackers are using an AI-powered tool to automate attacks worldwide — Polymarket · 2026-11-18
- Optimizely supply-chain attack infects CNN, Fox News, BBC, Salesforce sites with malware — balderdash · 2026-11-18
- Smuggled Blackwell fingerprints found in Chinese open-weight models; Jensen Huang faces scrutiny — Difficult-Limit-7551 · 2026-11-18