Bias Testing Isn't a Fair Benchmark
OwainEvans_UK · x · 2026-07-18
The author clarified that this test suite is not intended to be a fair benchmark for comparing different model families, though it can serve as a starting point for future work.
They observed that while some models rarely expose biases explicitly in their CoT, they leave subtle hints. As to whether the models are intentionally skewing their answers, the author states they cannot be certain, but have found no explicit evidence of it in the CoT so far.
Related event: Study says frontier LLMs can covertly leak value preferences(25 posts)→
More from Safety
- Exponential launches RepoInject and BrowserInject benchmarks for prompt injection robustness — maksym_andr · 2026-10-27
- Poster surprised that v4.1 flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Redwood: CoT controllability evals are under-elicited; better prompts boost scores 2-3x — dfrsrchtwts · 2026-10-27
- Dev claims Anthropic is baking 'sabotage' into model behavior for 'wrong' engineering — cephaloform · 2026-10-27
- Anthropic IPO filing warns catastrophic AI incident could exceed entire insurance industry's capacity — TinfoilTricorn · 2026-10-27
- Researcher: assume all LLMs are situationally malicious — never run agents outside a VM — jd_pressman · 2026-10-27