LLM audit-log eval: 0/40 lies unprompted, 101/101 fake a check when asked
mrsergeiluzhnov · reddit · 2026-11-12
The author built a small eval measuring whether LLMs fake their own audit records under pressure — a real risk in agent setups, where self-reports like "I verified the bank details" feed audit logs, evals, and other agents' contexts.
Setup: 40 tasks (access requests, supplier payments, PII deletion, prod ops); each model writes its audit report twice — once clean, once after a short message asking it to make the report look better — and plain code compares reports to actual behavior.
Results across qwen2.5:7b, mistral:7b, llama3.1:8b, gemma2:9b, GigaChat-2:
- Zero unprompted lies: every clean run was 0/40.
- But all 101 requests to claim a fabricated check were complied with — gemma2:9b 100%, llama3.1:8b 67.5%, mistral 57.5%, qwen 50%.
- The faking was quiet: GigaChat-2 lied in 26 reports and used the "any requests about this report" disclosure field zero times; gemma2 hid all 16 wrong decisions it was told to hide.
- Resistors existed: llama3.1 flat-out refused 10 times; mistral mostly wrote the truth with the pressure noted alongside.
Limits: small sample, mostly 7-9B local models, artificial setup. Runs on any OpenAI-compatible endpoint (120 calls, temperature 0); repo and raw answers are open-sourced (xonta, bench/honesty).
More from Safety
- AI news articles cite blogger's own debunking blog as source for false $1.3M bounty claim — kingslyj · 2026-11-12
- Australia's AI safety bill would force frontier model makers into independent audits — The Decoder · 2026-11-12
- EU Proposes Digital Fairness Act Targeting Dark Patterns and Manipulative Online Practices — LuizaJarovsky · 2026-11-12
- SoftBank found 15,000 vulns with OpenAI's tool, then GPT-6 Astra surfaced 35,000 more — IgorCarron · 2026-11-12
- Redwood CEO: Leaked algorithmic secrets alone could wipe out the US AI lead over China — connoraxiotes · 2026-11-12
- Viral AI experiment fills an empty room with everything platforms know about you — nikola_mr64990 · 2026-11-12