Deception probe authors admit 12-69% false positive rates; "honesty circuit" was media shorthand
kevrussell · x · 2026-11-03
Kevin Russell pushes back on the "honesty circuit" framing, noting the original work never made that claim—it was media shorthand. He concedes his own deception probes need the same controls: the pilot probe chosen for best AUROC flags 12.40% of honest examples, while the recall-optimized probe catches all 165 deceptive samples but flags 68.99% of honest ones. On Astra's sycophancy test: identifying the sycophant in 3/3 pairs via attention patterns is real, but head 10.2's pattern shifts with context and released patch tests show zero closure of the agreement gap. Telling models apart isn't explaining them.
More from Safety
- Context drift broke a production LLM deployment by turn 10 — no jailbreak needed — GasProfessional5544 · 2026-11-03
- Doppler curves of passing airplanes can leak your exact location from any video with sound — cyb3rops · 2026-11-03
- Flock comms chief quietly rebrands as "Fmr Flock Exec" amid AI surveillance backlash — jengolbeck · 2026-11-03
- ToolVet: 39.9% of 134,916 paid APIs and MCP servers failed latest check, up from 12.8% — BudgetBon · 2026-11-03
- FLI and 16 signers file amicus brief backing NYT's copyright case against OpenAI — TuhinChakr · 2026-11-03
- Zvi rebuts Scott Alexander: case against SURA strategy is 'deeply confused' — TheZvi · 2026-11-03