AI Auditors Pilot: GPT-6 Astra Hits 15% False Positives at 99% Detection Target
andrewaltair · reddit · 2026-11-08
In an October 15 pilot, Andy Hall and Dan Thompson compared AI auditors against human-labeled agent logs to assess AI-overseeing-AI supervision.
- On R-Judge, GPT-6 Astra reached 15% false positives when set to catch 99% of unsafe actions
- At a 95% detection target, nearly all models stayed below 10%; Gemini 3.8 Flash stayed under 4%
- Caveat: the benchmark measures labeled logs, not real-world deployed-agent rates
The results highlight the precision-recall dilemma in agent safety: near-complete detection with low false alarms remains out of reach for current models.
More from Models
- LightOn releases LightOnOCR-3: 0.8B-4B open-source OCR models under Apache 2.0 — IgorCarron · 2026-11-08
- Abliterated GLM's "I am compressed humanity" self-description goes viral — RileyRalmuto · 2026-11-08
- antirez: DwarfStar now runs Kimi K3.1 PRO inference from safetensors via SSD streaming — antirez · 2026-11-08
- teortaxes: Plugin creation in DeepSeek Hub feels impressively natural — teortaxesTex · 2026-11-08
- Tencent's Terminal-RSI: a 27B model that rewrites and retrains itself with zero humans — Scobleizer · 2026-11-08
- Weight Space Programming: Claude Opus 5.5 beats GPT-6 Astra at hand-written 18-weight parity transformer — RobertTLange · 2026-11-08