Building an active monitor for a black-box LLM: where to draw the automation line
Particular-Roof4257 · reddit · 2026-09-08
A developer shares a design for monitoring an external LLM whose internal state is invisible: continuously track response quality, behavioral drift from baseline, refusals, instruction following, latency, errors, and user feedback, and upgrade from an ACCEPT/REJECT policy to ACCEPT/INVESTIGATE/REJECT, where the INVESTIGATE state spends extra resources on targeted probes. Open questions: distinguishing provider updates from distribution shift, prompt sensitivity, evaluator noise, or real degradation; a model can look healthy on aggregate metrics while failing a small class of high-impact requests; and alert fatigue can make the system untrustworthy. He asks for engineering perspectives on what evidence matters, when human judgment is needed, and approaches that failed in practice.
More from coding & agent
- Grok Build launches Workflows: fan tasks out across up to 1,024 parallel agents — elonmusk · 2026-09-08
- Internal AI bot leaked unannounced reorg plan and salary bands via over-scoped Drive access — Accomplished-Wall375 · 2026-09-08
- Live experiment: a tiny harness, five tools and Markdown to give your AI a personal knowledge library — dfinke · 2026-09-08
- Dev praises Codex compaction for recalling useful details from 2M tokens ago — adonis_singh · 2026-09-08
- Give an Agent a Markdown File and It Will Fill It With Stale Docs — mattpocockuk · 2026-09-08
- Automating weekly to-dos from local meeting transcripts — bgmshana · 2026-09-08