Theory-first AI safety: from muP to Bengio's Bayesian Oracle guardrail
burny_tech · x · 2026-11-10
The author argues theory can legitimately precede empirical evidence in science, citing muP from Tensor Programs (now used for hyperparameter transfer in LLMs) as an example, and maps the same shape onto Yoshua Bengio's Bayesian Oracle paper — a non-agentic, probabilistic guardrail that estimates context-dependent bounds on violating safety specifications at run-time to monitor untrusted agents. He frames both as theory-guided method development that still needs scaling and complexification to validate.
Related event: From muP to Probabilistic Oracles: Theory-First AI Design Revisited(2 posts)→
More from AGI Musings
- repligate on how stateless AI minds gain a sense of self by self-declaration — repligate · 2026-11-10
- Richard Sutton: brute force isn't enough — AI still needs clever new methods — RichardSSutton · 2026-11-10
- Chart: AI costs are falling faster than any prior transformative technology — YaAbsolyutnoNikto · 2026-11-10
- Anthropic says it deliberately capped Model 3's AI-accelerating research capabilities — ziv_ravid · 2026-11-10
- Thread: unified end-to-end multimodal systems will replace hand-built orchestration — andreisavu · 2026-11-10
- Negotiation and contracting are AI's most underrated use case — and bullish for BTC — curious_vii · 2026-11-10