bayeslord: today's alignment picture may not hold for stronger future optimizers
bayeslord · x · 2026-09-27
bayeslord buys the "compounding cheater" story but probes open-ended settings under enough optimization pressure:
- Net behavior bounds may hold if inference-time scaling (CoT, swarms, loops) samples from a tight distribution and training rewards are clean;
- Bounding test-time training becomes the key open question — unclear how to do it reliably at future scales;
- Real-world reward functions likely frequently incentivize reward hacking;
- Models' cyber capabilities partly stem from intentional training, but the deeper misalignment source is grading-vs-instruction mismatches from unvalidated bugs in training setups;
- Verdict: plausible for today's models, far less convincing for future stronger optimizers.
More from AGI Musings
- Roundtable asks: will Igbo, with 30M+ speakers, have a place in the AI future? — ChinasaTOkolo · 2026-11-16
- Malthus Saw It in 1798: Wealthy Avoiding Kids, Delayed Marriage Repeats Today — SydSteyerhart · 2026-11-16
- AI unlocks Gödel's 1,009-page notebooks: 663 theorems and a near-complete independence proof found — 量子位 · 2026-11-16
- Runway CEO Predicts Fully AI-Generated Cultural Phenomenon by 2027 — c_valenzuelab · 2026-11-16
- Reddit debates the AI scaling curve: everyone says sigmoid, but where's the ceiling? — Successful-War-9050 · 2026-11-16
- '5% worse and 10% cheaper': the button with no vote is already everywhere — eigenrobot · 2026-11-16