jd_pressman shares alignment roadmap: Weave algorithm plus Gillen quantilizer against Goodhart
jd_pressman · x · 2026-10-16
jdpressman elaborated his thinking with Discord screenshots: Fedorov proposes a "Weave algorithm" to mitigate causal Goodhart — Goodhart-resistant the way humans are heroin-resistant — and a Gillen quantilizer (from the SERI paper "Soft optimization makes the value target bigger") for extremal Goodhart, capping optimization power based on goal-representation reliability.
Other points: value loading is mostly a data problem, with judicial and economic data as the least-noisy signal of human values since these are domains where humans are forced to be unusually reasonable and consistent; video games could probe intrinsic motivation structure; jailbreak resistance/adversarial examples is deprioritized — "homework for the seed AI(s)."
Related event: Alignment Researcher: Core Problems Known but Almost Nobody Solving Them(3 posts)→
More from AGI Musings
- Blogger predicts recursive self-improvement arrives around November 2027 — imjustnewatai · 2026-10-16
- 31 Tech Curves Visualized: Kurzweil's $1,000-Brain-Compute Call Missed by 400x — cpaik · 2026-10-16
- DeepMind Paper: First-Price Auctions in Agentic Markets Reach 1-1/e^2 Efficiency — dair_ai · 2026-10-16
- Garrison Lovely's new book 'Obsolete' argues AGI is neither impossible nor inevitable — GarrisonLovely · 2026-10-16
- Majmudar clarifies: controllability may be solved in 1-2 years, not full alignment — MajmudarAdam · 2026-10-16
- AI-generated fake Feynman lectures are flooding the internet, and they're bad — dreamwieber · 2026-10-16