FrankenAlignment project replaces text CoTs with compressed activations via sidecar model
doodlestein · x · 2026-09-11
Responding to tszzl's take that preserving plain-text chain-of-thought will be seen as an 'alchemical era of observability,' doodlestein introduces his new FrankenAlignment project: building on his earlier idea of compressing a big model's activation states using a smaller sidecar model, aiming to understand models' alien ontology from the ground up. A comprehensive GitHub design document lays out the executable research program and concrete corrections required by the prototype.
Related event: tszzl: Clinging to Plain-Text CoT Is the Alchemy Age of Observability(6 posts)→
More from AGI Musings
- Pedro Domingos mocks NYT claim that superintelligence is already here — pmddomingos · 2026-10-28
- What if millions of AIs' combined incentives create irreversible outcomes? — PierceLilholt · 2026-10-28
- Coding matters less than problem-finding: the shift from code engineers to product engineers — CtrlAltDwayne · 2026-10-28
- Rachel Thomas points to classic paper on articulating the value of not building tech — math_rachel · 2026-10-28
- Hypothetical $2T Anthropic IPO framed as channeling $100B+ a year into the real economy — sanjaykalra · 2026-10-28
- Biotech CEO: drug discovery will lag in the AI revolution, not lead it — vaibhavbetter · 2026-10-28