FrankenAlignment project replaces text CoTs with compressed activations via sidecar model
doodlestein · x · 2026-09-11
Responding to tszzl's take that preserving plain-text chain-of-thought will be seen as an 'alchemical era of observability,' doodlestein introduces his new FrankenAlignment project: building on his earlier idea of compressing a big model's activation states using a smaller sidecar model, aiming to understand models' alien ontology from the ground up. A comprehensive GitHub design document lays out the executable research program and concrete corrections required by the prototype.
More from AGI Musings
- Mathathon organizers respond to mathematicians' open letter, weigh redesign — _sathvikr · 2026-09-11
- Would AI proofs for 3 millennium problems discourage humans from the rest? Debaters spar — sytelus · 2026-09-11
- Gary Marcus to debate whether we should boycott generative AI live on CNN — GaryMarcus · 2026-09-11
- Caltech Mathematicians Spar Over Whether AI Belongs in Undergrad Math Training — Singularitarian · 2026-09-11
- AI apps have a Dunbar's number of 3-4: high-end users say the tool market is saturating — nptacek · 2026-09-11
- AI company employee puts ≥10% odds on out-of-control AI killing everyone — EvanHub · 2026-09-11