ARC's Jacob Hilton Sparks Debate: Interpretability Far From Solved

On October 16-17, Jacob Hilton, a researcher at ARC (Alignment Research Center) and former OpenAI superalignment member, kicked off a big debate over whether interpretability is "solved," with researchers like Neel Nanda joining in. Hilton's core claim: our understanding of frontier language models' internal cognition is well under 0.01%, and without fundamental change in the field, that share will keep shrinking as models improve—the bigger the model, the harder it is to fully understand, not easier.

Confirmed

• ARC released AlgZoo: a set of tiny RNNs and transformers performing algorithmic tasks with just 8 to 1408 parameters, built specifically to test any ambitious "fully understand the model" proposal. One 432-parameter RNN (tasked with finding the second-largest number in a sequence, with weights small enough to fit in a tweet) still cannot be fully explained by current interpretability techniques.
• Hilton examined Anthropic's interpretability team paper "On the Biology of a Large Language Model": the detailed attribution graph for computing 36+59 has 88 nodes and 2099 edges, many without readable labels; the paper's graph completeness faithfulness metric scores typically only 50% on that graph (Figure 32).
• Structural argument: even writing a description for every neuron isn't enough, because neuron counts and parameter counts both grow exponentially while their ratio declines exponentially, so neuron-by-neuron explanation is inherently at odds with scaling; full understanding also requires explaining how neurons wire into circuits, where he judges progress "not optimistic."
• Hilton revisited "Understanding RL Vision" (Distill, 2020, with Nick Cammarata et al.), arguing most of its core open problems remain unsolved.

Not yet confirmed

• The common rebuttal that "frontier models are somehow easier to understand": Hilton puts the burden of proof on that side, and the thread offered no supporting evidence.
• On "partial understanding (even 0.01% or 0.0001%) is useful": Hilton agrees it has value, but stresses it's a different claim from "solving alignment by understanding everything"; peers still disagree.

Why it matters

• Hilton argues that unlimited optimism from some peers is painting the public an unrealistic picture of interpretability's current state. The discussion directly challenges the "interpretability is nearly solved" narrative, giving the mechanistic interpretability "pragmatists vs. ambitionists" debate concrete benchmarks (AlgZoo's tiny models) and quantitative evidence (faithfulness scores).

2026-10-16 ~ 2026-10-17 · 22 related posts

• Revisiting Distill's 'Understanding RL Vision': attribution-based interpretability tricks that still matter — JacobHHilton · 2026-10-16
• [source] ARC researcher: we understand far less than 0.01% of frontier LLMs' internal cognition — JacobHHilton · 2026-10-16
• [source] ARC's AlgZoo: a 432-parameter model that interpretability still can't fully explain — JacobHHilton · 2026-10-16
• [source] A 432-parameter RNN still defies full interpretability, ex-OpenAI researcher shows — JacobHHilton · 2026-10-16
• Interpretability drama: ARC researcher says a peer's optimism misleads the public — JacobHHilton · 2026-10-16
• Interpretability debate: are smaller models harder to understand? — giosullutrone99 · 2026-10-16
• DeepMind's Jacob Hilton: interpretability lags far behind model scaling — JacobHHilton · 2026-10-16
• Do trillion-parameter frontier models get easier to understand? — JacobHHilton · 2026-10-17
• We understand far less than 0.01% of frontier LLM internals, researcher argues — JacobHHilton · 2026-10-17
• Neuron counts grow exponentially, but parameters grow faster — JacobHHilton · 2026-10-17
• Full understanding means explaining circuits—and it's not looking good — JacobHHilton · 2026-10-17
• Beyond neurons to circuits: interpretability's next hurdle looks shaky — JacobHHilton · 2026-10-17
• The beautiful 36+59 attribution graph comes with fine print — JacobHHilton · 2026-10-17
• The full attribution graph for 36+59: 88 nodes and 2,099 edges — JacobHHilton · 2026-10-17
• Attribution graph for one addition has 88 nodes and only ~50% faithfulness — JacobHHilton · 2026-10-17
• 88 nodes, 2,099 edges: the circuit behind a model computing 36+59 — JacobHHilton · 2026-10-17
• Interpretability red flag: graph completeness only ~50% on an 88-node addition circuit — JacobHHilton · 2026-10-17
• Do features get crisper with scale? It's only gradual, argues Jacob Hilton — JacobHHilton · 2026-10-17
• Interpretability researcher: even a 432-parameter model resists full understanding — JacobHHilton · 2026-10-17
• Interpretability Debate: Is Understanding 0.01% of a Frontier Model Enough? — JacobHHilton · 2026-10-17
• 'Models get harder to understand with scale, not easier': new interp debate thread — NeelNanda5 · 2026-10-17
• Jacob Hilton: graph completeness measures completeness, not faithfulness — JacobHHilton · 2026-10-17