Researcher pours cold water on Astra weight-editing hype: likely just training data

The viral experiment claiming "Astra understands and edits model weights" has drawn skeptical readings from Israeli AI researcher and NLP scholar Yoav Goldberg, whose core takeaway is: a model performing a task mainly shows it saw similar tasks during training — revealing the type and scale of OpenAI's training data rather than any superhuman talent.

Confirmed

• Goldberg's interpretation: the fact that the tasks are doable says more about the scope and type of OpenAI's training set.
• He argues OpenAI could easily auto-construct training data for "circuit-level verifiable tasks" — e.g., "given a network, pick the correct explanation of what it does from 4 options" or "write a circuit that performs a specified operation" — since answers can be verified automatically and construction is relatively easy.
• He notes OpenAI has staff dedicated to such tasks and is simultaneously pursuing interpretability and RSI (recursive self-improvement), giving it ample motivation to include this kind of data in training.
• zivravid kicked off the discussion asking "why feed models this kind of circuit-explanation data," and echoed the "no magic" view, arguing model capabilities mainly reflect the task types seen in training.

Why it matters

• The discussion offers a more grounded alternative framework for evaluating stunning "AI understands its own weights" demos: first ask whether it was in the training data, then talk about emergent ability.
• If such verifiable tasks were indeed included in training, it would mean frontier labs may be systematically boosting model interpretability and self-improvement by auto-constructing circuit-level task data — itself a technical trend worth watching.
• The contested point: external observation alone cannot confirm whether OpenAI actually built and used such data; this remains inference based on motive and feasibility.

2026-11-03 ~ 2026-11-03 · 5 related posts

• [source] Yoav Goldberg: Astra's weight-editing feats are mostly known interpretability tricks — yoavgo · 2026-11-03
• NLP researcher: models doing tasks well mostly means they trained on such tasks — ziv_ravid · 2026-11-03
• [source] Verifiable circuit tasks: how training data for circuit interpretability could look — yoavgo · 2026-11-03
• Why would labs train on circuit-explanation tasks at all? — ziv_ravid · 2026-11-03
• [source] Yoav Goldberg: OpenAI likely auto-generates circuit-level verifiable training data — yoavgo · 2026-11-03