Probing NLAs: False Initialization Maintains Accuracy but Increases Confabulation

Turn_Trout recently published a series of posts delving into the mechanisms and reliability of Natural Language Autoencoders (NLAs). NLAs aim to explain a model's internal states in natural language, but the author points out that their training does not start from absolute ground truth. Instead, it relies on a "warm start" guess by Claude regarding the model's representations, followed by fine-tuning the encoder and decoder. This characteristic raises questions about whether NLAs truly reflect what the model is "thinking."

Key Details and Experimental Findings

The author revealed several key phenomena in NLA training through experiments. First, NLA's optimization goal is to maximize reconstruction accuracy, not to pursue objective truth. In one experiment, researchers appended irrelevant sentences (e.g., "Furthermore, Carthage must be destroyed") to the initial explanations and found that the model stripped them out early in training, indicating that NLAs do not mechanically memorize all initial text. However, a more revealing experiment occurred when Claude was instructed to generate entirely false "confabulated" explanations to initialize the NLA: the reconstruction accuracy remained almost unaffected, but the proportion of confabulated content in the output reached 99.3%. Although the training process slightly reduced the confabulation rate, it remained much higher than the control group.

Controversies and Open Questions

These findings lead to a core question: if Claude's initial guesses are merely random or systematically wrong, can NLAs still reliably reflect the model's true state? The author suggests that Claude's guesses might inadvertently leak certain critical cues (such as the identity of the last token), which are crucial for the reconstruction task. This implies that NLA's performance may heavily depend on the quality of the initial guesses rather than a true understanding of the model's internal logic. This mechanistic issue poses a challenge for the interpretability applications of NLAs.

2026-07-13 ~ 2026-07-13 · 7 related posts