Experiment suggests modern LLMs like Qwen hide tiny GPT2 self-models inside
paraschopra · x · 2026-09-22
Paras Chopra ran an exploratory study testing whether modern LLMs contain self-models of LLMs. Using 12 post-cutoff Sept 2026 headlines, he had GPT2-medium generate continuations, then compared GPT2's own completions against Qwen3-base (4B) finishing the same text, with Qwen's natural completions as control.
Key finding: Qwen's completions of GPT2-started text matched GPT2's style more than its own natural style, suggesting Qwen can simulate GPT2's distribution internally — consistent with the hypothesis that LLMs pretrain on so much AI-generated text that they implicitly model LLMs themselves. He plans a follow-up asking models to judge which model produced a completion. Caveat: quick-and-dirty exploratory work.
More from Models
- Will OpenAI eat Jev's lunch? The single-token classifier hypothesis — JnBrymn · 2026-09-22
- Will OpenAI Eat Jev's Lunch? Jev Is a Fine-Tuned LLM Classifier, Analysis Argues — JnBrymn · 2026-09-22
- Xiaomi's open MiMo-V2.6 Pro hits Intelligence Index 46, tops open weights models — dair_ai · 2026-09-22
- GGUF models can now run directly in Hugging Face transformers with ggml Metal kernels — pcuenq · 2026-09-22
- As OpenAI Preps a ChatGPT Bot to Rival Meta Muse and Grok, Here's What Personal AI Assistants Can Already Do — Efistoffeles · 2026-09-22
- New Claude model beats Pokemon in 50 hours, 5x faster than Opus 4.7 five months ago — ben_j_todd · 2026-09-22