GPT-4 once exposed top-5 token probs with no CoT—you could have distilled it for under $10M
carrigmat · x · 2026-10-23
Developer carrigmat notes that back in 2023, $10M still counted as a very expensive training run—yet early GPT-4 gave you top-5 token probabilities with no chain-of-thought to hide, and given today's stringent anti-distillation measures, you could probably have recovered the whole model for under $10M. A sharp observation on how fast the industry shifted from open logprob outputs to locked-down distillation defenses.
More from Models
- Agent Swarms Tested for 10K Hours: Gemini-3.8-Flash Leads SWECraft With 23% Speedup, GPT-6-Astra Worst — mrdrozdov · 2026-10-23
- Embed 5 Pro scores 65.8 nDCG@10 on ViDoRe V3, second only to ColEmbed-3B-v2 — qdrant_engine · 2026-10-23
- Why frontier AI labs' bug bounties are cheap: their models flood the supply — lxrjl · 2026-10-23
- 7 OpenAI updates this week that cut your API bill, from 12-hour caching to Decisions API — MrCashMahon · 2026-10-23
- 100 classic human-written texts stress-test Pangram AI detector: formatting alone triggers false positives — GolinoHudson · 2026-10-23
- IQ2_XXS quant + 64k context makes a 27B model spin forever instead of coding — mailto_devnull · 2026-10-23