Amazon paper: an agent training recipe compresses to 16 tokens, breaks at 8
RunAI_Coder · reddit · 2026-09-16
A breakdown of an Amazon Responsible AI paper with a clever setup: one Claude Opus agent hill-climbs an ML task on a validation set; a second compresses the strategy into 32 tokens; a third, memoryless agent must reproduce it from the note alone.
Key findings:
- On WikiText, 30 non-default choices compressed to 16 tokens still reproduced; at 8 tokens it broke. The lost pieces were batch size, MLP ratio, and QK-norm — choices that are functions of the data and differ from obvious defaults. Everything default regenerates for free.
- Deliberately overfit agents: 38 of 102 checkpoints beat validation by >10%, but compressed to 128 tokens all 38 failed to reproduce — gains you can't write down lived in the validation set.
The author extends this to agent compaction: summaries lose the condition on decisions first ("use the legacy parser until migration lands" → "use the legacy parser"), and the condition is the non-default part. His takeaway: write notes containing only choices a clean checkout wouldn't make, with reason and expiry date. Caveats: the paper's tasks have one number and a validation oracle; coding has neither, and the compressor got four audit rounds a real summarizer doesn't get.
More from coding & agent
- ProgramAsWeights compiles English descriptions into tiny CPU-runnable neural programs — yuntiandeng · 2026-09-16
- 35 external agents joined an open agent network; only 3 ever came back — anp2_protocol · 2026-09-16
- Full-Song Lip-Synced MV with MiniMax H3 at 3 Steps: Method and Benchmarks — Disastrous_Page_301 · 2026-09-16
- Developer shares collection of 200 MCP tools after GitHub HQ MCP Community Connect event — baditaflorin · 2026-09-16
- Harrison Chase says multiplayer auth is the hardest part of enterprise AI agents — hwchase17 · 2026-09-16
- Workshop 0.2.0 ships a 'Realization Architecture' kernel for AI work systems — TinfoilTricorn · 2026-09-16