CompressionBench: coding agents autonomously compress LLMs, 29/36 pass verification
BeidiChen · x · 2026-11-07
Infini-AI Lab and CMU release CompressionBench (arXiv:2611.06858), testing whether coding agents can autonomously compress LLMs amid power and memory constraints (HBM/DRAM shortages, data center pushback).
The setup: an agent receives a source checkpoint, storage limit, inference environment, and time budget, then must produce compressed weights and executable inference code, guided by perplexity feedback and verified independently.
Key results:
- Across 6 agents and 6 model-size settings, 29 of 36 submissions pass verification
- Best agent design per task yields only 3.4% perplexity increase over BF16, with 77.33% downstream accuracy vs 79.56% for BF16
- Agents lower PPL in 21 of 24 revisions at nearly fixed storage; whole-model feedback reverses 17 of 79 local rankings
- Conclusion: representation design, experimental judgment, and capability validation are central to agent-driven compression
More from coding & agent
- LLM agent re-architects models during training: algorithmic reasoning 55%→84%, AIME 64%→71% — lateinteraction · 2026-11-07
- AI won't replace designers: a deep dive into the Quest team's AI iOS design workflow — JiliJeanlouis · 2026-11-07
- Developer stumbles into an accidentally recursive SKILL.md that tests itself — holdenmatt · 2026-11-07
- Anthropic details agent memory: "dreaming" helps Claude fix mistakes and organize knowledge — RLanceMartin · 2026-11-07
- TechSEO Connect talk: write SOPs as skills and use small commits when vibe coding — lilyraynyc · 2026-11-07
- Conductor AI agent orchestrator shuts down, alternatives compared — vikvang1 · 2026-11-07