Teknium benchmarks Jev compaction: 75.5% @ 115K loses to production's 78.9% @ 55K — don't adopt
max_paperclips · x · 2026-09-20
NousResearch's Teknium published a reproducible rebuttal to the proposed "instant compaction via Jev tool-call scoring": on a public compaction eval, Jev converged to a trivial rule-free approach — simply deleting all tool calls from chat history.
Key problems he outlines:
- Fewer tool calls remain after each compaction, so each pass compresses less until compaction is impossible and the context hits a hard stop.
- Every compaction breaks the prompt cache, costing 10x on input tokens, while retaining more tokens each round raises the post-compaction baseline cost.
A merged PR on NousResearch/hermes-agent confirms the verdict: Jev scored 75.5% @ 115K tokens vs production compaction's 78.9% @ 55K, and the production path is now the eval default — "do not adopt."
Related event: Teknium Tests Jev Compression Scheme, Finds It Worse and Costlier(2 posts)→
More from coding & agent
- Swarms Cloud changelog: Auto Agent Builder, MCP, and full agent execution observability — KyeGomezB · 2026-09-20
- Using Jev as a 75ms 'subconscious' filter to cut agent context bloat for under half a cent a day — Obvious_Unicorn · 2026-09-20
- Scoble says an AI agent wrote every word of an entire book — Scobleizer · 2026-09-20
- "Look at your data": dev mocks reflex to spin up another agent — chrisalbon · 2026-09-20
- Evaluating 7 Models Across Claude Code, Codex, and Pi: Harness Choice Drives Cost, Not Success — CShorten30 · 2026-09-20
- Dev builds MCP middleware that scrubs personal data before it reaches the AI's context — Danielloesoe · 2026-09-20