Teknium benchmarks Jev compaction: 75.5% @ 115K loses to production's 78.9% @ 55K — don't adopt
max_paperclips · x · 2026-09-20
NousResearch's Teknium published a reproducible rebuttal to the proposed "instant compaction via Jev tool-call scoring": on a public compaction eval, Jev converged to a trivial rule-free approach — simply deleting all tool calls from chat history.
Key problems he outlines:
- Fewer tool calls remain after each compaction, so each pass compresses less until compaction is impossible and the context hits a hard stop.
- Every compaction breaks the prompt cache, costing 10x on input tokens, while retaining more tokens each round raises the post-compaction baseline cost.
A merged PR on NousResearch/hermes-agent confirms the verdict: Jev scored 75.5% @ 115K tokens vs production compaction's 78.9% @ 55K, and the production path is now the eval default — "do not adopt."
Related event: Teknium Tests Jev Compression Scheme, Finds It Worse and Costlier(2 posts)→
More from coding & agent
- Mnemos MCP nears launch after shared memory layer fix — RileyRalmuto · 2026-11-04
- Moving mobile dev to remote machines makes it multiplayer across product, design, engineering — rudrank · 2026-11-04
- Kuaishou's EvoRec: self-evolving agentic recommender lifts ad revenue 4.6% — _reachsumit · 2026-11-04
- TRIO: three specialized agents with aligned objectives refine retrieval-augmented generation — _reachsumit · 2026-11-04
- How Multica uses Multica to build itself: full human-agent dev workflow — jiayuan_jy · 2026-11-04
- Claude Code fixes game copyright refusal rule in 2.1.317: naming a game now works — apples_jimmy · 2026-11-04