Open Source Tool Slashes 87% of LLM API Input Tokens
naruto_uzumaki00 · reddit · 2026-07-05
A developer open-sourced a vendor-neutral token optimization layer, serving Claude Code/Codex via a proxy + MCP plugin. On real billed tokens for GPT-5.5 and Opus 4.8, inputs dropped from 16875/26573 to 2232/3343 respectively (an 86.8%/87.4% reduction), while task quality held steady at 3/3.
The stack relies on four core mechanisms: prefix caching, tail compression with numeric guards, AST/text retrieval, and cascading verifiable steps to a local model. The author transparently lists limitations: static embeddings didn't beat keyword retrieval, the 3B compressor occasionally dropped critical numbers, and the evaluation sample size was small and biased favorably. Released under Apache-2.0 with reproducible benchmarks.
Related event: Open MCP Server Slashes LLM Input Tokens by 87%(2 posts)→
More from coding & agent
- Prompt guardrails aren't enough: layered defenses for AI agents that take real actions — Informal-Dust4499 · 2026-10-27
- AIE Code Summit 2026 opens applications: 800 seats, keynotes streamed free — trq212 · 2026-10-27
- Per-surface intent thresholds: a practical guardrail design for AI email agents — marlene_zw · 2026-10-27
- Decade-long cloud architect: enterprise AI architecture is far more than bolting on an LLM — kashifmanzoor · 2026-10-27
- Analysis of thousands of coding sessions: 40-89% of files agents read are never used — rseroter · 2026-10-27
- TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27