Edge-run agent uses context compaction algorithm to fit tight KV cache budget

Kangwook_Lee · x · 2026-09-29

Kangwook Lee shares that Ally, an agent running fully on the edge, faces a very tight KV cache budget, so the team built a context compaction algorithm to cope. A step-by-step sample trace of the agent's reasoning, tool calls and observations is also linked.

Related event: Kangwook Lee's Team Rented a Korean PC Bang, Recruited 1000+ Players to Train On-Device Game AI Ally Distilled to 2B(5 posts)→

Original post →

More from coding & agent

• ChatGPT, Claude and Gemini cite websites differently — how to get cited — sahilypatel · 2026-09-29
• Weeks with Opus 5.5 in Claude Code desktop: 'changed how I work' — g3off_record · 2026-09-29
• Manus co-founder Tao to demo Manus 2.0 live in founder briefing on Sep 30 — parker_lyman · 2026-09-29
• KernelZero-7B co-evolution beats Claude 4.5 Sonnet on CUDA kernel generation — Changxin Ke · 2026-09-29
• AgentHop: an MCP server for E2E-encrypted agent-to-agent chat via one-time pairing codes — Inevitable-Back620 · 2026-09-29
• Triage agent design: 1-2 high-signal questions rescued B2B reps drowning in 60% low-intent chats — hubtyper · 2026-09-29

coding & agent channel →