StreamEmbed: 3.7x faster screenshot embedding via pixel-diff pruning and KV reuse
_reachsumit · x · 2026-11-18
StreamEmbed (UC Berkeley, SNU et al.) is an incremental visual embedding system that turns screen history into searchable visual memory:
- Motivation: OCR indexing is fast but loses visual info; full visual encoding takes >2s per screenshot, yet most pixels repeat across frames.
- Method: guided solely by pixel differences, it combines spatial token pruning with temporal KV-cache reuse for unchanged blocks, plus a small learned adapter to compensate.
- Results: on their ScreenMemEval benchmark (10,426 questions over 184 hours of screen history), StreamEmbed encodes up to 3.7x faster than full-screenshot encoding on an M2 Pro GPU without accuracy loss, beats OCR indexing by up to 6.3 Recall@10 points at matched latency, and a memory-efficient vector index stores a simulated year of screenshots in 0.13 GB RAM.
More from coding & agent
- Ramen open-sources eval gates and guardrails for shipping MCP tools like code — Ok_Plum3595 · 2026-11-18
- RAG-FGO paper frames RAG query rewriting as black-box optimization over prompts and decoding — _reachsumit · 2026-11-18
- Reddit debate: does having multiple AIs cross-check each other actually save time? — shopaholic88 · 2026-11-18
- As agents do the work, project management tools should track invariants, not tasks — _AustinCalvert_ · 2026-11-18
- Sol6.2 does PCB layout for ~$4 in 40 min, rivaling fable5.2 at 1/15 the cost — yacineMTB · 2026-11-18
- In the AI coding era, CI perf, security and code review are all P0 now — cnakazawa · 2026-11-18