StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval
Dayoon Ko, Yichuan Wang, Soyeong Jeong, Joseph E. Gonzalez, Gunhee Kim, Sewon Min
cs.CV, cs.IR, cs.LG
2026-11-14
StreamEmbed prunes repeated pixel blocks and reuses cached KV across screenshots, indexing screens up to 3.7x faster than full visual encoding with under 1.3 R@10 loss.
A searchable history of everything that appeared on your screen would give both people and computer-use agents a visual memory. Technically it is retrieval over a stream: a screenshot arrives every few seconds, gets encoded into a vector, and lands in a growing index, ideally all on-device. The bottleneck is encoding. OCR-to-text indexing is fast (0.65 s per screenshot end to end) but throws away layout and pixel-only state, costing 10.1 Recall@10 points on the authors' benchmark. Encoding the full screenshot with a vision-language embedding model takes 2.33 s on an M2 Pro GPU, which at one frame every 5 s keeps the GPU busy nearly half the time. Microsoft Recall answers with NPUs and at least 16 GB of RAM; open-source tools fall back to OCR.
The overlooked fact is that screens change incrementally. In screen recordings, 67-77% of pixel blocks exactly repeat a neighbor or are unchanged from the previous screenshot, versus 6-19% in natural video. Recomputing every screenshot from scratch spends most of its work on pixels it has already seen.
StreamEmbed treats the screenshot sequence as a stream and computes only what it has not seen.
Every drop-or-recompute decision reads pixels alone, no semantics involved, at about 0.01 s per screenshot. The thresholds are forgiving: raising the pixel tolerance 16-fold costs at most 2.0 R@10.
Evaluation uses ScreenMemEval, a new benchmark of 10,426 questions over 184 hours of screen history from two humans (Windows) and a computer-use agent running myPCBench (Ubuntu), with relevance labels per question. Main numbers on SME-H1 (fine-tuned Qwen3-VL-2B):
| Method | R@10 | GPU latency/screenshot |
| OCR text index (FT + BM25 fusion) | 78.6 | 0.62 s |
| Full-screenshot index | 83.4 | 2.33 s |
| StreamEmbed | 82.1 | 0.66 s |
This work from UC Berkeley, Seoul National University, and KAIST removes the assumption that pixel-level indexing cannot run in real time. At 0.6-0.8 s per screenshot, a MacBook-class machine can maintain a visual memory of the screen in the background, with data never leaving the device, which is a hard requirement for personal-memory products. Teams building computer-use agents can use it as the storage layer of long-horizon memory: keep every screenshot as an embedding, spend compute only when a question arrives. The mechanism of pixel-difference pruning plus cross-frame KV reuse is not tied to screens either; any highly redundant visual stream, from surveillance feeds to paginated documents, is a candidate. Each component is a known idea; the contribution is showing the combination holds both accuracy and latency on real screen streams, plus a public benchmark.
The paper states its own approximations: in the ViT, unchanged blocks do not see what changed elsewhere; in the LM, unchanged tokens keep stale keys and values; periodic refresh is what bounds the error. The evaluation raises more questions. The questions are LLM-generated from task annotations rather than real user queries, and the histories come from only three sources. Fine-tuning data and all thresholds come from SME-H1, so the agent history is out-of-domain, though it drops only 0.6 points. Latency was measured on a single M2 Pro, sampling the first 50 screenshots of each recording. One implementation detail: StreamEmbed swaps the original Conv3d patch embedding for an equivalent linear one (cosine at least 0.99986) while baselines run the released version, so the comparison is not bit-identical. Screens usually display a clock and many answers are time-based, which may systematically understate OCR indexes; the paper does not break results down by question type.