Multi-vector visual document indices can be inverted: 47% of words recovered, 98.4% source-page recall

Zhuchenyang Liu · hf · 2026-10-08

New security research shows multi-vector visual document retrievers store 1,000 patch vectors per page—often in third-party vector stores—and these indices are far more sensitive than assumed.

Findings:

Since vectors are stored in raster order and computed by a document-reading VLM, inversion can be framed as conditional document image generation, requiring only the encoder, page shape, and (for shuffled vectors) their order;
On ViDoRe v3, pages inverted from raw indices recover 47% of words and 45% of sensitive tokens, and rank their source page first 98.4% of the time;
Token pooling and shuffling cut word recall to 8%, but a model that restores shuffled order raises first-ranked source pages from 3.8% to 93.5%; inverting pooled indices remains open;
The same attack transfers to another multi-vector retriever with 70.2% source-page first ranking.

Conclusion: these indices should be protected like the documents they encode.

Original post →

Multi-vector visual document indices can be inverted: 47% of words recovered, 98.4% source-page recall

More from Safety