GIST paper: semantically steerable image tokenizer cuts LPIPS up to 30% vs pixel-based control
cgmsnoek · x · 2026-10-21
Danilo de Goede et al. (University of Amsterdam) introduce GIST (Guided Image Semantic Tokenizer), a semantically steerable adaptive image tokenizer (arXiv:2610.15515).
- Problem: prior adaptive tokenizers allocate tokens by pixel-wise reconstruction error, which is blind to the spatial structure carrying an image's meaning; under heavy compression they abruptly discard tokens encoding essential content
- Method: token allocation is jointly controlled by pixel-level and semantic target errors, with semantic error measured in the feature space of a frozen DINOv3 encoder; fixing the semantic target at zero lets the reconstruction target relax, discarding pixel detail while preserving what the image depicts
- Results: on ImageNet-100, GIST beats its pixel-controlled counterpart KARL at every token count, with up to 30% lower LPIPS and 37% lower DreamSim; lowering the semantic target reduces LPIPS 2x faster per token than lowering the reconstruction target, up to 5x in the low-token regime
Takeaway: semantic fidelity is a more effective control signal for compression than pixel error.
More from Research
- ICA Lens in action: decomposing Qwen residuals reveals clean city/France features — furongh · 2026-10-21
- Into the Unknown: none of 135 submissions by 9 frontier agents passes scientific bar — wellecks · 2026-10-21
- REUF2F Benchmarks Coding Agents on Formalized Undergraduate Research Problems — wellecks · 2026-10-21
- Baker Lab and NVIDIA Unveil RFD4-Proteina, a Faster Protein Design Model That Beats Prior Methods — jueseph · 2026-10-21
- NVIDIA Kumo Forecast Pairs Pretrained Time-Series Representations with Pattern Retrieval — Hugging Face Blog · 2026-10-21
- Protein designers adopt RFD4-Proteina for Adaptyv x Anthropic competition — AllThingsApx · 2026-10-21