460M VLM Cuts First-Token Latency to 0.3s on iPhone Using Only 64 Visual Tokens
BTA_Labs · reddit · 2026-08-05
VisionPsy-Nano-460M-Flash is a 460M parameter Vision-Language Model (VLM) optimized for on-device deployment. It achieves massive inference speedups on smartphones through a radical mechanism: compressing visual tokens down to just 64.
- Core Mechanism: Unlike comparable small VLMs (like SmolVLM2 or nanoVLM which typically require 1,088 visual tokens), this model avoids the standard practice of upsampling images before tiling. Instead, it preserves the native resolution (minimum 512x512), preventing artificially introduced pixels from generating extra tokens.
- Performance: Using Q4 quantization, the time to first token (TTFT) on an iPhone 15 is merely 0.3 seconds, whereas similar models take several seconds. Generating a full one-sentence image description takes just 0.7s.
- Trade-offs: While retaining around 99% of the full model's normalized benchmark score (62.3 vs 61.4), this aggressive compression noticeably degrades performance on OCR and fine-detail tasks like TextVQA.
The author notes important caveats: the 0.3s metric is only TTFT, it currently relies on a patched llama.cpp fork, context is limited to 8K, and it is designed primarily for single-image queries.
More from Infra
- Anthropic to Develop Custom AI Chips for Faster and More Efficient Claude — Polymarket · 2026-08-05
- Wall Street Expects AI Capex Surge: OpenAI Quarterly Spending Could Top $18B — PTrubey · 2026-08-05
- Benchmarking MiniMax H3 on a 4090: Sage Attention Slashes Generation Time — thegr8anand · 2026-08-05
- 3090 Upgrade Dilemma: Is 24GB VRAM Enough or Jump to 32GB? — gtech02 · 2026-08-05
- Stanford Hazy Research: AI Agents Are Retiring CUDA Abstraction Layers — sumitdotml · 2026-08-05
- AI Agent Autonomously Writes Triton Kernel, Breaking NanoGPT Speedrun Record — cong_ml · 2026-08-05