Gemini agentic video understanding: 88% fewer tokens, 66% lower cost, ~7% higher accuracy
_philschmid · x · 2026-09-02
Phil Schmid breaks down Gemini's agentic video understanding: the model iteratively navigates video timelines, decides what to watch, picks frame rates (0.1 or 10 FPS), and chooses whether it needs speech transcripts, audio, or visual frames. Long videos get up to 88% fewer tokens, 66% lower costs, and 7% higher accuracy on benchmarks.
How it works:
- Receives a lightweight URI reference (Files API or YouTube) and loads content via tool;
- Scans speech transcripts to pinpoint relevant moments before fetching frames;
- Navigates key timestamps and picks its own frame rate;
- Pulls audio tracks when acoustic cues matter.
Set processing="agentic" on video to enable; keep static for videos under 2 minutes. Available now for Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash Lite.
More from coding & agent
- Rust Glancer: A Rust language server built to use 100x less RAM than rust-analyzer — JeremyCMorgan · 2026-09-02
- Doberman: An execution gateway to prevent agent disasters — Da_Lil_Fu · 2026-09-02
- Redditor pitches an AI agent as property developer: prefab homes could cut 30% off house prices — epSos-DE · 2026-09-02
- CommerceAgentBench hits 1K stars: 107 e-commerce agent tasks distilled from 1.6M real conversations — VibeMarketer_ · 2026-09-02
- Fable 5.1 ships with Anthropic's statistical watermark on every platform plus cache-friendly API updates — Angaisb_ · 2026-09-02
- Tips: Long Sessions Waste Tokens and Degrade Model Performance — hudsonhateno · 2026-09-02