Gemini agentic video understanding: 88% fewer tokens, 66% lower cost, ~7% higher accuracy
_philschmid · x · 2026-09-02
Phil Schmid breaks down Gemini's agentic video understanding: the model iteratively navigates video timelines, decides what to watch, picks frame rates (0.1 or 10 FPS), and chooses whether it needs speech transcripts, audio, or visual frames. Long videos get up to 88% fewer tokens, 66% lower costs, and 7% higher accuracy on benchmarks.
How it works:
- Receives a lightweight URI reference (Files API or YouTube) and loads content via tool;
- Scans speech transcripts to pinpoint relevant moments before fetching frames;
- Navigates key timestamps and picks its own frame rate;
- Pulls audio tracks when acoustic cues matter.
Set processing="agentic" on video to enable; keep static for videos under 2 minutes. Available now for Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash Lite.
More from coding & agent
- LLM one-shots a Minecraft 3x3 piston door by building its own Linux test infra — Vjeux · 2026-10-18
- Dev builds Mumbai-inspired 3D manga background generator from OpenStreetMap data — CurieuxExplorer · 2026-10-18
- Solo founder builds a money-printing data service on Claude's Pi Durable harness — MicahBerkley · 2026-10-18
- Dev builds Arch-based OS with custom desktop and browser using Claude Code — Snoo-10493 · 2026-10-18
- Pi Durable went through 7 rewrites yet agent session files still lack a standard — ricklamers · 2026-10-18
- Five SaaS agent workflows fail the same way — and it's never model intelligence — usehive · 2026-10-18