Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics
Zhixi Zhu, Kristina Gligoric
cs.CV, cs.CL, cs.SI
2026-10-01
A 16-frame image grid nears full-video VLM understanding at 15% token cost, yet the top-accuracy config can flip downstream coefficients; PPI correction restores valid inference.
Annotating video with VLMs burns tokens in proportion to frame count: a 60-second clip at one frame per second runs past two million tokens. The standard fixes are heuristics: sample a subset of frames, pack them into an image grid, or keep a single modality. Nobody had compared them systematically on the two questions that matter, which ones actually save money, and whether the downstream conclusions built on those annotations survive.
The second question bites hardest in computational social science, where annotations feed regressions and frequency estimates and the goal is correct conclusions, not classification accuracy. Prior evaluations of compact video representations stopped at video QA accuracy, so the accuracy-versus-validity question stayed open.
A Johns Hopkins team ran a grid of experiments on two tasks: sentiment classification on MELD (900 Friends dialogue clips) and 18-way topic classification on TikTok (900 videos, median 46s). Three switches varied:
Downstream sits a prediction-powered inference (PPI) layer: a small gold set of human labels (about 100-200) estimates the bias in model predictions, which corrects the statistics computed over the full dataset and yields valid confidence intervals. GPT-4o-mini is the main model; Claude Sonnet 4.6, GPT-5.1, and Grok 4.3 show the same trends.
On accuracy, the two datasets disagree about modality:
| Config | MELD | TikTok |
| Text-only | .610 (best overall) | .713 |
| Best multimodal grid | .589 (ShotT 4x4) | .834 (Uni 3x3) |
| Frames-only | .442 | .712 |
| Random forest (embeddings) | .507-.560 | .717-.718 |
On the validity axis the ranking collapses. On MELD, the highest-accuracy config (text-only) flips the sign of the Negative-sentiment coefficient (β=-.43, p=.038, against a ground truth of +.27) and misses the real Neutral effect; the random forest finds a spurious significant effect on Positive. On TikTok, raw LLM predictions overestimate the funnyvideos frequency by 5.7 points (10.7% vs. 5.0%); with 100 gold labels, PPI pulls all 18 true frequencies inside their 95% confidence intervals and cuts that error to 2.0.
On cost, a 2x8 grid averages 47,665 tokens ($0.007) per video against 2,275,411 ($0.34) for 1-FPS sampling, roughly 50x (over 500x in the worst case), and the grid's cost is capped per image regardless of duration. Holding Gemini fixed, one ShotT 2x8 grid costs 2,332 tokens per video versus 15,774 for native full-video input, or 14.8%, with accuracy .870 and κ within .05 of full-video understanding. The two cost curves cross at about 9 seconds; shorter videos gain nothing from the grid.
This is an evaluation paper: grids, shot detection, and PPI are all off-the-shelf. The contribution is putting accuracy, validity, and cost on one table, plus a copy-ready procedure. Pilot 50 clips per modality and drop channels that add no signal; dialogue sentiment needs no frames. Use shot-transition sampling when the frame budget is tight, uniform otherwise. Start with the smallest grid with acceptable κ. Reserve 100-200 gold labels for PPI rather than trusting raw model labels in statistics.
The warning that generalizes beyond CSS: selecting configurations by accuracy can pick exactly the one that biases your inference.
The authors' own list: two datasets of 900 videos each, narrow label spaces, single-label only. The gold labels are themselves noisy, and TikTok hashtags reflect creator intent rather than verified content, so PPI's guarantee holds relative to that reference. The cost-decoupling result rides on the current per-image token cap; a pricing change invalidates the 14.8% figure.
Reading closely adds more. Shot-transition and uniform sampling differ trivially in both datasets (largest gap .018, p=.177), so the advice to choose sampling by shot density rests on thin evidence. PPI repairs the aggregate estimate, not individual predictions, and one class (homedecor) lands further from truth after correction. Everything was tested on videos up to 12 minutes; whether one grid still fits longer videos is unverified.