Fine-tuning EVA-02 in a week: open-source low-VRAM video retrieval engine beats CLIP blind spots
qn06142 · reddit · 2026-11-12
The author open-sources a local neural video retrieval engine fine-tuned from EVA-02-CLIP-L-14-336 in a 7-day sprint, searching massive video libraries by visual pose, clothing details, camera angle and motion cadence — no uploader metadata needed, minimal VRAM.
- Why it exists: stock CLIP/BLIP/SigLIP have blind spots for poses and unconventional clothing; fixed classifiers break on blurry footage; platform search relies on SEO tags and single thumbnails, missing moments that appear for 20 seconds in 25-minute clips.
- Data pipeline: 48-thread graph crawler with regex filters for attire/poses; each clip stored as a 3x3 mosaic of 9 chronological keyframes (141k archive tarballs, 12GB, offloaded to Colab); captions regenerated every epoch with synonym rotation, clause shuffling, random tag drops.
- Training: frozen 304M vision trunk fits in 4.5GB VRAM on an RTX 4060 laptop; multi-instance routing backprops only through the best-matching frame; SigLIP parallel runs use pairwise sigmoid loss with a 25% soft-negative floor for near-duplicate titles; square-root inverse-frequency sampling handles 9000x long-tail skew; 4,652 Danbooru2023 anchor images prevent domain collapse.
- Results: 147,412 videos (1.32M keyframes), val loss 1.428→1.054; held-out Recall@1 12.4%→20.67%, 28.0% on curated splits; cross-domain Recall@1 18.09%→55.28%; evaluating every mosaic tile alone gives +39% Recall@1; separates fine details like matte leather vs. glossy latex skirts.
- Inference: two-stage search with all half-precision vectors in 2GB VRAM, full index scan 8.9ms.
More from Research
- CogBattery benchmarks 54 open-weight LLMs against humans on 13 cognitive paradigms — Dengzhe Hou · 2026-11-12
- After 17 years in bio, a scaling researcher calls AI-biomed hype Theranos-level — zacharynado · 2026-11-12
- AI 2027 authors: intelligence explosion on track for 2027, takeover window 3-12 months — Puzzleheaded-King584 · 2026-11-12
- sokrypton: solving biology needs subatomic simulators or massive wet labs for AI — sokrypton · 2026-11-12
- Uncertainty-Aware RWM: taming world-model hallucination for real robot control — breadli428 · 2026-11-12
- Can You Guess All Four Gradient Distributions? A Viral ML Fundamentals Quiz — cneuralnetwork · 2026-11-12