Custom VQA benchmark under 48GB VRAM: Qwen3.8-Flash-Next leads, Qwen3.8-27B regressed
EmilPi · reddit · 2026-10-27
EmilPi built a small visual-QA benchmark that fits in 48GB VRAM + 128GB RAM, stress-testing models that each claim SOTA in their tech reports. Qwen3.8-Flash-Next currently leads.
Caveats: includes Lithuanian/Russian OCR tasks (biasing toward models strong in those), and the task count is still small.
Observations:
- A clear top-tier group of models
- Qwen3.8-27B regressed vs 3.6/3.5 on vision while gaining agentic/STEM capabilities; the 3.8 uncensored variant took further small damage
Surprises:
- Meta's Muse-Glimmer-30B scored high with excellent detail recognition (attributed to Meta's CV work like SAM3)
- GLM-5.3-Flash scored well even at low quant
- DeepSeek-V4-Flash-Vision-Exp came last, though it's DeepSeek's first truly multimodal experimental LLM
Typical errors: broken markdown output, hallucinated generic management advice; some models mistook a comet for a meteor or satellite. The author invites others to help build the benchmark on the website.
More from Models
- OpenAI Launches GPT-6.1 Terra: Computer-Use Model With 50% Cheaper API — soleio · 2026-10-27
- Together AI sits down with Meta to explain Muse Spark Open — togethercompute · 2026-10-27
- OpenAI launches budget computer-use model GPT-6.1 Terra, but why choose it over Sol? — lyraxana · 2026-10-27
- GPT-6.1 Terra hits ~60% on Agents' Last Exam at $1/task, 9x cheaper than Opus 5.5 — ChrisGPT · 2026-10-27
- ChatGPT keeps telling tech-support users to email 404 Media's Jason Koebler — petrusenko_max · 2026-10-27
- GPT-6.1 Terra called the awkward middle tier, priced at half of GPT-6.1 Sol — cedric_chee · 2026-10-27