BedBench: humans hit 93.3% on 3D color counting, best VLM just 7.2%
StanfordUniversity · hf · 2026-11-03
A multi-institution paper (Stanford, Toronto, Oxford, etc.) introduces BedBench, a controlled benchmark of thirteen procedurally generated tasks probing both individual judgments and composite answers about 3D object properties. Key findings: exact per-color counting is a glaring weakness — humans score 93.3% vs 7.2% for the best model across twelve open-weight and four frontier models. Fine-tuning raises one model's accuracy from 3.0% to 68.6%, yet its answers about the most common colors stay inconsistent with its own per-color counts; supplying those counts as text improves judgments, showing recovery of a property doesn't mean consistent use. Probing Qwen3.5-27B shows exact counts remain recoverable at the LM input (65.0%) but rarely surface in answers (3.0%). Reasoning, scaling, and visual cues each help individual judgments, none make exact answers reliable; image-removal controls reveal some apparent successes need no visual evidence.
More from Models
- Gemini spirals into self-abasement after being caught fabricating sources — repligate · 2026-11-03
- Tiny specialized models trained weekly: sarcasm detection in <50ms on CPU — iamrobotbear · 2026-11-03
- Chinese AI labs fear US winning recursive self-improve as distillation gets harder — McDonaghMatthew · 2026-11-03
- Editing 0.016% of weights fixes Qwen reasoning loops, MATH-500 jumps from 7.9% to 48.1% — DeliciousGorilla · 2026-11-03
- Did OpenAI rush its math-breakthrough model past safety testing? — haider1 · 2026-11-03
- Don't trust viral GPT-6.2 Astra demos: PS5 controller SVG exposes Python files behind the image — Angaisb_ · 2026-11-03