I tested Jev's new image input on 48 COCO photos — 408 calls, key gotchas
SeveralSeat2176 · reddit · 2026-10-16
The author ran 408 calls across 48 labeled COCO val2017 photos to test Jev's (TypeSafe's typed-probability model) new 1.14 image input and Box question.
Core results (1.13 / 1.14)
- Yes/no accuracy 98.4% → 99.1%, Brier score 0.0105 → 0.0088
- Size-level accuracy 60.4% → 80.2%; Box IoU mean/median 0.85/0.90, 93.8% of boxes hit IoU ≥ 0.5
- 0.88M input tokens for $0.04 across 408 calls; median latency 848–950ms
Key findings
- Box never says "not found": it hallucinates boxes for absent objects (a quarter land on wrong objects); pairing each Box with a yes/no presence check rejected all 96 absent-object boxes
- Wording beats thresholds: "Is there a horse?" was uncertain, but "living horse" scored 0.03 and "sign of a horse" 0.99 on an ambiguous photo
- Thresholds don't transfer across versions: the same router sent 6 photos to human review on 1.13 vs 20 on 1.14 — pin the version and retune
- Picture-option misses were all low-confidence (0.26–0.47), so a confidence floor catches them
Takeaways: one question per thing, pair Boxes with presence checks, set per-question thresholds from your own labeled set, pin the model version. Full write-up includes an interactive replay explorer.
More from coding & agent
- SandMatter co-founder: a convincing final answer isn't enough to ship an agent — VariousMechanic5811 · 2026-10-17
- Pest-Control Engineer Falls for OpenClaw, Buys Mac Studio Mid-Road-Trip — heyneighbor · 2026-10-17
- Consumer agents creating virtual cards on their own BIN means users lose credit card rewards — km · 2026-10-17
- He's reverse-engineering Grammarly with Perplexity's new REA skill and 10K free credits — ChrisUniverse · 2026-10-17
- The $20 Claude Pro power setup: Opus 5.5 as orchestrator, cheap models do the grunt work — dotey · 2026-10-17
- Multi-agent systems have a serious context problem and are burning tokens for nothing — alex_verem · 2026-10-17