Personal benchmark 'I don't know' fine-tunes open models via Tinker to test if models know when they don't know
brandon_galang · x · 2026-11-17
Developer loganzev released The Trust Table, a personal "I don't know" benchmark built with Thinking Machines' Tinker API. He fine-tuned open-source models and developed games for them to play and learn from, exploring whether models know when they don't know the answer. He cautions it's a first-of-its-kind product/research experiment, with results specific to this game, but says he learned a lot from the process.
More from Research
- Oxford Internet Institute PhD thesis on human-AI alignment dynamics passes viva — natashajaques · 2026-11-17
- Nature paper uses AI to decode regulatory programs behind glioblastoma plasticity — anshulkundaje · 2026-11-17
- Vault-LD: markdown wiki + YAML-LD knowledge graph as an open standard for agent memory — _AustinCalvert_ · 2026-11-17
- MemInStream benchmark separates real memory from mere recognition in streaming video — Zhenqi He · 2026-11-17
- Evolving Infra beats NVIDIA Dynamo on agent workloads: 4.1x lower latency, 14x lower P95 TTFT — YouJiacheng · 2026-11-17
- Datology claims 6x compute multiplier from data curation alone; 12B beats 30B baseline — pratyushmaini · 2026-11-17