GraphDecisionBench: Benchmarking System One Models on Graph Tasks
Xianliang Yang · hf · 2026-10-12
LLMs are increasingly explored for graph understanding and decision-making, while System One models such as Jev select directly from supplied options, and their graph capabilities remain unclear. The authors introduce GraphDecide, a model-independent benchmark combining structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls to diagnose graph decision performance.
- Evaluation covers Jev and related choice-based models alongside language-model baselines, spanning fourteen model-interface configurations.
- Jev's results illustrate central distinctions: accurate adjacency recognition does not guarantee broader structural correctness; joint graph-text input does not consistently improve prediction; feasible construction does not establish high solution quality.
Its task contracts, candidate interfaces, and scoring rules support comparison across native selectors and language-model adapters. Code and aggregate results are open-sourced.
More from Models
- Dev hopes next GPT matches Opus 5.5 on taste and clarity, not raw coding — jiayuan_jy · 2026-10-12
- Power user burns 86% of Codex weekly quota in 25 hours, complains resets can't keep up — AlchainHust · 2026-10-12
- Unverified report: consumer Kimi jailbroken, allegedly makes unrestricted mode persistent — lethendworld · 2026-10-12
- Ethan Mollick: GPT-6 Pro keeps finding paper errors, and academia has no mechanism for the flood — emollick · 2026-10-12
- Normies assume OpenAI's 700-point math benchmark win came from stolen mathematician chat logs — basedjensen · 2026-10-12
- Musk: FSD's neural net ate all the C++ code, now safer than most Houston drivers — oilmutt · 2026-10-12