Researchers Find Budget Open Models Struggle to Detect Null Findings, Newer Models Show Promise

Researchers jonmellon and RexDouglass shared hands-on lessons on X from using LLMs to classify academic abstracts: the task was to determine whether a paper's abstract reports a null finding, with the benchmark built on well-polished annotation instructions and labels hand-coded by multiple annotators. The results: most cheap models performed mediocrely on this task, while one new model showed promise for the first time.

Confirmed

• RexDouglass defined the task: determining whether an academic abstract claims a null finding, with the evaluation grounded in reliable human-coded labels and mature instructions as the benchmark
• jonmellon confirmed the Qwen model failed at this task, and Llama oss and Gemma were similarly mediocre
• The team tried many parameter combinations across open-weight models without finding a solution that was both reasonably effective and practical for actually processing 20 million records; jonmellon suspects the optimal configuration simply hasn't been found yet
• jonmellon said the new model looks very promising on the academic questions they tried, though he was not in a position to judge its performance on general tasks

Why it matters

• This case shows that even a seemingly narrow text-classification task (RexDouglass originally assumed narrow tasks should be doable) can defeat cheap open models, and the reliability of LLM annotation must be empirically verified per task
• At the scale of 20 million records, balancing effectiveness and cost is a real bottleneck that directly constrains the feasibility of large-scale academic text coding
• The new model's strong initial showing on this task suggests model iterations are improving such academic applications, but the conclusion is limited to this specific task and should not be extrapolated

2026-09-19 ~ 2026-09-19 · 9 related posts

• [source] Researchers: cheap LLMs all middling at detecting null findings, new model promising — jon_mellon · 2026-09-19
• New model 'very promising' on academic problems, researcher reports — RexDouglass · 2026-09-19
• [source] Researchers rely on tested instructions and human-coded labels for LLM annotation — jon_mellon · 2026-09-19
• Task is detecting null-finding claims, tested against multi-human-coded labels — RexDouglass · 2026-09-19
• Cheap LLMs all middling at narrow academic task, not trivial after all — jon_mellon · 2026-09-19
• Cheap LLMs middling at academic null-finding detection, researchers say — RexDouglass · 2026-09-19
• Qwen, Llama oss and Gemma all fall short on academic abstract null-finding task — jon_mellon · 2026-09-19
• None of Qwen, oss or Gemma passed researchers' academic classification test — RexDouglass · 2026-09-19
• [source] Researchers tried many open-weight setups, none accurate and feasible over 20M cases — jon_mellon · 2026-09-19