Testing 32 Local Models: Why F1 Scores Hide Dangerous Hallucination Traps

KitchenAmoeba4438 · reddit · 2026-08-06

The author benchmarked 32 local LLMs on a fact-extraction task using 1,001 notes to evaluate their ability to stay silent when no facts are present.

Key Findings

Deceptive F1 Scores: gemma-4-31B and gemma-4-12B have nearly identical F1 scores. However, on factless notes, the 31B model stays silent only 46% of the time and invents 180 false triples, while the 12B model abstains 70% of the time, inventing only 97.
Restraint vs. Recall: If your pipeline has a review gate, choose high recall (e.g., 31B). If output goes directly into an unchecked knowledge graph, choose restraint (e.g., granite-4.1-3b, which abstains 93% of the time and generates minimal hallucinations).
Parse Rate ≠ Accuracy: LFM2.5-230M boasts a 1.00 clean parse rate but scores a dismal 0.1309, proving it fluently generates incorrect answers.

The author advises evaluators to track abstention rates and invented-triple counts alongside F1, and to ensure test corpora include cases where the correct answer is silence.

Original post →

Testing 32 Local Models: Why F1 Scores Hide Dangerous Hallucination Traps

More from Models