LlamaIndex Benchmark Scoring Bug Fixed, Datalab Jumps from 65% to 93.6%

VikParuchuri found a flaw in LlamaIndex's benchmark scoring logic: when computing perfect-extraction scores, the scorer penalized extra fields without first stripping metadata/confidence fields, systematically understating API model scores; after a two-step fix, Datalab's score jumped from 65% to 93.6%. He also noted that LlamaExtract Agentic Plus still tops the leaderboard after the fix but its lead is questionable, and publicly criticized such basic errors being packaged into press releases and marketing materials, urging users to run their own evaluations.

Confirmed

• Defect mechanism: when computing perfect-extraction scores, the scorer penalized extra fields in outputs without first stripping metadata/confidence fields, so API model scores were understated (pre-fix: API 65%, open-source models 77%)
• The fix came in two steps: first stripping metadata fields in the scorer, reaching 91%; then ensuring scorer fairness on fuzzy fields, reaching 93.6% (the post's title says 94%)
• After the fix, Datalab rose from 65% to 93.6%, and LlamaExtract Agentic Plus remains ranked first
• VikParuchuri criticized benchmarks with basic errors being used in press releases and marketing, stressing that users should run their own evaluations

Unconfirmed

• The real reason for LlamaExtract Agentic Plus's lead remains undetermined. VikParuchuri listed three possibilities: it is genuinely strong on this document type; the benchmark is overfit (vendors often lead on their own benchmarks); or scoring still contains major errors

Why it matters

• Fragility of vendor benchmarks: leaderboard rankings may reflect scoring bugs or overfitting rather than actual model capability
• Users should run their own evaluations when selecting tools instead of trusting benchmark numbers in vendor press releases

2026-08-15 ~ 2026-08-15 · 6 related posts

• LlamaIndex Benchmark Flawed: Datalab Jumps from 65% to 93.6% After Fix — VikParuchuri · 2026-08-15
• Open Source Model Scores Higher Than API? Flawed Evaluator Penalizes Extra Fields — VikParuchuri · 2026-08-15
• [source] LlamaExtract Agentic Plus still leads benchmark, but overfitting or scoring errors possible — VikParuchuri · 2026-08-15
• VikParuchuri: Basic Benchmark Mistakes Spun into Press Releases, Run Your Own Evals — VikParuchuri · 2026-08-15
• [source] Fixing Benchmark Flaws Boosts Accuracy to 94% — VikParuchuri · 2026-08-15

1 near-duplicate retellings: VikParuchuri