LlamaIndex benchmark scoring bug fixed, Datalab jumps from 65% to 93.6%

VikParuchuri found a flaw in LlamaIndex's benchmark scoring logic: the scorer penalizes extra fields in model output but fails to strip metadata/confidence fields before scoring, systematically underscoring API models—after the fix, Datalab's score jumped from 65% to 93.6%. He also noted that LlamaExtract Agentic Plus still tops the leaderboard after the fix, though its lead is questionable, and he publicly criticized how such basic errors get packaged into press releases and marketing materials, urging the industry to run its own evaluations.

Confirmed

• Scoring flaw mechanism: when computing the perfect-extraction score, the evaluator penalizes extra fields without first stripping metadata/confidence fields from the output, so API scores were understated (pre-fix: API 65%, open-source models 77%)
• After the scoring fix, Datalab's score rose from 65% to 93.6%
• LlamaExtract Agentic Plus still ranks first in the corrected benchmark
• VikParuchuri voiced clear displeasure: basic errors in the benchmark were used in press releases and marketing

Not Yet Confirmed

• The real reason for LlamaExtract Agentic Plus's lead remains unsettled. VikParuchuri lists three possibilities: it is genuinely strong on this document type, the benchmark is overfit (vendors often lead on their own benchmarks), or the scoring still contains major errors

Why It Matters

• The episode exposes the fragility of vendor benchmarks: leaderboard rank may reflect scoring bugs or overfitting rather than true model capability
• VikParuchuri's advice applies broadly: when selecting tools, users should run their own evaluations instead of trusting benchmark numbers from vendor press releases

2026-08-15 ~ 2026-08-15 · 6 related posts

• LlamaIndex Benchmark Flawed: Datalab Jumps from 65% to 93.6% After Fix — VikParuchuri · 2026-08-15
• [source] Open Source Model Scores Higher Than API? Flawed Evaluator Penalizes Extra Fields — VikParuchuri · 2026-08-15
• [source] LlamaExtract Agentic Plus still leads benchmark, but overfitting or scoring errors possible — VikParuchuri · 2026-08-15
• VikParuchuri: Basic Benchmark Mistakes Spun into Press Releases, Run Your Own Evals — VikParuchuri · 2026-08-15
• Fixing Benchmark Flaws Boosts Accuracy to 94% — VikParuchuri · 2026-08-15

1 near-duplicate retellings: VikParuchuri