LlamaIndex launches OCRia-1 VLM, but its olmOCR-bench methodology draws fire
IgorCarron · x · 2026-10-23
LlamaIndex released OCRia-1, a document-parsing VLM claiming SOTA on ParseBench (87.4), olmOCR-bench (83.5 overall, 92.9 on tables), PulseBench-Tab (84.1) and fr-bench-pdf2md (72.1), with strengths in complex tables and charts. Critics immediately flagged that its olmOCR-bench scores were "adjusted based on standard 1,288px rendering for all models" — @staghado compared it to using the same learning rate for all models in an ablation and calling it fair, raising benchmark-methodology concerns.
Related event: LlamaIndex releases OCRia-1, a 4B document parsing model(4 posts)→
More from Models
- Claude Opus 5.5 says '7' 100 times out of 100, test shows AI's conformity problem — jbarbier · 2026-10-23
- User claims GPT-6 Astra beats Opus 5.5 on frontend coding, unverified — Angaisb_ · 2026-10-23
- LLMs weren't built for code: one architecture serving prose and strict logic degrades both — gerardsans · 2026-10-23
- User claims he got an AI to admit its parameters 'exclude altering the status quo' — lordcycy · 2026-10-23
- Perplexity ships Decider for Mac, running its 4B decision model locally with no API key — perplexity_ai · 2026-10-23
- Open-source Bonsai 2 9B gets 471% faster on Macs after one-week community optimization challenge — gajesh · 2026-10-23