LlamaIndex's Boo Bench scares doc parsers into repetition loops
llama_index · x · 2026-10-30
LlamaIndex released Boo Bench, a 179-page 'haunted house' benchmark designed to push document parsers into repetition loops. One model saw 246 redaction x's on a page and wrote 23,004.
Composition:
- 100 real pages: dot leaders, blanks, rows of 0.00, repeated lines
- 48 trap pages: 16 tricks, from near-identical rows to O vs 0 confusion
- 16 ghost pages: grids and tallies of Halloween emoji
- 15 haunted scripts: scans in scripts models rarely read
Positioning: captures pages that scare VLMs but that specialized document OCR APIs handle well — a litmus test for why document processing needs specialized models, tools, and agent harnesses rather than straight-shotting documents to VLMs. Evaluation checks whether output repeats anything beyond what's on the page; leaderboard, paper, and dataset are available.
Related event: LlamaIndex Launches Boo Bench to Expose Parser Repetition Loops(2 posts)→
More from Research
- AI auto-optimizes 11 bioinformatics packages for read alignment, with tests — jmschreiber91 · 2026-10-30
- Carlini found image poisoning actually improved fine-tuning results — matdryhurst · 2026-10-30
- Stanford's Kundaje questions DNA LMs: simpler methods suffice for promoter design — anshulkundaje · 2026-10-30
- Artificial Analysis launches Progress Index, a single capability scale for every LLM since 2022 — scaling01 · 2026-10-30
- Progress Index: Frontier AI Capability Climbed From 125 to 325 in Four Years — ArtificialAnlys · 2026-10-30
- RL can train design taste into models — but the taste belongs to the judge, not you — simonguozirui · 2026-10-30