$40 experiment: Opus 5 hits ~0.94 F1 on ExtractBench, Qwen3.8 matches at 1/3 the price
Ok-Challenge-7810 · reddit · 2026-08-25
An AI engineer with a computational physics background ran a $40 experiment on LlamaIndex's ExtractBench (schema-guided document extraction): 36 government docs, customs forms and municipal audits, text pulled with pypdf, one-shot extraction with Claude Opus 5, Qwen3.8-2.4T and Qwen3.6-35B via OpenRouter, scored with the benchmark's cell-level F1.
Two surprises: Opus 5 one-shot on plain text scored 0.94, on par with the coding agents in the paper at a fraction of the per-page cost; and Qwen3.8 basically matched it (0.936) at a third of the price. F1 also correlated much more with document length (ρ ≈ −0.50) than with the number of values to extract (ρ ≈ −0.28), fitting the recent pre-inference routing paper (arXiv 2608.06607).
Caveats: n=36, short/medium docs only, text-layer only (scanned forms excluded), and the scorer is a reimplementation, so absolute numbers may drift from the leaderboard. Notebook is public.
More from Models
- Startup's neural-operator model may be source of huge-context-window rumors — McDonaghMatthew · 2026-08-25
- Zach Mueller Polls Users on Adding NVFP4 Weights and Base Models to Lambda Cards — TheZachMueller · 2026-08-25
- Qwen 3.8 Flash Next rumored: 125B + 51B N-gram lookup tables, launches tomorrow — teortaxesTex · 2026-08-25
- Ex-NVIDIA scientist's startup launches physics AI model handling 5T data points per prompt — thesaraharminta · 2026-08-25
- OpenAI blog: Codex brought three open-weight models to high performance in two months — eliebakouch · 2026-08-25
- User Wishes Anthropic Would Cut Prices and Release Full Mythos at $80 — scaling01 · 2026-08-25