TEAS benchmark: measures inference at natural lengths, reporting cost, accuracy, and energy
PontiEdoardo · x · 2026-09-01
Edoardo Ponti introduces TEAS: instead of distorting fixed-length measurements, it uses each dataset's natural lengths, handling chunked prefill, refusals, and steps that mix prefill with decode—capturing real sparse activations and bandwidth utilisation. It reports cost, accuracy, performance (user experience and system output), and energy estimates; accuracy confirms run validity and reveals available trade-offs.
More from Infra
- ClickHouse aggregates 100B rows in 0.4s with parallel replicas auto-scaling — Abhishekcur · 2026-10-18
- Auction-based LLM routing paper finds fixed token pricing wastes 10%-71% in costs — _onionesque · 2026-10-18
- Running Ling 3.1 Flash (560B MoE, 25B Active) Across 4 DGX Sparks via llama.cpp RPC — BerryBlush_Rogers691 · 2026-10-18
- SGLang Summit sponsors jump from 11 to 36 in one week — ying11231 · 2026-10-18
- Oracle CDS hits record 300bps, called first AI trade domino after Barclays sell downgrade — arthurcolle · 2026-10-18
- Agent Coding Workflow Would Cost ~$40k/Month on API Pricing, Dev Reveals — ChrisRackauckas · 2026-10-18