Dev launches Local AI benchmark scoring models by hardware on agent tasks and speed
draginol · x · 2026-11-08
draginol released a Local AI benchmark with a companion website that measures local inference by model and hardware. It offers three tiers — Simple (80 labeled classifications, 1-4 min), Normal (turn a sales CSV into a dashboard, 3-6 min), and Extended (multi-step agent tasks in a virtual workspace, 15-20 min). Tests include citing evidence from notes, fixing JavaScript against hidden tests, and recovering from an injected tool fault; scoring covers agent tasks passed, prefill, decode speed, first token, and peak memory. Any OpenAI-compatible endpoint works, engines are auto-detected, and results use the median of 3 runs. Extended readiness requires 4-of-5 tasks at 15 tok/s decode.
More from coding & agent
- Open-source satmux boosts inflight GEO Wi-Fi up to 6.4x with $0.03/h VM relay — steipete · 2026-11-08
- IV Studio 1.0: Open-Source ComfyUI Film Studio Adds Linux, Mac and Local LoRA Training — itsVariance · 2026-11-08
- Dev builds research wiki for agent swarms, following Anthropic's wiki memory notes — teortaxesTex · 2026-11-08
- Nadella: Business apps will collapse in the agent era, becoming dumb databases — _AustinCalvert_ · 2026-11-08
- Reddit dev asks how to build an LLM-agnostic agent stack and escape Claude lock-in — DataBeek · 2026-11-08
- RL env startups urged to focus on finance; Anthropic ships Claude for Financial Services — himanshustwts · 2026-11-08