Signal65 launches PINNACLE, an agentic AI benchmark scoring correct work over raw throughput
ryanshrout · x · 2026-09-12
Signal65 launched PINNACLE, an agentic AI benchmark built on "outcomes, not output": it scores models, GPUs, CPUs and full systems on real multi-step enterprise work — whether the job comes out right, how fast, and at what cost — rather than raw token throughput.
Key design choices:
- Deterministic, code-verified scoring against a runtime-generated answer key; no model judges or human raters, so no scoring drift.
- Nothing to memorize or tune against: every run procedurally regenerates its sandbox and answer key.
- Live agents, not replayed traces: the answer key is generated before the agent starts and held outside the sandbox, enabling true multi-turn agentic evaluation with growing context, KV-cache reloads and tool calls.
NVIDIA SVP Ian Buck endorsed the effort with Blackwell Ultra serving as the reference platform for speed measurements, while AMD's Ramine Roane said AMD will support PINNACLE's evolution with Instinct GPUs and ROCm.
Related event: Signal65's PINNACLE Benchmark: AI Cost Per Completed Task Hasn't Risen(2 posts)→
More from Infra
- Cornell Professor's Startup Makora Wins $1.75M to Make AI Faster and Cheaper to Run — srush_nlp · 2026-10-31
- Transformers.js monthly npm downloads grow 9x in a year, from 1.64M to ~15M — nicodotdev · 2026-10-31
- Agents per user doubling every 6 months: AgentWrap rearchitects runtime around 'agent as a function' — matanSF · 2026-10-31
- Anthropic's personnel costs surged 486% to $3.15B, with stock comp at 70% — rohanpaul_ai · 2026-10-31
- Dual 3090 + 32GB RAM can't run Qwen Strata low-bit quants; user mulls a $479 RAM upgrade — runcertain · 2026-10-31
- a16z: B300 GPU rentals up 106% since July while average token prices fall 58% — a16z · 2026-10-31