Agent-based kernel optimization faces a fidelity-throughput tradeoff: random data misleads, cold starts cost 12.7 minutes
simran_s_arora · x · 2026-11-05
Together's team identifies a fundamental tradeoff in agentic performance optimization: (1) Low-fidelity, high-throughput systems tune individual kernels on random benchmark data, but MoE, sparse top-k attention, and speculative decoding are data-dependent—random data activates far more experts than real traffic, and a top-k kernel tuned on random data degrades from 16µs to 62ms on real traffic. (2) High-fidelity, low-throughput systems relaunch the engine per experiment, costing 12.7 minutes of cold start (265s model loading, 226s CUDA-graph capture, 192s autotuning, 73s KV-cache sizing). The Together Compiler's persistent engines target exactly this gap.
More from coding & agent
- Dev finds pi coding tool defaults to Opus 4.8 on a fresh install — lucasmeijer · 2026-11-05
- Kylon puts whole teams and agents in shared rooms, with a Neon branch per app draft — jefrankle · 2026-11-05
- "Evals > vibes": agent workflow debates are just guessing without eval stats — annetgriffin · 2026-11-05
- OpenAI hackathon winner joins YC, launches Raven, an agent-first phone — ycombinator · 2026-11-05
- Open-source Cicada brings real-time translation and AI meeting notes to any Mac call, free 50h/month — dotey · 2026-11-05
- IBM launches AI coding assistant named "Bob", devs can't stop laughing — zeeg · 2026-11-05