The core dilemma of agentic benchmarks: web access gives realism but kills replicability
davidmanheim · x · 2026-11-20
davidmanheim highlights a fundamental tradeoff in agentic benchmark design: realism vs. reliability. An agentic system with web access faces a constantly changing internet, making results non-replicable — but without web access, the benchmark loses realism and no longer reflects real-world conditions. This tension sits at the heart of how agent evaluations should be built.
More from Research
- New 79-minute review maps deep learning for gene regulation across cellular contexts, from maxATAC to Bolero — anshulkundaje · 2026-11-20
- Stanford/EPFL Paper: Coding Agents Train Optical Neural Networks In Situ, Near-Backprop Accuracy — GordonWetzstein · 2026-11-20
- Agent infers optics geometry from 48 probes, then tunes a few parameters instead of 100k+ pixels — GordonWetzstein · 2026-11-20
- Same harness trains optical encoders; paradigm generalizes to any controllable physical system — GordonWetzstein · 2026-11-20
- Optics as Code: Stanford agent tunes optical neural nets from 35% to 96.8% via hardware-in-loop experiments — GordonWetzstein · 2026-11-20
- Gordon Wetzstein: coding agent in the loop boosts optical NN accuracy 35% to 96.8% — GordonWetzstein · 2026-11-20