Coding Agents Train Optical Neural Networks In Situ, Lifting Accuracy from 35% to 96.8%
Stanford's Gordon Wetzstein group, together with EPFL, published "In Situ Training of Optical Neural Networks by Coding Agents" (arXiv:2611.23875), proposing a new "Optics as Code" paradigm: a coding agent directly operates optical hardware for experiments through a harness, closing the loop with an optical neural network and auto-tuning its parameters — lifting classification accuracy from 35% to 96.8% in real tests. This offers a new path past the calibration woes and sim2real gap that have long plagued the field, and is worth attention from researchers in optical computing and agents.
Confirmed
- The agent read optics textbooks and papers during pretraining and works like an expert at inference time: forming hypotheses, sending probes, observing results, and iterating.
- The method's core is "physics first, design second": with just 48 probe measurements the agent deduced the optical system's geometry; in the design phase, instead of updating millions of pixels, it wrote the phase mask as a combination of lenses and prisms, tuning only a few device parameters and drastically shrinking the search space.
- Baseline comparison: backpropagation through a calibrated digital twin reaches 96.3% but needs 67,000 calibration images and heavy GPU training; a gradient-free baseline plateaus around 52%. Optics as Code hit 96.8% with extremely few measurements — about 1/1000 of what backpropagation required — beating both baselines.
- The same harness can also train an optical image encoder, and stronger agents yield better results.
Why it matters
- Optical neural networks have been constrained by difficult calibration and the sim2real gap; this work demonstrates the feasibility of agent-driven "in-situ" experimental tuning.
- The agent only needs two interfaces — "able to set device parameters" and "able to observe device outputs" — so the paradigm generalizes to any controllable, measurable physical system, with potential impact well beyond optical computing.
2026-11-20 ~ 2026-11-21 · 7 related posts
Primary sources
- Gordon Wetzstein: coding agent in the loop boosts optical NN accuracy 35% to 96.8% — GordonWetzstein ·
- Stanford/EPFL Paper: Coding Agents Train Optical Neural Networks In Situ, Near-Backprop Accuracy — GordonWetzstein ·
- Stanford paper: coding agents train optical neural networks in situ from measurements alone — StanfordUniversity ·
- [source] Gordon Wetzstein: coding agent in the loop boosts optical NN accuracy 35% to 96.8% — GordonWetzstein · 2026-11-20
- Optics as Code: Stanford agent tunes optical neural nets from 35% to 96.8% via hardware-in-loop experiments — GordonWetzstein · 2026-11-20
- Optics as Code hits 96.8% accuracy with a fraction of the 67,000 images backprop-based twins need — GordonWetzstein · 2026-11-20
- Agent infers optics geometry from 48 probes, then tunes a few parameters instead of 100k+ pixels — GordonWetzstein · 2026-11-20
- Same harness trains optical encoders; paradigm generalizes to any controllable physical system — GordonWetzstein · 2026-11-20
- [source] Stanford/EPFL Paper: Coding Agents Train Optical Neural Networks In Situ, Near-Backprop Accuracy — GordonWetzstein · 2026-11-20
- [source] Stanford paper: coding agents train optical neural networks in situ from measurements alone — StanfordUniversity · 2026-11-21