StreamDuo pairs a commentator with a wiki research agent to commentate unseen games
Zixian Gao · hf · 2026-11-04
A paper from Alaya Lab, University of Tokyo, Georgia Tech and others tackles cross-game streaming commentary: streaming video LLMs trained on one game transfer their timing to unseen games but not their game-world knowledge.
- Method: StreamDuo pairs a trainable commentator with a frozen research agent that reads the wiki of the game being played. When the commentator can't name something, it sends current frames and gets back wiki entries 2 seconds later while the stream continues.
- Training: only the commentator is trained, on a single game (Baldur's Gate 3), in two stages. Information-gain labels teach it to ask only when an answer helps; on-policy self-distillation from a copy that sees the ideal answer teaches it to use noisy answers.
- Results: tested on ten unseen games, entity recall rises from 0.082 to 0.116 over Proact-VL trained on the same data, source-game leakage drops from 9.9% to 1.0%, and it achieves the best timing, liveliness and overall judged quality among streaming models.
Code and project page are open-sourced.
More from Research
- 20-40% of alignment data is noisy: a loss-reduction scoring framework to fix it — burkov · 2026-11-04
- Scientists image quantum entangled photons in real time, and the result looks like a taijitu — CurieuxExplorer · 2026-11-04
- kalomaze: crude diffusion autoencoders may lose low-frequency info at the latent stage — kalomaze · 2026-11-04
- Structured captions boost text-to-music alignment, new Music DSG metric beats CLAP — affige_yang · 2026-11-04
- Fruit fly's 166,000-neuron circuit map is being used to drive robot behavior — CurieuxExplorer · 2026-11-04
- Orbital engineers use AI OS CurieOS to draft two-phase cold plate designs — TimothyDuignan · 2026-11-04