wzenus Open-Sources Sooool: GPT-6 Sol Plays Real-Time Games Frame by Frame

On October 17, the wzenus team open-sourced Sooool, a project exploring how GPT-6 Sol can play real-time games in an "Agent as Policy" fashion: rather than code-as-a-policy, the model watches the screen frame by frame, directly outputs actions, observes the outcome, and decides the next move. It has been made to work on Touhou, rhythm games, racing games, and the arcade game Asterix.

Confirmed

• Core challenge: the environment keeps changing while the model processes images, so actions may already be stale by the time they arrive; the project therefore builds a latency-aware autoresearch loop (with accompanying latency modeling).
• Rhythm game task: Sol reads falling notes from video frames and directly outputs D/F/J/K keys with exact hit timing; it learns to plan ahead using measured latency, cover newly appearing notes before they become urgent, and keep note IDs stable to avoid double hits.
• Racing task: Sol selects speed and lane targets frame by frame from the onboard camera feed; it learns to firmly hold a lane when overtaking, avoid erratic lane changes on curves, and pick targets that remain valid after the response arrives.
• Asterix arcade task: using only screenshots, it directly selects movement commands, holding each direction until the next response arrives in real time; it learns to prioritize fleeing over chasing food, finish reachable pickups before switching targets, and choose a new direction after a pickup rather than repeating the current action. Failures become experience for the next round.

Why it matters

This project pushes large models from "generating a chunk of policy code" to "real-time frame-by-frame decision-making," directly confronting the real-world challenge of latency and dynamic environments. It offers a reusable research paradigm for general-purpose vision agents in real-time control scenarios beyond games, such as robotics and driving — and the code is open source and ready to build on.

2026-10-17 ~ 2026-10-17 · 8 related posts

• [source] Sooool: latency-aware agent plays Touhou in real time, repo and guide open-sourced — wzenus · 2026-10-17
• [source] Sooool open-sourced: GPT-6 Sol as a real-time policy, with parallel requests to beat latency — wzenus · 2026-10-17
• GPT-6 Sol plays rhythm games in real time, picking keys and tap times from frames — wzenus · 2026-10-17
• Sol picks speed and lane targets frame-by-frame from onboard camera while racing — wzenus · 2026-10-17
• Sol plays Asterix from screenshots, holding each move until its next reply arrives — wzenus · 2026-10-17
• [source] GPT-6 Sol plays arcade games from screenshots via autoresearch loop — wzenus · 2026-10-17
• Open-source Sol plays games frame-by-frame, beating LLM agent latency with 8 parallel requests — wzenus · 2026-10-17
• Sooool: coding agents that reason about real-time API latency — wzenus · 2026-10-17