AgentGarten: coupling code engines with a shared neural renderer to build worlds for evolving agents
Stefania_druga · x · 2026-10-09
New arXiv paper AgentGarten proposes a framework for building real-time interactive environments for evolving agents:
- Architecture: simulators/game engines maintain persistent world state and execute program-defined rules, while a shared neural renderer turns exported geometry conditions (depth, normals) into what the agent sees, block by block, in real time. A new world is just a new program — the renderer stays the same.
- Renderer: a pretrained video model is adapted to geometry conditions and distilled with Adversarial Forcing, which makes history prefilling differentiable through exact replay so losses on later predictions update how prior observations are encoded, plus real-data adversarial supervision for visual quality; inference is optimized for real-time interaction.
- Agents perceive via visual observations, interact in real time, and distill each round of experience into playbooks inherited and refined by subsequent agents.
Paper, code, and project page are all public.
Related event: AgentGarten: Neural Rendering Powers Interactive Worlds for Evolving Agents(3 posts)→
More from coding & agent
- Voyager launches as 'Codex for creative work', driving Blender, Resolve and After Effects — rohanpaul_ai · 2026-10-09
- PartyKit founder shuts down free hosted platform 2.5 years after Cloudflare acquisition — threepointone · 2026-10-09
- New MCP server books hotels across DACH region via AI assistants — bookwithbluerails · 2026-10-09
- world-intel-mcp: open-source MCP server with 132 tools for real-time global intelligence — tom_doerr · 2026-10-09
- nanobot: open-source self-hosted agent framework, 99% smaller with 48.9k GitHub stars — mdancho84 · 2026-10-09
- UCL's Memento 3 lets a frozen LLM self-improve via rulebooks, clearing all 25 ARC-AGI-3 games — UniversityCollegeLondon · 2026-10-09