arXiv:2610.12374 · Paper published 2026-10-08 · Original paper ↗
AgentGarten couples physics engines to a shared neural renderer through untextured 3D geometry maps. This separation lets AI agents interact with photorealistic virtual worlds in real time while keeping physical rules and object coordinates explicitly controllable in code.
Traditional physics engines offer explicit state coordinates and enforceable rules, but creating detailed 3D visual assets for new scenes requires costly manual labor. Generative video world models synthesize realistic observations directly from data, but conceal spatial state and physical rules inside neural network weights. Consequently, developers cannot directly inspect object coordinates, modify interaction rules, or guarantee consistent physical dynamics.
How it works
Update physical scene state The simulation engine receives an agent action, updates object coordinates via program code, and outputs new state positions. Because state updates do not consume rendered pixels, visual rendering errors cannot directly overwrite simulator coordinates.
Export 3D geometry maps The engine converts scene state into untextured 3D geometry outlines from the viewpoint. These include depth maps (measuring distance to surfaces) and surface normal maps (indicating 3D surface orientations).
Synthesize visual observation A neural renderer combines the geometry map, a static reference photo, and past image frames to synthesize a photorealistic first-person view. Operating via block-causal generation, it outputs frames at over 35 frames per second on a single GPU.
Observe and select action The agent inspects the rendered frame, decides its next command, and submits it to the engine. While rendering glitches cannot directly alter simulator state, a misleading image can indirectly prompt the agent to choose an improper action, altering subsequent physical trajectories.
Illustrative example
Illustrative scenario: Imagine an agent driving a car across a narrow bridge. The physics engine updates physical coordinates s_{t+1} = f_p(s_t, a_t) and exports an untextured surface normal map showing 3D surface orientations. The neural renderer combines this normal map with a reference photo to synthesize a windshield view. The bridge engine explicitly stores numerical spatial coordinates, whereas a pure video world model holds spatial state implicitly in generated pixel tokens. If the rendered view contains a visual artifact like a blurry railing, the car's stored physical position remains uncorrupted, though a misleading blur could indirectly prompt the driver to steer improperly.
Evidence
Reported evidence and its scope
Reported result
Test conditions
What it supports
Adversarial Forcing maintained natural textures across 30-second rollouts, whereas Self Forcing developed repetitive surface patterns over time.
Evaluated on identical 30-second camera trajectories at 480x832 resolution on an NVIDIA H100 GPU.
Supports that combining real-data adversarial loss with exact history-gradient replay prevents visual drift during long rollouts; does not measure task success rates.
Foundation model agents perceiving only neural-rendered images built shelters by round 4 and used ramps by round 10.
Evaluated in a hide-and-seek physics world where agents act via Python scripts and reflect using Markdown playbooks, compared to a 2019 reinforcement learning baseline requiring 25 million and 100 million episodes respectively.
Demonstrates that foundation model agents can rapidly ground web-pretrained knowledge into physical tactics when provided persistent state and written playbooks; this reflects distinct learning paradigms rather than a direct sample-efficiency ratio.
Limits
The neural renderer relies strictly on exported geometry maps; non-geometric properties—such as a wheel spinning in place—cannot be rendered if unrepresented in depth or normal maps.
Visual attributes like color and material texture are maintained through visual history, but can fade or erode during long object occlusions.
While rendering glitches do not directly overwrite simulator coordinates, a distorted frame can mislead an agent into choosing a flawed action, indirectly altering the physical trajectory.
Terms used here
Code World
A virtual environment that pairs an executable simulation engine maintaining physical state with a shared neural renderer that turns 3D geometry maps into realistic visual images.
Adversarial Forcing
A training method that distills a video model into a real-time renderer on its own generated outputs using distribution matching, bitwise-identical history replay, and real-data adversarial supervision.
Playbook
A persistent text document where an AI agent writes failure diagnoses and strategic lessons after an episode for future agent instances to inherit.
Visual summarySource and authors
Paper authors: Jiawei Chi, Shangchen Miao, Zhiyuan Shi, Kailu Wu, Hanyang Wang, Weiliang Chen, Qiyu Dai, Jinshan Ren, Jun Gao, Mingsheng Long, Yueqi Duan, Jiangran Lyu, Jialong Wu, Fangfu Liu
arXiv:2610.12374 ↗ · Episode released 2026-10-13 · Explanation updated 2026-10-13
RealCompanion introduces a longitudinal benchmark derived from 27,218 real-world human-companion messages across up to 120 days, enabling systematic evaluation of an AI companion's selective memory retrieval, profile fact extraction, trait inferencing, and restraint when history is unnecessary.
2026-10-01 · Evaluation and benchmarks · AI agents · Language models
By measuring student learning progress through divergence from initialization, this paper fits joint power laws that predict a student language model's peak test accuracy and learning speed from model scale and teacher accuracy before distillation begins.
2026-09-26 · Language models · Reinforcement learning · Efficient AI
Raven packages language models with domain tools into reusable worker units, using a central host agent to break user requests into workflow graphs, validate file-based handoffs before execution, and run subtasks in parallel.
2026-09-27 · AI agents · Evaluation and benchmarks · Language models
papers.luk.media