Wizard lets players build and control new game worlds in real time
arXiv: 2610.40339v1
A new system called Wizard lets users design and manipulate virtual game scenes on the fly, giving fine‑grained control over characters and keeping gameplay logic consistent. Wizard treats the world as a sequence of visual proxies that encode the geometry of the scene and the skeletons of the characters. An agent observes user‑supplied reference images and a set of proxy inputs, then tells a video generator how to animate the characters and what the environment should look like. The generator produces frames in real time at 24 frames per second using only two denoising steps.
Figure 1. Includes the actor and scene proxy network that gives fine‑grained character and environment control while maintaining gameplay logic.Wu, Yuzhe, Lin, Zijun, Jin, Yeying · arXiv:2610.40339v1 · Figure 1: · CC BY 4.0
The key insight is to separate the visual rendering from the game logic. When a player loads a new character, the system retrieves motions from a shared motion library that match the character’s body shape and the intended style of movement. The chosen motions are retargeted to the character’s specific skeleton, so the animation feels natural. At the same time the agent keeps track of combat states—health, distance and orientation—so that game rules such as a character’s death are applied automatically.
Wizard’s effectiveness was shown on 50.3 hours of data from 750 different scenes and 178 distinct characters drawn from online gameplay, synthetic videos, and simulation tools. The trained model can take a previously unseen scene or character, together with reference images for appearance, and generate a matching video with zero additional training. In a set of qualitative tests the rendering remains faithful to the posed skeletons and scene geometry while respecting combat rules.
Figure 5. Illustrates Wizard’s zero‑shot generalization: reference images lead to visual proxies that drive plausible, mechanics‑consistent gameplay in unseen scenes and characters.Wu, Yuzhe, Lin, Zijun, Jin, Yeying · arXiv:2610.40339v1 · Figure 5: · CC BY 4.0
The system is not perfect. If a character has a very unusual body layout, finding a compatible motion is harder, which reduces the range of motions that can be applied. In long sequence play, small mismatches between the proxy layout and the generated video can become visible, but the overall gameplay remains coherent.
The visuals below illustrate these claims: figure 1 shows the architecture that enables fine‑grained control, and figure 5 presents examples where the model generalizes to unfamiliar scenes, characters and motions while keeping gameplay consistent.
