First-hand information for everyone
Roboticists have long hoped for generalist policies—single controllers that can perform many different tasks—because they promise to shortcut the expensive process of training separate models for every new skill. But there’s a catch: policies trained from human demonstrations or offline datasets are only as good as those data. Reinforcement learning can extend or improve such policies, but naively fine‑tuning weights on new tasks causes catastrophic forgetting: the robot loses prior abilities as it learns new ones.
A new paper on arXiv, “Dream to Remember: Continual Steering of Generalist Robot Policies with World Models” (arXiv:2610.13816v1), proposes a practical middle way that combines a frozen generalist policy, a small steerable module, and a pretrained world model that can imagine past behavior. The method—called Dream to Remember, or D2R—aims to let a frozen policy learn new tasks over time while holding on to earlier skills, without keeping a large replay buffer of old data.
How it works
D2R keeps the main policy fixed and trains only a compact steering module that nudges the policy’s inputs (the paper frames this as steering the input noise of a frozen generative policy). To prevent the steering module from drifting away from earlier skills while it learns a new task, D2R uses a frozen, pretrained world model to “rehearse” previous tasks. The world model generates imagined rollouts of those earlier tasks; the steering module is trained both to perform the new task and to match its behavior on the imagined past rollouts, effectively anchoring it to prior performance.
This rehearsal is done without storing real past trajectories or collecting extra data from the environment. When the world model is accurate enough, the new task can be learned entirely in imagination (i.e., using the model’s rollouts for both new-task learning and rehearsal). When the world model is less faithful, D2R learns the new task in the real environment, while still using the model’s imagined rollouts only to protect previously learned behaviors.
Key results
The authors evaluate D2R across simulation suites and on a real robot. On CALVIN, a multi‑task manipulation benchmark, D2R improves eight tasks in sequence entirely in imagination and raises average success from 63.7% to 74.8% on the sequence they study. For comparison, a common fine‑tuning technique (LoRA-style adaptation) applied to the same frozen policy suffers severe forgetting and drops average performance to 11.7% after the same sequence.
The method also transfers to a scenario where both the policy and the world model are off-the-shelf: using a frozen generalist policy π0 and a pretrained world model (A2World), D2R learns previously unseen tasks from the LIBERO-90 suite in the real environment while preserving π0’s performance on earlier LIBERO suites that π0 had been fine‑tuned on. That experiment highlights D2R’s ability to maintain earlier skills even when the world model is not accurate enough to fully optimize new policies in imagination.
Finally, the team reports real‑robot experiments on a Franka manipulator where D2R improves three tasks in sequence entirely in imagination while retaining earlier improvements—an encouraging sign for sim‑to‑real and sample‑efficient updates.
Caveats and open questions
The method’s benefits depend strongly on the fidelity of the world model. If the model’s imagined rollouts are poor or systematically biased, rehearsal may be less effective, and the algorithm will need to collect real interactions to learn the new task (though it still uses imagination to protect prior skills). The approach also adds complexity (maintaining a world model and a steering module) and raises questions about scaling to very long task sequences, richer task distributions, and real‑world safety constraints.
Broader implications
D2R offers a pragmatic and sample‑efficient route toward continual adaptation of generalist robot policies. By rehearsing past skills in imagination rather than storing and replaying real trajectories, it reduces memory and data‑collection burdens and mitigates catastrophic forgetting. That makes it a compelling step for systems that must acquire new abilities over time while maintaining previously learned behaviors—so long as world models keep getting better and researchers remain careful about safety, robustness and domain gaps.
The results are promising but cautious: D2R is not a silver bullet, but a useful building block in ongoing efforts to make robots that can learn continuously without erasing what they already know.