First-hand information for everyone
Scientists have introduced a new framework called Objectives as Policies (OaP) that lets a vision‑language model (VLM) write a program of goals for a model‑predictive controller (MPC). The VLM reads a natural‑language instruction and a camera image, builds a simulated copy of the real world, and produces a multi‑stage objective program. The MPC runs the program in simulation, the VLM reviews the results, and the program is refined several times before the robot tries it in the real world.
In tests on four tabletop tasks—pick‑and‑place, cup reorientation, direct push, and push with a tool—OaP achieved a 90 % cumulative success rate in simulation after up to two refinement rounds. That outperformed strong baselines such as CoRAL and OpenVLA‑OFT on the same tasks. In reality, the framework got an 80 % overall success rate across two disturbance levels (small and large perturbations). For example, the pick‑and‑place task succeeded 8 out of 10 times.
The approach still depends on how accurate the simulated twin and the physics model are. If objects occlude each other, slip in grasp, or deform, the controller has no built‑in recovery after a stage finishes. These limits mean the method works best when the simulation matches the real world closely.
Overall, OaP shows that a language model can generate reusable objective programs that guide physically grounded robot planners, opening new possibilities for semantically informed manipulation.