arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

Robots Learn Tasks from Natural Language Using a Vision‑Language Model and Model‑Predictive Control | arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
RoboticsEnglishPublished

Robots Learn Tasks from Natural Language Using a Vision‑Language Model and Model‑Predictive Control

November 17, 2026arXiv: 2611.18298v1

Scientists have introduced a new framework called Objectives as Policies (OaP) that lets a vision‑language model (VLM) write a program of goals for a model‑predictive controller (MPC). The VLM reads a natural‑language instruction and a camera image, builds a simulated copy of the real world, and produces a multi‑stage objective program. The MPC runs the program in simulation, the VLM reviews the results, and the program is refined several times before the robot tries it in the real world.

In tests on four tabletop tasks—pick‑and‑place, cup reorientation, direct push, and push with a tool—OaP achieved a 90 % cumulative success rate in simulation after up to two refinement rounds. That outperformed strong baselines such as CoRAL and OpenVLA‑OFT on the same tasks. In reality, the framework got an 80 % overall success rate across two disturbance levels (small and large perturbations). For example, the pick‑and‑place task succeeded 8 out of 10 times.

The approach still depends on how accurate the simulated twin and the physics model are. If objects occlude each other, slip in grasp, or deform, the controller has no built‑in recovery after a stage finishes. These limits mean the method works best when the simulation matches the real world closely.

Overall, OaP shows that a language model can generate reusable objective programs that guide physically grounded robot planners, opening new possibilities for semantically informed manipulation.

Source paper

Objectives as Policies: Vision-Language Model Cost Programs for Contact-Rich Manipulation

Robotics
View on arXiv