arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
RoboticsEnglishPublished

New VABAR Benchmark Maps Six Real‑World Manipulation Tasks for Robots

November 6, 2026arXiv: 2611.06881v1

VABAR (Variational Automation Benchmark for Agentic Robotics) is the first open, simulation‑based test that asks robots to solve six manipulation challenges drawn from industry, medicine, and infrastructure – tool sorting, tool hanging, gear meshing, syringe packing, cable insertion, and cable routing. The benchmark defines each task by initial and goal configurations, an asset library, and robot specifications, then introduces systematic variations in objects, arrangements, and positions to create low‑ and high‑difficulty levels.

The implementation uses the Newton physics engine and the Isaac Lab Arena, giving researchers two interchangeable backends. A reusable library of 33 reusable skills and standardized success metrics lets teams plug in their own learning or planning pipelines. The framework stays simple enough that any lab can reproduce the exact conditions without spending months on simulation tuning.

Visual 1: Overview of VABAR’s modular task specification, variation space, and dual‑backend evaluation.

Diagram showing the VABAR framework with six task classes, a library of reusable skills, systematic task variations, and two interchangeable simulation backends – Newton and Isaac Lab Arena.
Overview of VABAR’s modular task specification, variation space, and dual‑backend evaluation.
Fig. 2.
Xie, Shuangyu, Chen, Kaiyuan, Yoo, Uksang, Zhu, Xinyue, Shen, Meadow, Drake, Jaimyn et al. · arXiv:2611.06881v1 · Fig. 2: · CC BY 4.0

Baseline tests show that the Graph‑as‑Policy framework, which composes skills into a computation graph, outperforms a state‑of‑the‑art coding agent (Claude) on almost every VABAR task‑difficulty pair. Across 30 independent instances per condition, the GaP system achieved higher success rates when run on Newton.

Despite these gains, GaP still lags the hand‑crafted human program on the Newton backend in every setting—a 70 % versus 100 % success gap on Level 1 gear meshing, for example. That remaining gap underscores that skill composition and parameterization remain significant challenges for autonomous policy generation.

Visual 4: Success rates and cycle times for GaP and Claude across VABAR tasks and difficulty levels.

Table summarizing success percentages and cycle times for several VABAR tasks, comparing the Graph‑as‑Policy baseline to the Claude coding agent on both Newton and Isaac Lab backends.
a-i. GaP [-1pt](Human w/ Newton)a-ii. GaP [-1pt](Human w/ ILA)b. Claude [-1pt]w/ Newtonc. GaP [-1pt]w/ Newton
TaskDiff.Succ. (%)Cyc. (s)Succ. (%)Cyc. (s)Succ. (%)Cyc. (s)Succ. (%)Cyc. (s)
Tool SortingLevel 147.1421.620.0555.00.0N/A26.7134.2
Level 233.7913.8––0.0N/A13.3232.3
Tool HangingLevel 186.786.167.598.316.796.233.381.5
Level 281.7158.450.0138.310.0190.810.0270.9
Gear MeshingLevel 1100.0165.987.5198.90.0N/A70.0156.6
Level 290.0235.580.0381.30.0N/A60.0284.1
Syringe PackingLevel 187.0228.852.5939.00.0N/A33.3371.1
Level 263.0472.340.01810.00.0N/A12.1652.2
Cable InsertionLevel 186.778.897.5104.90.0N/A36.779.0
Level 230.074.217.5118.80.0N/A23.380.5
Cable RoutingLevel 146.7116.050.0145.93.3120.736.750.8
Level 246.7189.85.0151.50.0N/A13.3120.3
TABLE II. Success rates and cycle times for Graph‑as‑Policy and Claude agents across VABAR tasks and difficulty levels.Xie, Shuangyu, Chen, Kaiyuan, Yoo, Uksang, Zhu, Xinyue, Shen, Meadow, Drake, Jaimyn et al. · arXiv:2611.06881v1 · TABLE II: · CC BY 4.0

Source paper

VABAR: A Variational Automation Benchmark for Agentic Robotics

Robotics
View on arXiv
New VABAR Benchmark Maps Six Real‑World Manipulation Tasks for Robots | arXiv News