First-hand information for everyone
VABAR (Variational Automation Benchmark for Agentic Robotics) is the first open, simulation‑based test that asks robots to solve six manipulation challenges drawn from industry, medicine, and infrastructure – tool sorting, tool hanging, gear meshing, syringe packing, cable insertion, and cable routing. The benchmark defines each task by initial and goal configurations, an asset library, and robot specifications, then introduces systematic variations in objects, arrangements, and positions to create low‑ and high‑difficulty levels.
The implementation uses the Newton physics engine and the Isaac Lab Arena, giving researchers two interchangeable backends. A reusable library of 33 reusable skills and standardized success metrics lets teams plug in their own learning or planning pipelines. The framework stays simple enough that any lab can reproduce the exact conditions without spending months on simulation tuning.
Visual 1: Overview of VABAR’s modular task specification, variation space, and dual‑backend evaluation.

Baseline tests show that the Graph‑as‑Policy framework, which composes skills into a computation graph, outperforms a state‑of‑the‑art coding agent (Claude) on almost every VABAR task‑difficulty pair. Across 30 independent instances per condition, the GaP system achieved higher success rates when run on Newton.
Despite these gains, GaP still lags the hand‑crafted human program on the Newton backend in every setting—a 70 % versus 100 % success gap on Level 1 gear meshing, for example. That remaining gap underscores that skill composition and parameterization remain significant challenges for autonomous policy generation.
Visual 4: Success rates and cycle times for GaP and Claude across VABAR tasks and difficulty levels.
| a-i. GaP [-1pt](Human w/ Newton) | a-ii. GaP [-1pt](Human w/ ILA) | b. Claude [-1pt]w/ Newton | c. GaP [-1pt]w/ Newton | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Task | Diff. | Succ. (%) | Cyc. (s) | Succ. (%) | Cyc. (s) | Succ. (%) | Cyc. (s) | Succ. (%) | Cyc. (s) |
| Tool Sorting | Level 1 | 47.1 | 421.6 | 20.0 | 555.0 | 0.0 | N/A | 26.7 | 134.2 |
| Level 2 | 33.7 | 913.8 | – | – | 0.0 | N/A | 13.3 | 232.3 | |
| Tool Hanging | Level 1 | 86.7 | 86.1 | 67.5 | 98.3 | 16.7 | 96.2 | 33.3 | 81.5 |
| Level 2 | 81.7 | 158.4 | 50.0 | 138.3 | 10.0 | 190.8 | 10.0 | 270.9 | |
| Gear Meshing | Level 1 | 100.0 | 165.9 | 87.5 | 198.9 | 0.0 | N/A | 70.0 | 156.6 |
| Level 2 | 90.0 | 235.5 | 80.0 | 381.3 | 0.0 | N/A | 60.0 | 284.1 | |
| Syringe Packing | Level 1 | 87.0 | 228.8 | 52.5 | 939.0 | 0.0 | N/A | 33.3 | 371.1 |
| Level 2 | 63.0 | 472.3 | 40.0 | 1810.0 | 0.0 | N/A | 12.1 | 652.2 | |
| Cable Insertion | Level 1 | 86.7 | 78.8 | 97.5 | 104.9 | 0.0 | N/A | 36.7 | 79.0 |
| Level 2 | 30.0 | 74.2 | 17.5 | 118.8 | 0.0 | N/A | 23.3 | 80.5 | |
| Cable Routing | Level 1 | 46.7 | 116.0 | 50.0 | 145.9 | 3.3 | 120.7 | 36.7 | 50.8 |
| Level 2 | 46.7 | 189.8 | 5.0 | 151.5 | 0.0 | N/A | 13.3 | 120.3 | |