arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

RoboARCade test shows GPT‑6 Astra outperforms the evaluated large‑language‑model agents on 22 long‑horizon tabletop manipulation tasks | arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
RoboticsEnglishPublished

RoboARCade test shows GPT‑6 Astra outperforms the evaluated large‑language‑model agents on 22 long‑horizon tabletop manipulation tasks

November 9, 2026arXiv: 2611.08209v1

A new benchmark called RoboARCade evaluates robot control when agents act directly as end‑to‑end policies, a mode called agentic robot control (ARC). The suite contains 22 tasks that combine precise manipulation, memory, planning, and bimanual work, with a wide range of initial conditions and high‑fidelity visual rendering.

Illustrations of diverse RoboARCade tabletop manipulation tasks
Figure 1. Figure 1 shows a representative sample of the 22 RoboARCade tasks, illustrating the range of manipulation, planning, and bimanual challenges covered by the benchmark.Sharma, Satvik, Hu, Hengyuan, Kamat, Mihika, Sadigh, Dorsa, Bohg, Jeannette · arXiv:2611.08209v1 · Figure 1: · CC BY 4.0

RoboARCade supplies two interchangeable interfaces for agents. The newer visual 3‑D interface (VIA) lets an agent control a virtual gripper in a simulated world. The minimal interface presents calibrated RGB, depth, camera calibration, and end‑effector commands. The benchmark shows that adding richer visual input gives only modest gains (1.5–6.8 percentage points) while more than doubling the cost.

Against all 22 tasks with 5 random seeds each, GPT‑6 Astra achieved a success rate of 49.8 % when using VIA and 48.3 % with the minimal RGBD interface. Claude Fable 5.1 reached 43.6 % with VIA and 36.8 % with RGBD, and GPT‑5.6 Sol lagged at 14.0 % with VIA. The differences are measured on the same simulator backend (MuJoCo) and share the same data packs, so the comparison isolates model capability and interface choice.

Summary table of agent success rates on RoboARCade tasks by interface
ModelInterfaceSuccess (%)Cost ($)Turns/EEFCalls/EEF
GPT-5.6-SolVIA14.025.053.63.3
Claude Opus 5VIA30.360.00†4.95.5
Claude Fable 5.1Multi-view RGBD36.833.161.01.7
VIA43.655.502.27.0
GPT-6-AstraMulti-view RGBD48.313.891.31.3
VIA49.831.842.33.0
Table 1. Table 1 reports the overall success percentages of GPT‑6 Astra, Claude Fable 5.1, and GPT‑5.6 Sol across all 22 tasks with 5 seeds each, separated by interface.Sharma, Satvik, Hu, Hengyuan, Kamat, Mihika, Sadigh, Dorsa, Bohg, Jeannette · arXiv:2611.08209v1 · Table 1: · CC BY 4.0

The results highlight that stronger agents can handle more subtasks, and that interface choice influences both task success and cost. Precision errors and incorrect success claims account for a large share of reported failures, with about 24 % to 39 % of subtask attempts mis‑classified as successful. Future work will focus on improving precision and reducing false positives.

Source paper

RoboARCade: Benchmarking Agentic Robot Control

Robotics
View on arXiv