First-hand information for everyone
Main discovery FLoRe is a framework that learns force‑reactive humanoid loco‑manipulation from robot‑free demonstrations that record both motion and contact wrench. A wrench is the contact force and torque measured by a force/torque (F/T) sensor. (FLoRe stands for the presented framework.) [D1]
How it works — data and policy Demonstrations are collected without a robot. A demonstrator holds two handheld UMI‑style grippers. Each gripper carries a wrist camera, a pose tracker, a six‑axis F/T sensor, and a jaw‑width encoder. The demonstrator also wears chest and ankle trackers. The same gripper–sensor assembly is mounted on the robot for deployment. [M1]
A learned high‑level policy is a force‑reactive diffusion policy. It conditions on wrist images, low‑dimensional robot state, and measured per‑hand wrenches. The policy predicts chunks of future actions. Those chunks include per‑hand desired force commands and pose/keypoint targets. (See Methods for architecture and training details.) [M2][M6]
How it is executed — controller A hybrid compliance whole‑body controller executes the policy outputs. The controller converts each hand’s force command into a small shift of the hand’s position target. It does this with an integral of the measured force error so that each hand tracks force along commanded axes and remains compliant along uncommanded axes. The controller uses soft‑contact phase hysteresis so the integral builds and clears without privileged contact signals. The controller outputs joint‑position targets; the unified reference generator computes p_tgt = p_* + f_cmd/K + I/K. [M3][M4]
Training and simulation details The controller and policy are trained in simulation (Isaac Lab). Each hand contacts a simulated compliant plane with randomized outward normal and stiffness. That virtual plane reacts only to penetration (training was effectively normal forces only). The simulated F/T reading includes randomized distal mass, bias and white noise to model sensor dynamics. [M5]
Concrete results — robot and simulation On a Unitree G1 robot and three whole‑body contact‑rich tasks (Push_Gantry, Rotate_Box, Wipe_Board), the force‑reactive policy succeeded in 24/30 rollouts (80%). Per task, success counts were 7/10 (Push_Gantry), 9/10 (Rotate_Box), and 8/10 (Wipe_Board). A position‑only diffusion policy trained on the same demonstrations and executed by the same controller (with force commands disabled) succeeded in 8/30 rollouts (26.7%), with per‑task counts 2/10, 3/10, and 3/10 respectively. Withholding only the predicted force command (predicted but not executed) reduced overall success to 11/30 rollouts (36.7%), with per‑task counts 3/10, 4/10, and 4/10. Removing only the measured force observation (so the policy could not see the wrench) reduced overall success to 20/30 rollouts (66.7%), with per‑task counts 6/10, 7/10, and 7/10. These comparisons used paired test conditions across policies and task‑specific success criteria. [D2][D3][C4]
In simulation the hybrid compliance controller tracked per‑hand force commands from about 5 N to 25 N with a gain near 1.03–1.06 and an in‑contact error of roughly 1 N per hand. This force tracking incurred little loss in whole‑body motion accuracy in the evaluated motion sets. On the real robot, steady‑state force tracking along the x axis reached ≈91% of commanded magnitude, with mean absolute steady‑state error ≈1.1 N, a ±1 N convergence band reached in ≈1.8 s on average, and steady‑state fluctuation about ±0.3–0.5 N. [D4][E4]
Baselines and metrics Baselines included a large‑scale whole‑body motion tracker (SONIC), SONIC plus an analytic admittance correction, a controller trained without the force channel (FLoRe w/o Force), and a position‑only diffusion policy (DP). Motion metrics reported included global root error, wrist position error and wrist orientation error. Force metrics included dose‑response gain (slope of realized vs. true contact force), in‑contact L1 error, and the F=0 floor (sensor reading with no command). Evaluations used held‑out motion sets and dose‑response tests across axes. [C1][C2][C3]
Significance Force‑reactive policies resolved failures that motion‑only policies could not. Motion‑only policies stalled, hovered, pressed too hard, or lost balance in scenarios where contact force matters but is not visible in images. Adding a learned per‑hand force channel let the robot sustain contact, coordinate both hands, and apply force while walking and balancing. [D2]
Supported caveats and limitations There is a measurable F=0 floor in the sensor/command pipeline of about 1.25–1.50 N. At deployment, commands below 1.5 N were zeroed and commands capped at 25 N. This bounds the smallest verifiable command. [E1]
Training used a frictionless virtual compliant plane, so the policy saw only normal (out‑of‑plane) forces during training. Tangential (shear) loads were not seen in training. On the real robot, lateral and downward commands reached a lower gain and partly leaked into other axes. [E2]
Adding the force channel had small tradeoffs in motion tracking. Relative to FLoRe w/o Force, FLoRe added 1.3–3.2 cm of global root error and up to 8.0 mm of wrist position error, while roughly halving wrist orientation error. These are implementation‑level tradeoffs reported in simulation evaluations. [E3]
Concluding note The FLoRe pipeline combines robot‑free forceful demonstrations, a force‑reactive diffusion policy, and a hybrid compliance whole‑body controller. It shows that learned per‑hand force commands can substantially improve real‑robot success on contact‑rich humanoid tasks, while also exhibiting measurable sensing and training limits that guide safe command ranges. [D1][D2][D3][D4][M1][M2][M3][M4][M5][M6][C1][C2][C3][C4][E1][E2][E3][E4]