Robotics benchmarks often vary objects, instructions and room appearances while leaving the machine itself stable. Deployment breaks that assumption. Maintenance, replacement parts and calibration changes can alter the mapping from a camera observation to a physical command even when the task and surrounding scene remain familiar.
LIBERO-RIG puts that neglected source of variation under controlled examination. It holds tasks fixed while changing hardware and setup parameters. The resulting question is unusually useful for operators: how much of a policy’s apparent generality depends on the exact machine configuration through which it was trained and evaluated?
Eleven ways to change the same task
LIBERO-RIG, posted October 13, varies hardware and setup along eleven axes while keeping LIBERO tasks fixed. The authors report substantial degradation across evaluated policies, with differences between perturbation types and action interfaces. Read the research paper.
The benchmark varies camera mounting and optics, camera-to-arm calibration, actuation gain, control rate, proprioceptive noise, base pose, table height and gripper pads among its eleven axes. Five severity levels produce 210 conditions across the four LIBERO suites. Fixed manifests describe the perturbations for each episode.
This design separates setup sensitivity from task difficulty. If a familiar task deteriorates after a controlled camera or actuation change, the difference cannot simply be credited to a more challenging instruction. That isolation is valuable because a single headline success score tends to mix perception, control and task understanding into one number.
A camera fault and an actuation fault demand different evidence
Moving a camera can change what the policy sees. Altering actuation gain can change what a command does while leaving the image initially identical. Those faults challenge different parts of the perception-to-action mapping, so success against one does not establish tolerance of the other.
LIBERO-RIG reports that training on visual perturbations does not mitigate faults that leave camera images unchanged. This is an important causal boundary for augmentation claims. More image variety cannot be assumed to teach an unrepresented change in control dynamics. The practical response may involve different sensing, calibration, adaptation or acceptance limits, depending on the fault. The benchmark does not identify one universal remedy. Its value is that controlled axes make it possible to discover which family of intervention is needed before spending another training budget on a broad robustness objective that may target the wrong failure.
The average hides the failure mechanism
Across nine evaluated vision-language-action models and 500 episodes per condition, the paper reports pooled success declining from 0.932 to 0.767 at middle severity and 0.509 at the highest severity. These are benchmark perturbations, not an observed rate of deterioration in a working fleet.
The variation between axes and action interfaces is as important as the aggregate decline. A failure caused by changed control dynamics may need a different remedy from one caused by altered images. Treating every robustness problem as a demand for more visual augmentation risks spending training resources on the wrong mechanism.
Maintenance becomes part of model evaluation
An installation test could establish tolerances for remounting a camera, replacing a gripper pad and changing a control frequency, then repeat a small diagnostic suite after maintenance. Those proposed checks would connect model evaluation to the operations that routinely change a robot’s effective embodiment.
The broader lesson is that generalisation includes continuity of the machine, not just diversity of the world. A policy may need setup-aware inputs, adaptation or explicit acceptance limits. LIBERO-RIG does not settle which remedy is best. It gives researchers a controlled way to identify the part of the problem they are actually trying to solve.
References
Questions, news tips or corrections? Contact [email protected].
