GROOT: Learning Generalizable Manipulation Policies with Object-Centric 3D Representations

1-min spotlight video. Turn on audio or the caption to learn about the narrative.

Abstract

We introduce GROOT, an imitation learning method for learning robust policies with object-centric and 3D priors. GROOT builds policies that generalize beyond their initial training conditions for vision-based manipulation. It constructs object-centric 3D representations that are robust toward background changes and camera views and reason over these representations using a transformer-based policy. Furthermore, we introduce a segmentation correspondence model that allows policies to generalize to new objects at test time. Through comprehensive experiments, we validate the robustness of GROOT policies against perceptual variations in simulated and real-world environments. GROOT's performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances, whereas both state-of-the-art end-to-end learning methods and object proposal-based approaches fall short. We also extensively evaluate GROOT policies on real robots, where we demonstrate the efficacy under very wild changes in setup.

Method

GROOT builds object-centric 3D representations to learn generalizable manipulation policies.

GROOT leverages an interactive segmentation model, S2M, to obtain a single-frame annotation from demonstrators. Then a Video Object Segmentation model, XMem, propagates segmentation masks across time frames. The object masks are then back-projected into point clouds, and a transformer-based policy processes the point clouds to output actions.

Segmentation Correspondence Model (SCM)

SCM enables new object generalization.

GROOT uses SCM based on an open-vocabulary segmentation model (SAM) and a pretrained semantic feature model (DINOv2) to allow generalization to new objects.