Web Analytics

Learning Visual Quadrupedal Loco-Manipulation from Demonstrations

Zhengmao He1,4             Kun Lei1             Yanjie Ze1             Koushil Sreenath2             Zhongyu Li2             Huazhe Xu1,3
1Shanghai Qi Zhi Institute. 2University of California, Berkeley. 3Tsinghua University, IIIS. 4The Hong Kong University of Science and Technology (Guangzhou).
Paper arXiv YouTube BiliBili Twitter Code(comming soon)

Abstract

Quadruped robots are progressively being integrated into human environments. Despite the growing locomotion capabilities of quadrupedal robots, their interaction with objects in realistic scenes is still limited. While additional robotic arms on quadrupedal robots enable manipulating objects, they are sometimes redundant given that a quadruped robot is essentially a mobile unit equipped with four limbs, each possessing 3 degrees of freedom (DoFs). Hence, we aim to empower a quadruped robot to execute real-world manipulation tasks using only its legs. We decompose the loco-manipulation process into a low-level reinforcement learning (RL)-based controller and a high-level Behavior Cloning (BC)-based planner. By parameterizing the manipulation trajectory, we synchronize the efforts of the upper and lower layers, thereby leveraging the advantages of both RL and BC. Our approach is validated through simulations and real-world experiments, demonstrating the robot's ability to perform tasks that demand mobility and high precision, such as lifting a basket from the ground while moving, closing a dishwasher, pressing a button, and pushing a door.

To address some concerns about sim2real, I have added some detailed information at the end.

Autonomous Loco-Manipulation

Fully autonomous without any human teleoperation.

Real-World Tasks

Close dishwasher

Press button

Open door

Lift basket



Simulation Tasks

Visualized point clouds are transformed from egocentric perspective.

Press button

Pull handle

Push door

Lift basket

Twist valve

Pull object

Shoot ball

Open dishwasher

Close dishwasher



Teleopration

Besides autonomous manipulation, we can also collect data via teleopration.

Method


Track Random Curve with Our Low-level Control Policy

The red sphere represents the Bézier control point,
the small coordinate axis represents the pose of the target trajectory at that time,
and the big coordinate axis represents the pose of the end-effector.



Expert Demonstration Collection

We design manipulation trajectories for different tasks and collect demonstrations rapidly through parallel simulation.



The post-processing process of point clouds in the real world

To address the obvious visual info gap, I performed a series of post-processing on the point cloud of the real-world. These complex steps clearly constrain the potential of the proposed pipeline, which has resulted in me having to migrate only 4 tasks from simulation to reality.


Real-World Basket Task

BibTeX

@inproceedings{he2024learning,
      title={Learning Visual Quadrupedal Loco-Manipulation from Demonstrations},
      author={Zhengmao He, Kun Lei, Yanjie Ze, Koushil Sreenath, Zhongyu Li, Huazhe Xu},
      year={2024},
      eprint={2403.20328},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
    }

Contact

Feel free to contact Zhengmao He if you have any questions on this project.