Lirui Wang, Xinlei Chen, Jialiang Zhao, Kaiming He
Neural Information Processing Systems (Spotlight), 2024
This is a Huggingface LeRobot implementation for pre-training Heterogenous Pre-trained Transformers (HPTs).
Create a virtual environment with Python 3.10 and activate it, e.g. with miniconda:
conda create -y -n lerobot python=3.10
conda activate lerobotInstall 🤗 LeRobot with simulation environments:
pip install -e ".[aloha, pusht]Check the following two folders for most of the modifications.
├── lerobot
| ├── configs # contains hydra yaml files with all options that you can override in the command line
| | ├── ... # various sim environments and their datasets: aloha.yaml, pusht.yaml, xarm.yaml
| | └── policy # including policies config for hpt.yaml
| ├── common # contains classes and utilities
| | ├── ... # various datasets of human demonstrations: aloha, pusht, xarm
| | ├── ... # various sim environments: aloha, pusht, xarm
| | ├── policies # including modeling and configuration for hpt
| ├── ...
-
By default, the HPT model loads the x-large pre-trained trunk. Use these config parameters
policy.embed_dim=256 policy.num_heads=8 policy.num_blocks=16to switch to the hpt-base trunk for example. -
Run the following scripts for aloha transfer cube experiments.
Aloha Experiments
python lerobot/scripts/train.py \
policy=hpt_transformer env=aloha env.task=AlohaTransferCube-v0 \
dataset_repo_id=lerobot/aloha_sim_transfer_cube_human \
wandb.enable=true
- Run the following scripts for push-T experiments.
PushT experiments
python lerobot/scripts/train.py \
policy=hpt_pusht env=pusht env.task=PushT-v0 \
dataset_repo_id=lerobot/pusht \
wandb.enable=true
- Run the following scripts for real-world Koch experiments.
Koch Experiments
python lerobot/scripts/train.py policy=hpt_koch_real env=koch_real \
dataset_repo_id=lerobot/koch_pick_place_5_lego \
wandb.enable=true
If you find HPT useful in your research, please consider citing:
@inproceedings{wang2024hpt,
author = {Lirui Wang, Xinlei Chen, Jialiang Zhao, Kaiming He, Russ Tedrake},
title = {Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers},
booktitle = {Neurips},
year = {2024}
}
Our implementation is built upon the excellent LeRobot codebase.