SO2 offers a fresh perspective on offline-to-online reinforcement learning through Q-value estimation and presents a straightforward yet effective implementation.
SO2: A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu, Yaodong Yang, Yu Liu, Wanli Ouyang arXiv:2312.07685
- (06/2024) Independent code has been released!
- (12/2023) Code has been released!
Considering the difficulty of tracking the main branch in the DI-engine updates, we have updated the independent code branch to reproduce the results in the paper.
You can check the details in the code branch.
Checkpoints can be downloaded from the following links: Google Drive
The scripts from installation to execution are all here👏.
# install MuJoCo for Linux
mkdir -p ~/.mujoco/mujoco210
wget https://mujoco.org/download/mujoco210-macos-x86_64.tar.gz -O mujoco210-macos-x86_64.tar.gz
tar -xf mujoco210-linux-x86_64.tar.gz -C ~/.mujoco/mujoco210
pip install -U 'mujoco-py<2.2,>=2.1'
# install D4RL
pip install git+https://github.com/Farama-Foundation/d4rl@master#egg=d4rl
# install SO2
conda create --name SO2 python=3.7 -y
conda activate SO2
git clone -b code https://github.com/opendilab/SO2.git
pip install -e SO2
# run d4rl experiment
# Note download checkpoints and unzip into ckpt folder
python3 -u train.py
The scripts from installation to execution are all here👏.
# install MuJoCo for Linux
mkdir -p ~/.mujoco/mujoco210
wget https://mujoco.org/download/mujoco210-macos-x86_64.tar.gz -O mujoco210-macos-x86_64.tar.gz
tar -xf mujoco210-linux-x86_64.tar.gz -C ~/.mujoco/mujoco210
pip install -U 'mujoco-py<2.2,>=2.1'
# install D4RL
pip install git+https://github.com/Farama-Foundation/d4rl@master#egg=d4rl
# install SO2
git clone https://github.com/opendilab/SO2
cd SO2
pip install -r requirements.txt
# run d4rl experiment
python3 -u so2/d4rl_main.py
- Download the MuJoCo version 2.1 binaries for Linux or OSX.
- Extract the downloaded
mujoco210directory into~/.mujoco/mujoco210.
To include mujoco-py in your own package, add it to your requirements like so:
pip install -U 'mujoco-py<2.2,>=2.1'
D4RL can be installed by cloning the repository as follows:
git clone https://github.com/Farama-Foundation/d4rl.git
cd d4rl
pip install -e .
Or, alternatively:
pip install git+https://github.com/Farama-Foundation/d4rl@master#egg=d4rl
git clone https://github.com/opendilab/SO2
cd SO2
pip install -r requirements.txt
python3 -u so2/d4rl_main.py
This project is released under the Apache 2.0 license. See LICENSE for details.
If you use SO2 in your research or wish to refer to the baseline results published here, please use the following BibTeX entry.
@inproceedings{zhang2023perspective,
title={A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning},
author={Zhang, Yinmin and Liu, Jie and Li, Chuming and Niu, Yazhe and Yang, Yaodong and Liu, Yu and Ouyang, Wanli},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
year={2024}
}