This repo contains training and evaluation code for the following paper:

Real-Time Video Deblurring via Lightweight Motion Compensation
*Hyeongseok Son, *Junyong Lee, Sunghyun Cho, and Seungyong Lee (*equal contribution)
POSTECH
Pacific Graphics (PG) 2022 (special issue of the Computer Graphics Forum (CGF))

Tested environment

$ git clone https://github.com/codeslake/RealTime_VDBLR.git
$ cd RealTime_VDBLR

$ conda create -y --name RealTime_VDBLR python=3.8 && conda activate RealTime_VDBLR

# Install Pytorch (1.12.1 for example,)
$ conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge

# Install required dependencies (one of below depend on CUDA version)
# for CUDA10.2
$ ./install/install_CUDA10.2.sh
# for CUDA11.1
$ ./install/install_CUDA11.1.sh
# for CUDA11.3
$ ./install/install_CUDA11.3.sh
# for CUDA11.6
$ ./install/install_CUDA11.6.sh

Download and unzip datasets under [DATASET_ROOT]:

• Su et al.'s DVD dataset: OneDrive | Dropbox
• Nah et al.'s GoPro dataset: OneDrive | Dropbox
• Nah et al.'s REDS dataset: download script

[DATASET_ROOT]
    ├── train_DVD
    ├── test_DVD
    ├── train_nah
    ├── test_nah
    └── REDS
        └── reds_lmdb
            ├── reds_info_train.pkl
            ├── reds_info_valid.pkl
            ├── reds_train
            ├── reds_train_gt
            ├── reds_valid
            └── reds_valid_gt

[DATASET_ROOT] can be modified with config.data_offset in ./configs/config.py.

Download pretrained weights (OneDrive | Dropbox) and place them under ./ckpt/:

RealTime_VDBLR
├── ...
├── ckpt
│   ├── liteFlowNet.pytorch
│   ├── MTU#_DVD.pytorch
│   ├── MTU#_GoPro.pytorch
│   ├── MTU#_REDS.pytorch
│   └── ...
└── ...

For PSNRs and SSIMs reported in the paper, we use the approach of Koehler et al. following Su et al., that first aligns two images using global translation to represent the ambiguity in the pixel location caused by blur.
Refer here for the evaluation code.

# [n]-stack MTUs evaluationed on [DVD|GoPro|REDS] datasets
./script_eval/MTU[n]_[dataset].py

# for 2-stack amp model,
./script_eval/MTU2_amp_[dataset].py

# for 10-stack large model,
./script_eval/MTU10_L_[dataset].py

# 4-stack model on REDS dataset, for example,
./script_eval/MTU4_REDS.py

Testing results will be saved in [LOG_ROOT]/PG2022_RealTime_VDBLR/[mode]/result/eval/[mode]_[epoch]/[data]/.

[LOG_ROOT] can be modified with config.log_offset in ./configs/config.py.

• --data: The name of a dataset to evaluate: DVD | nah | REDS. Default: DVD
  • The data structure can be modified in the function set_eval_path(..) in ./configs/config.py.

• Logging
• Training and testing details

Open an issue for any inquiries. You may also have contact with [email protected] or [email protected]

This software is being made available under the terms in the LICENSE file. Any exemptions to these terms require a license from the Pohang University of Science and Technology.

If you find this code useful, please consider citing:

@InProceedings{Son2022RTDeblur,
    author    = {Hyeongseok Son and Junyong Lee and Sunghyun Cho and Seungyong Lee},
    title     = {Real-Time Video Deblurring via Lightweight Motion Compensation},
    booktitle = {Pacific Graphics},
    year      = {2022},
}