This Project is a Tensorflow implementation of
- "High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction" IEEE Transactions on Pattern Analysis and Machine Intelligence, Nan Meng, Hayden K-H. So, Xing Sun, Edmund Y. Lam, 2019. [Paper]
- "High-order Residual Network for Light Field Super-Resolution" The 34th AAAI Conference on Artificial Intelligence, Nan Meng, Xiaofei Wu, Jianzhuang Liu, Edmund Y. Lam, 2020. [Paper]
- Python2==2.7
- Python3>=3.5
- Tensorflow r1.*>=r1.8
- tqmd
- opencv
- unrar
- Download the project
git clone https://github.com/monaen/LightFieldReconstruction.git --branch master --single-branch
- Training the model for spatial super-resolution (e.g. Sx4). You need to specify the
gamma_Sfor different spatial SR tasks.
python train_SpatialSR.py --datadir data/train/Spatial/5x5
--gamma_S 4
--verbose
--perceptual_loss
- Training the model for angular super-resolution (e.g. Ax4). You need to specify the
gamma_Afor different angular SR tasks.
python train_ViewSysthesis.py --datadir data/train/Angular/9x9
--gamma_A 4
--verbose
- Training the model for spatio-angular super-resolution (e.g. Sx2Ax2). You need to specify both the
gamma_Sandgamma_Afor different spatio-angular SR tasks.
python train_SpatialAngularSR.py --datadir data/train/SpatialAngular/5x5
--gamma_S 2
--gamma_A 2
--verbose
--perceptual_loss
usage:train_SpatialSR.py [-h] [--datadir DATADIR] [--lr_start LR_START] [--lr_beta1 LR_BETA1] [--batchSize BATCHSIZE]
[--imageSize IMAGESIZE] [--viewSize VIEWSIZE] [--channels CHANNELS] [--verbose VERBOSE]
[--num_epoch NUM_EPOCH] [--start_epoch START_EPOCH] [--gamma_S {1,2,3,4}]
[--gamma_A {0,1,2,3,4}] [--num_GRL_HRB NUM_GRL_HRB] [--num_SRe_HRB NUM_SRE_HRB]
[--resume RESUME] [--select_gpu SELECT_GPU] [--perceptual_loss PERCEPTUAL_LOSS]
[--vgg_model VGG_MODEL] [--save_folder SAVE_FOLDER]
optional arguments:
-h, --help Show this help message and exit
--datadir The training and testing data path
--lr_start The start learning rate
--lr_beta1 The exponential decay rate for the 1st moment estimates
--batchSize The batchsize of the input data
--imageSize Spatial size of the input light fields
--viewSize Angular size of the input light fields
--channels Channels=1 means only the luma channel; Channels=3 means RGB channels (not supported)
--verbose Whether print the network structure or not
--num_epoch The total number of training epoch
--start_epoch The start epoch counting number
--gamma_S {1,2,3,4} Spatial downscaling factor
--gamma_A {0,1,2,3,4} Angular downscaling factor, '0' represents 3x3->7x7
--num_GRL_HRB The number of HRB in GRLNet (only for AAAI model)
--num_SRe_HRB The number of HRB in SReNet (only for AAAI model)
--resume Need to resume the pretrained model or not
--select_gpu Select the gpu for training or evaluation
--perceptual_loss Need to use perceptual loss or not, if true, one also have to set the vgg_model item
--vgg_model Pretrained VGG model path
--save_folder Model save path
We provide the training samples to facilitate users to generate custom training set.
# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_trainingsamples.sh
- Spatial SR evaluation (Sx4, Sx3, Sx2)
python evaluation_SpatialSR.py --datapath $PATH_TO_THE_EVALUATION_LF_DATA \
--gamma_S $Upscaling_Factor \
--pretrained_model $PATH_TO_THE_PRETRAINED_MODEL \
--select_gpu 0 --verbose
Here, we provide an example to evaluate the performance on Sx4 task as guidance.
# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_evaluation_data.sh
cd ../pretrained_models
bash download_pretrained_models_HDDRNet_Sx4.sh
cd ..
python evaluation_SpatialSR.py --datapath data/evaluation/buddha.mat \
--gamma_S 4 \
--pretrained_model pretrained_models/HDDRNet/Sx4/HDDRNet \
--select_gpu 0 --verbose
- Angular SR evaluation (Ax4, Ax3, Ax2, A3x3_7x7)
# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_occlusions20.sh
cd ../pretrained_models
bash download_pretrained_models_HDDRNet_Ax4.sh
cd ..
python evaluation_ViewSynthesis.py --datapath data/testset/occlusions20/occlusions_48.mat \
--gamma_A 4 \
--pretrained_model pretrained_models/HDDRNet/Ax4/HDDRNet \
--select_gpu 0 --verbose
python evaluation_ViewSynthesis_Patchwise.py --datapath data/testset/occlusions20/occlusions_48.mat \
--gamma_A 4 \
--pretrained_model pretrained_models/HDDRNet/Ax4/HDDRNet \
--select_gpu 0 --verbose
- Spatial and Angular SR evaluation (Sx2Ax2, Sx3Ax2)
Note: We provide the pretrained model for Sx2Ax2, but the model is not well-trained. We also provide the code for Sx3Ax2 task. Here, we provide an example to evaluate the performance on Sx2Ax2 task as guidance.
# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_evaluation_data.sh
cd ../pretrained_models
bash download_pretrained_models_Sx2Ax2.sh
cd ..
python evaluation_SpatialAngularSR.py --datapath data/evaluation/buddha.mat \
--gamma_S 2 \
--gamma_A 2 \
--pretrained_model pretrained_models/Others/Sx2Ax2/weights \
--select_gpu 0 --verbose
- Evaluate the entire dataset
Note: All the reported results are tested using Matlab, and here we provide an example to show how to evaluate the results on reflective20 dataset.
# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_reflective20.sh
cd ../pretrained_models
bash download_pretrained_models_M-HDDRNet_Ax4.sh
cd ..
python evaluation_on_dataset.py --datafolder data/testset/reflective20 \
--gamma_S 1 \
--gamma_A 4 \
--pretrained_model pretrained_models/M-HDDRNet/Ax4/M-HDDRNet \
--result_folder results \
--select_gpu 0 --verbose
# After the execution of "evaluation_on_dataset.py", run the m-file "evaluation.m"
matlab -nodisplay -nosplash -nodesktop -r "run evaluation.m; exit;"
To download the pretrained mdoels, please change the directory into the folder pretrained_models and run the corresponding bash files. For example, to download the HDDRNet_Sx4 pretrained model,
# path = Path to LightFieldReconstruction
cd pretrained_models
bash download_pretrained_models_HDDRNet_Sx4.sh
We provide a detailed instruction on how to download the pretrained models for differnet SR models and tasks.
Paper
@article{Meng2019High,
title = {High-dimensional dense residual convolutional neural network for light field reconstruction},
author = {Meng, Nan and So, Hayden Kwok-Hay and Sun, Xing and Lam, Edmund},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2019}
}
@article{Meng2020High,
title = {High-order residual network for light field super-resolution},
author = {Meng, Nan and Wu, Xiaofei and Liu, Jianzhuang and Lam, Edmund},
journal = {Association for the Advancement of Artificial Intelligence},
volume = {34},
number = {7},
pages = {11757-11764},
month = {February},
year = {2020}
}
- T-PAMI
- Spatial super-resolution 2x framework.
- Spatial super-resolution 3x framework.
- Spatial super-resolution 4x framework.
- Angular super-resolution 2x (5x5->9x9) framework.
- Angular super-resolution 2.3x (3x3->7x7) framework.
- Angular super-resolution 3x (3x3->9x9) framework.
- Angular super-resolution 4x (2x2->8x8) framework.
- Spatial super-resolution 2x Angular super-resolution 2x framework.
- The pretrained models for spatial Sx2, Sx3, Sx4.
- The pretrained models for angular Ax2, Ax2.3, Ax3, Ax4.
- The pretrained models for spatio-angular Sx2Ax2.
- Training samples.
- Testset.
- Evaluation code.
- Results and analysis.
- AAAI
- Spatial super-resolution 2x framework.
- Spatial super-resolution 3x framework.
- Spatial super-resolution 4x framework.
- Angular super-resolution 2x (5x5->9x9).
- Angular super-resolution 2.3x (3x3->7x7).
- Angular super-resolution 3x (3x3->9x9).
- Angular super-resolution 4x (2x2->8x8).
- Other materials.
- Matlab
- Matlab evaluation code.
For some reasons, we are not able to provide the pre-trained models for the AAAI work.
Our code is released under MIT License. We would like to appreciate the GPU support from Lee Ultrasound Imaging Group of Prof.Wei-Ning Lee