Byeonghyeon Lee*, Howoong Lee*, Usman Ali, and Eunbyung Park
Our code is based on TensoRF (https://github.com/apchenstu/TensoRF).
conda create -n SharpNeRF python=3.8
conda activate SharpNeRF
// Please install pytorch compatible to your cuda version.
pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install six tqdm scikit-image==0.19.2 opencv-python configargparse lpips imageio-ffmpeg kornia lpips tensorboard setuptools==59.5.0 plyfile
We tested our model with the following enivornment:
- Ubuntu 18.04
- Python 3.8.0
- Pytorch 1.10.0
- Cuda 11.4
Download 'real_defocus_blur' and 'synthetic_defocus_blur' dataset at
https://hkustconnect-my.sharepoint.com/:f:/g/personal/lmaag_connect_ust_hk/EqB3QrnNG5FMpGzENQq_hBMBSaCQiZXP7yGCVlBHIGuSVA?e=UaSQCC
Then, the dataset directory should look like below:
data/
real_defocus_blur/
defocusbush/
...
defocustools/
images/
images_4/
sparse/
...
synthetic_defocus_blur/
defocuscozy2room/
images/
images_4/
sparse/
...
...
Preprocess stage for computing sharpness level consists of two substeps: depth map estimation and focus map estimation.
You can download the precomputed one at https://drive.google.com/file/d/1IzucVQsnzA-fYw2LdB8odur1DHUIaNlO/view?usp=sharing
Unzip the downloaded file and replace the 'preprocess' directory with the downloaded files.
Or, you can compute it with the following steps.
We use MiDas (https://github.com/isl-org/MiDaS) with dpt_beit_large_512 weight to estimate scene depth.
Please run MiDas to compute depth map.
Then run
python preprocess_depth.py PATH_TO_OUTPUT SCENE_NAME
// ex. python preprocess_depth.py ./MiDaS/output tools
cf. You can process depth map of all scenes by setting SCENE_NAME as 'all'
python preprocess_depth.py ./MiDaS/output all // Process depth map of all scenes, including both real and synthetic dataset.
We use DMENet (https://github.com/codeslake/DMENet) to estimate focus map.
Download pretrained weight (dmenet_pretrained.npz) at https://drive.google.com/file/d/1wAahb4D8ldjigvPxYuFNFJOZh3Y29B_C/view?usp=drive_link and move it to the 'preprocess' directory.
Then run
python preprocess_focus.py PATH_TO_DATASET SCENE_NAME
// ex. python preprocess_focus.py ./data tools
// cf. python preprocess_focus.py ./data all
Now you can compute sharpness level.
python preprocess_sharpness_level.py SCENE_NAME
// ex. python preprocess_sharpness_level.py tools
// cf. python preprocess_sharpness_level.py all
Now you can run SharpNeRF with following command:
bash run.sh GPU_ID DATA_PATH EXP_NAME
// ex. bash run.sh 0 ./data/real_defocus_blur/defocustools test