• An API for the depth estimation pipeline
• PyTorch code for training and testing our models
• Demo code to see ClearGraspPlus in action with a RealSense D400 series camera
• Real images dataset capture utility available in another repo named IRL_transparent_objects_set for kincet v1 RGBD camera

• Download Data - Training Set
• Download Data - Testing and Validation Set
• Additional Training dataset COCO
• Additional Training dataset NYU_v2
• Additional Data for testing
• Download 3D Models

depth2depth is a C++ global optimization module used for depth completion, adapted from the DeepCompletion project. It resides in the api/depth2depth/ directory.

To compile the depth2depth binary, you will first need to identify the path to libhdf5. Note the location of hdf5/serial. Run the following command in terminal:

find /usr -iname "*hdf5.h*"

It will look similar to: /usr/include/hdf5/serial/hdf5.h.

Edit BOTH lines 28-29 of the makefile at api/depth2depth/gaps/apps/depth2depth/Makefile to add the path you just found as shown below:

USER_LIBS=-L/usr/include/hdf5/serial/ -lhdf5_serial
USER_CFLAGS=-DRN_USE_CSPARSE "/usr/include/hdf5/serial/"

Compile the binary:

cd api/depth2depth/gaps
export CPATH="/usr/include/hdf5/serial/"  # Ensure this path is same as read from output of `find /usr -iname "*hdf5.h*"`

make

This should create an executable, api/depth2depth/gaps/bin/x86_64/depth2depth. The config files will need the path to this executable to run the depth estimation pipeline.

Check the executable, by passing in the provided sample files:

cd api/depth2depth/gaps
bash depth2depth.sh

This will generate gaps/sample_files/output-depth.png, which should match the expected-output-depth.png sample file. It will also generate RGB visualizations of all the intermediate files.

We provide a script to run our full pipeline on a dataset and calculate accuracy metrics (RMSE, MAE, etc). Resides in the directory eval_depth_completion/.

Create a local copy of the config file:

cd eval_depth_completion/
cp config/config.yaml.sample config/config.yaml

Edit the config/config.yaml file to set pathWeightsFile parameters to the paths of the respective model checkpoints. To run evaluation on the different datasets, set the path(s) to their director(ies) within the files parameter.

Run ClearGrasp on the sample dataset:

python eval_depth_completion.py -c config/config.yaml

The script will run ClearGrasp on the given dataset, storing all it's output and calculating accuracy metrics of the depth completion of transparent objects. The metrics (RMSE, etc.) are stored in a csv file in the results dir. Resized inputs, output depths, output pointclouds and other intermediate files are also saved.

We provide a demonstration of how to use our API on images streaming from realsense D400 series camera. Each new frame coming from the camera stream is passed through the depth completion module to obtain completed depth of transparent objects and the results are displayed in a window.
Resides in the folder live-demo/. This demo requires the Librealsense SDK to be installed.

• Create a copy of the sample config file:

  cd live_demo
  cp config/config.yaml.sample config/config.yaml

• Edit config.yaml with paths to checkpoints of networks and depth2depth executable. Edit parameters as per your camera.

• Compile realsense.cpp:

  cd live-demo/realsense/
  mkdir build
  cd build
  cmake ..
  make

  This will create a binary build/realsense which is used to stream images from the realsense camera over TCP/IP. In case of issues, check FAQ.

• Connect a realsense d400 series camera to USB and start the camera stream:

  cd live_demo/realsense
  ./build/realsense

  This application will capture RGB and Depth images from the realsense and stream them on an TCP/IP port. It will also open a window with the RGB and Depth images displayed.

• Run demo:

  python live_demo.py -c config/config.yaml

  This will open a new window displaying input image, input depth, intermediate outputs (surface normals, occlusion boundaries, mask), modified input depth and output depth. Expect around 1 FPS with an i7 7700K CPU and 1080ti GPU. The global optimization module is CPU bound and takes almost 1 sec per image at 256x144p resolution with CPU at 4.2GHz.

The folder pytorch_networks/ contains the code used to train the surface normals, occlusion boundary and semantic segmentation models.

Go the to respective folder (eg: pytorch_networks/surface_normals) and create a local copy of the config file:

cp config/config.yaml.sample config/config.yaml

Edit the config.yaml file to fill in the paths to the dataset, select hyperparameter values, etc. All the parameters are explained in comments within the config file.

Start training:

python train.py -c config/config.yaml

Eval script can be run by:

python eval.py -c config/config.yaml

• input_depth.png: The path for the raw depth map from sensor, which is the depth to refine. It should be saved as 4000 x depth in meter in a 16bit PNG.
• output_depth.png: The path for the result, which is the completed depth. It is also saved as 4000 x depth in meter in a 16bit PNG.
• Occlusion Weights: The depth discontinuities channel is extracted from the occlusion outlines models' outputs scaled and saved as png file.
• Surface Normals: The output of surface normals model is saved as an .h5 file
• xres, yres: The resolution of image in x and y axes.
• fx, fy: The focal length used to take image in pixels
• cx, cy: The centre of the image. Ideally it is equal to (height/2, width/2)
• inertia weight: The strength of the penalty on the difference between the input and the output depth map on observed pixels. Set this value higher if you want to maintain the observed depth from input_depth.png.
• smoothness_weight: The strength of the penalty on the difference between the depths of neighboring pixels. Higher smoothness weight will produce soap-film-like result.
• tangent_weight: The universal strength of the surface normal constraint. Higher tangent weight will force the output to have the same surface normal with the given one.

Change the below line:

// Find and colorize the depth data
rs2::frame depth_colorized = color_map.colorize(aligned_depth);

to

// Find and colorize the depth data
rs2::frame depth_colorized = color_map(aligned_depth);