This repository is a fork from ORB-SLAM3.

In this work, we add the RGB-L (LiDAR) mode to the well-known ORB-SLAM3. This allows to directly integrate LiDAR depth measurements in the visual SLAM. By doing this, we get precision close to Stereo mode with greatly reduced computation times.

We suggest using Docker to run the code!

If you want to build the code locally, please refer to the Dockerfile for installation instructions.

To build the docker image, simply run from main directory:

sudo ./scripts/docker_build.sh

Some examples for Mono camera mode and some of the example datasets have been removed to increase the clarity. These can be copied from the original repository and added to the CMakeLists.txt file.

Following modes can be directly used:

• RGB-L to fuse camera and LiDAR data
• RGB-D to use offline generated depth maps
• Stereo to use stereo images

Exemplary KITTI folder structure (where path in docker_run.sh points to):

KITTI Odometry dataset --> Docker mounting point
└───poses/
│   │   00.txt
│   │   01.txt
│   │   ...txt
│
└───sequences
│   └───00/
|   |       calib.txt
|   |       times.txt
|   |       image_0/
|   |       image_1/
|   |       image_2/
|   |       image_3/
|   |       velodyne/
│   └───01/
|   |       ...
│   └───02/
|   |       ...
└───└───../

Add your path to the KITTI Odometry dataset in the following files: scripts/docker_run.sh and scripts/docker_run_dev.sh. (Replace "/LocalStorage/KITTI_odometry/dataset/" with your local path.)

To start the Docker container, run

sudo ./scripts/docker_run.sh

or

sudo ./scripts/docker_run_dev.sh

to run a container with the mounted code instead of the copied code. This allows to change the code or config files from your local machine and directly running or rebuilding within the container. No need to rebuild the Dockerfile. \

Inside the container, change to the ORB SLAM3 directory:

cd ORBSLAM3

To run the different modes with the KITTI Odometry dataset, use the following bash scripts inside the container:

./scripts/kitti_rgbl.sh

./scripts/kitti_rgbd.sh

./scripts/kitti_stereo.sh

In the scripts, select the sequence you want to evaluate.
To set parameters for the modes, check out the configuration files for the corresponding sequences: e.g. Examples/RGB-L/KITTIIxx-xx.yaml Set the correct config file in kitti_rgbl.sh. Once the code launched, you should see two windows opening up: visualization of the camera images (in RGB-L mode with LiDAR depth) and the feature map building up.

Existing open-source networks such as PEnet, Enet, or MSG_CHNet can be used to generate depth maps.

In the tools directory, we provide a Python script to transform the Velodyne pointclouds (.bin) to depth images: proj_velo2cam_kitti_odom.py. These can be used to evaluate neural networks. Note that the image resolutions might have to be adjusted in the RGB-D config files.

python3 proj_velo2cam_kitti_odom.py

Adjust the script to your needs by setting the correct path- and file names.

The ROS2 node kitti_publisher in the depth_map_creator package can be used to generate depth maps. The correct publicFrame for the corresponding upsampling method has to be selected. Use this command to build the node:

source /opt/ros/galactic/setup.bash
colcon build --packages-select depth_map_creator
source install/setup.bash

Example start of the node:

ros2 run depth_map_creator kittipublisher --ros-args -p PublishFrequency:=10.0 -p SaveDenseDepth:=true -p CreateOverlay:=true -p SequencePath:=/kitti/sequences/00/ << Path to Dataset >>/dataset/sequences/00/

Upsamled depth images will be saved under /path/to/dataset/sequences/00/image_0_Depth and image_0_Depth_Overlay.

Florian Sauerbeck, Benjamin Obermeier, Martin Rudolph, and Johannes Betz "RGB-L: Enhancing Indirect Visual SLAM using LiDAR-based Dense Depth Maps" (under review)

Link to preprint

@article{sauerbeck2022rgb,
  title={RGB-L: Enhancing Indirect Visual SLAM using LiDAR-based Dense Depth Maps},
  author={Sauerbeck, Florian and Obermeier, Benjamin and Rudolph, Martin and Betz, Johannes},
  journal={arXiv preprint arXiv:2212.02085},
  year={2022}
}