Skip to content

Commit

Permalink
RAFT skeleton project template (#1312)
Browse files Browse the repository at this point in the history 
This is a copy and modification of a user's project but I think this is going to be generally useful to users as the same types of challenges are going to come up again. In this case, the user wasn't able to build/link because they weren't using `rapids-cmake` to propagate important configuration settings. I think having a skeleton project available that we build in CI and keep up to date will help new users build more applications on RAFT. 


TODO:
- [x] Make building the template optional 
- [x] Verify this can build in CMake and reuse already built/installed bits
- [x] Add to docs / readme and reference in README.md
- [x] Add a little example of invoking an API (maybe `pairwise_distances`?) to `main()`

Authors:
  - Corey J. Nolet (https://github.com/cjnolet)
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Micka (https://github.com/lowener)
  - Dante Gama Dessavre (https://github.com/dantegd)
  - Divye Gala (https://github.com/divyegala)
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #1312
  • Loading branch information
@cjnolet
cjnolet authored Mar 24, 2023
1 parent 8f1fa07 commit 9389108
Show file tree
Hide file tree
Showing 14 changed files with 409 additions and 88 deletions.
1 change: 1 addition & 0 deletions .github/workflows/build.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ jobs:
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
skip_upload_pkgs: libraft-template
docs-build:
if: github.ref_type == 'branch' && github.event_name == 'push'
needs: python-build
Expand Down
61 changes: 12 additions & 49 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# <div align="left"><img src="https://rapids.ai/assets/images/rapids_logo.png" width="90px"/>&nbsp;RAFT: Reusable Accelerated Functions and Tools</div>


![Navigating the canyons of accelerated possibilities](img/raft.png)

## Resources
Expand Down Expand Up @@ -85,9 +84,9 @@ raft::device_resources handle;
int n_samples = 5000;
int n_features = 50;

auto input = raft::make_device_matrix<float>(handle, n_samples, n_features);
auto labels = raft::make_device_vector<int>(handle, n_samples);
auto output = raft::make_device_matrix<float>(handle, n_samples, n_samples);
auto input = raft::make_device_matrix<float, int>(handle, n_samples, n_features);
auto labels = raft::make_device_vector<int, int>(handle, n_samples);
auto output = raft::make_device_matrix<float, int>(handle, n_samples, n_samples);

raft::random::make_blobs(handle, input.view(), labels.view());

Expand Down Expand Up @@ -222,52 +221,15 @@ pip install raft-dask-cu11 --extra-index-url=https://pypi.ngc.nvidia.com

### CMake & CPM

RAFT uses the [RAPIDS-CMake](https://github.com/rapidsai/rapids-cmake) library, which makes it simple to include in downstream cmake projects. RAPIDS CMake provides a convenience layer around CPM.

After [installing](https://github.com/rapidsai/rapids-cmake#installation) rapids-cmake in your project, you can begin using RAFT by placing the code snippet below in a file named `get_raft.cmake` and including it in your cmake build with `include(get_raft.cmake)`. This will make available several targets to add to configure the link libraries for your artifacts.

```cmake
set(RAFT_VERSION "22.12")
set(RAFT_FORK "rapidsai")
set(RAFT_PINNED_TAG "branch-${RAFT_VERSION}")
function(find_and_configure_raft)
set(oneValueArgs VERSION FORK PINNED_TAG COMPILE_LIBRARIES)
cmake_parse_arguments(PKG "${options}" "${oneValueArgs}"
"${multiValueArgs}" ${ARGN} )
#-----------------------------------------------------
# Invoke CPM find_package()
#-----------------------------------------------------
rapids_cpm_find(raft ${PKG_VERSION}
GLOBAL_TARGETS raft::raft
BUILD_EXPORT_SET projname-exports
INSTALL_EXPORT_SET projname-exports
CPM_ARGS
GIT_REPOSITORY https://github.com/${PKG_FORK}/raft.git
GIT_TAG ${PKG_PINNED_TAG}
SOURCE_SUBDIR cpp
OPTIONS
"BUILD_TESTS OFF"
"BUILD_BENCH OFF"
"RAFT_COMPILE_LIBRARIES ${PKG_COMPILE_LIBRARIES}"
)
endfunction()
# Change pinned tag here to test a commit in CI
# To use a different RAFT locally, set the CMake variable
# CPM_raft_SOURCE=/path/to/local/raft
find_and_configure_raft(VERSION ${RAFT_VERSION}.00
FORK ${RAFT_FORK}
PINNED_TAG ${RAFT_PINNED_TAG}
COMPILE_LIBRARIES NO
)
```
RAFT uses the [RAPIDS-CMake](https://github.com/rapidsai/rapids-cmake) library, which makes it easy to include in downstream cmake projects. RAPIDS-CMake provides a convenience layer around CPM. Please refer to [these instructions](https://github.com/rapidsai/rapids-cmake#installation) to install and use rapids-cmake in your project.

#### Example Template Project

You can find an [example RAFT](cpp/template/README.md) project template in the `cpp/template` directory, which demonstrates how to build a new application with RAFT or incorporate RAFT into an existing cmake project.

#### CMake Targets

Several CMake targets can be made available by adding components in the table below to the `RAFT_COMPONENTS` list above, separated by spaces. The `raft::raft` target will always be available. RAFT headers require, at a minimum, the CUDA toolkit libraries and RMM dependencies.
Additional CMake targets can be made available by adding components in the table below to the `RAFT_COMPONENTS` list above, separated by spaces. The `raft::raft` target will always be available. RAFT headers require, at a minimum, the CUDA toolkit libraries and RMM dependencies.

| Component | Target | Description | Base Dependencies |
|-------------|---------------------|-----------------------------------------------------------|---------------------------------------|
Expand Down Expand Up @@ -321,6 +283,7 @@ The folder structure mirrors other RAPIDS repos, with the following folders:
- `internal`: A private header-only component that hosts the code shared between benchmarks and tests.
- `scripts`: Helpful scripts for development
- `src`: Compiled APIs and template specializations for the shared libraries
- `template`: A skeleton template containing the bare-bones file structure and cmake configuration for writing applications with RAFT.
- `test`: Googletests source code
- `docs`: Source code and scripts for building library documentation (Uses breath, doxygen, & pydocs)
- `python`: Source code for Python libraries.
Expand Down
15 changes: 12 additions & 3 deletions build.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ ARGS=$*
# script, and that this script resides in the repo dir!
REPODIR=$(cd $(dirname $0); pwd)

VALIDARGS="clean libraft pylibraft raft-dask docs tests bench clean --uninstall -v -g -n --compile-lib --allgpuarch --no-nvtx --show_depr_warn -h"
VALIDARGS="clean libraft pylibraft raft-dask docs tests bench template clean --uninstall -v -g -n --compile-lib --allgpuarch --no-nvtx --show_depr_warn -h"
HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<tool>] [--limit-tests=<targets>] [--limit-bench=<targets>]
where <target> is:
clean - remove all existing build artifacts and configuration (start over)
Expand All @@ -29,6 +29,7 @@ HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<to
docs - build the documentation
tests - build the tests
bench - build the benchmarks
template - build the example RAFT application template
and <flag> is:
-v - verbose build mode
Expand Down Expand Up @@ -354,13 +355,12 @@ if (( ${NUMARGS} == 0 )) || hasArg libraft || hasArg docs || hasArg tests || has
-DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \
-DCMAKE_CUDA_ARCHITECTURES=${RAFT_CMAKE_CUDA_ARCHITECTURES} \
-DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
-DRAFT_COMPILE_LIBRARIES=${COMPILE_LIBRARIES} \
-DRAFT_COMPILE_LIBRARY=${COMPILE_LIBRARY} \
-DRAFT_NVTX=${NVTX} \
-DDISABLE_DEPRECATION_WARNINGS=${DISABLE_DEPRECATION_WARNINGS} \
-DBUILD_TESTS=${BUILD_TESTS} \
-DBUILD_BENCH=${BUILD_BENCH} \
-DCMAKE_MESSAGE_LOG_LEVEL=${CMAKE_LOG_LEVEL} \
-DRAFT_COMPILE_LIBRARY=${COMPILE_LIBRARY} \
${CACHE_ARGS} \
${EXTRA_CMAKE_ARGS}

Expand Down Expand Up @@ -410,3 +410,12 @@ if hasArg docs; then
cd ${SPHINX_BUILD_DIR}
sphinx-build -b html source _html
fi

################################################################################
# Initiate build for example RAFT application template (if needed)

if hasArg template; then
pushd cpp/template
./build.sh
popd
fi
5 changes: 5 additions & 0 deletions conda/recipes/libraft/build_libraft_template.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Copyright (c) 2022-2023, NVIDIA CORPORATION.

# Just building template so we verify it uses libraft.so and fail if it doesn't build
./build.sh template
36 changes: 36 additions & 0 deletions conda/recipes/libraft/meta.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -150,3 +150,39 @@ outputs:
home: https://rapids.ai/
license: Apache-2.0
summary: libraft tests
- name: libraft-template
version: {{ version }}
script: build_libraft_template.sh
build:
script_env: *script_env
number: {{ GIT_DESCRIBE_NUMBER }}
string: cuda{{ cuda_major }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }}
ignore_run_exports_from:
- {{ compiler('cuda') }}
requirements:
build:
- {{ compiler('c') }}
- {{ compiler('cuda') }} {{ cuda_version }}
- {{ compiler('cxx') }}
- cmake {{ cmake_version }}
- ninja
- sysroot_{{ target_platform }} {{ sysroot_version }}
host:
- {{ pin_subpackage('libraft', exact=True) }}
- {{ pin_subpackage('libraft-headers', exact=True) }}
- cuda-profiler-api {{ cuda_profiler_api_host_version }}
- libcublas {{ libcublas_host_version }}
- libcublas-dev {{ libcublas_host_version }}
- libcurand {{ libcurand_host_version }}
- libcurand-dev {{ libcurand_host_version }}
- libcusolver {{ libcusolver_host_version }}
- libcusolver-dev {{ libcusolver_host_version }}
- libcusparse {{ libcusparse_host_version }}
- libcusparse-dev {{ libcusparse_host_version }}
run:
- {{ pin_subpackage('libraft', exact=True) }}
- {{ pin_subpackage('libraft-headers', exact=True) }}
about:
home: https://rapids.ai/
license: Apache-2.0
summary: libraft template
38 changes: 38 additions & 0 deletions cpp/template/CMakeLists.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# =============================================================================
# Copyright (c) 2023, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under
# the License.

cmake_minimum_required(VERSION 3.23.1 FATAL_ERROR)

# ------------- configure rapids-cmake --------------#

include(cmake/thirdparty/fetch_rapids.cmake)
include(rapids-cmake)
include(rapids-cpm)
include(rapids-cuda)
include(rapids-export)
include(rapids-find)

# ------------- configure project --------------#

rapids_cuda_init_architectures(test_raft)

project(test_raft LANGUAGES CXX CUDA)

# ------------- configure raft -----------------#

rapids_cpm_init()
include(cmake/thirdparty/get_raft.cmake)

# -------------- compile tasks ----------------- #
add_executable(TEST_RAFT src/test_distance.cu)
target_link_libraries(TEST_RAFT PRIVATE raft::raft raft::compiled)
18 changes: 18 additions & 0 deletions cpp/template/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Example RAFT Project Template

This template project provides a drop-in sample to either start building a new application with, or using RAFT in an existing CMake project.

First, please refer to our [installation docs](https://docs.rapids.ai/api/raft/stable/build.html#cuda-gpu-requirements) for the minimum requirements to use RAFT.

Once the minimum requirements are satisfied, this example template application can be built with the provided `build.sh` script. This is a bash script that calls the appropriate CMake commands, so you can look into it to see the typical CMake based build workflow.

This directory (`RAFT_SOURCE/cpp/template`) can be copied directly in order to build a new application with RAFT.

RAFT can be integrated into an existing CMake project by copying the contents in the `configure rapids-cmake` and `configure raft` sections of the provided `CMakeLists.txt` into your project, along with `cmake/thirdparty/get_raft.cmake`.

Make sure to link against the appropriate Cmake targets. Use `raft::raft`to add make the headers available and `raft::compiled` when utilizing the shared library.

```cmake
target_link_libraries(your_app_target PRIVATE raft::raft raft::compiled)
```

41 changes: 41 additions & 0 deletions cpp/template/build.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/bin/bash

# Copyright (c) 2023, NVIDIA CORPORATION.

# raft empty project template build script

# Abort script on first error
set -e

PARALLEL_LEVEL=${PARALLEL_LEVEL:=`nproc`}

BUILD_TYPE=Release
BUILD_DIR=build/

RAFT_REPO_REL=""
EXTRA_CMAKE_ARGS=""
set -e


if [[ ${RAFT_REPO_REL} != "" ]]; then
RAFT_REPO_PATH="`readlink -f \"${RAFT_REPO_REL}\"`"
EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS} -DCPM_raft_SOURCE=${RAFT_REPO_PATH}"
fi

if [ "$1" == "clean" ]; then
rm -rf build
exit 0
fi

mkdir -p $BUILD_DIR
cd $BUILD_DIR

cmake \
-DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
-DRAFT_NVTX=OFF \
-DCMAKE_CUDA_ARCHITECTURES="NATIVE" \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
${EXTRA_CMAKE_ARGS} \
../

cmake --build . -j${PARALLEL_LEVEL}
21 changes: 21 additions & 0 deletions cpp/template/cmake/thirdparty/fetch_rapids.cmake
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# =============================================================================
# Copyright (c) 2023, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under
# the License.

# Use this variable to update RAPIDS and RAFT versions
set(RAPIDS_VERSION "23.04")

if(NOT EXISTS ${CMAKE_CURRENT_BINARY_DIR}/RAFT_RAPIDS.cmake)
file(DOWNLOAD https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-${RAPIDS_VERSION}/RAPIDS.cmake
${CMAKE_CURRENT_BINARY_DIR}/RAFT_RAPIDS.cmake)
endif()
include(${CMAKE_CURRENT_BINARY_DIR}/RAFT_RAPIDS.cmake)
62 changes: 62 additions & 0 deletions cpp/template/cmake/thirdparty/get_raft.cmake
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# =============================================================================
# Copyright (c) 2023, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under
# the License.

# Use RAPIDS_VERSION from cmake/thirdparty/fetch_rapids.cmake
set(RAFT_VERSION "${RAPIDS_VERSION}")
set(RAFT_FORK "rapidsai")
set(RAFT_PINNED_TAG "branch-${RAPIDS_VERSION}")

function(find_and_configure_raft)
set(oneValueArgs VERSION FORK PINNED_TAG COMPILE_LIBRARY ENABLE_NVTX ENABLE_MNMG_DEPENDENCIES)
cmake_parse_arguments(PKG "${options}" "${oneValueArgs}"
"${multiValueArgs}" ${ARGN} )

set(RAFT_COMPONENTS "")
if(PKG_COMPILE_LIBRARY)
string(APPEND RAFT_COMPONENTS " compiled")
endif()

if(PKG_ENABLE_MNMG_DEPENDENCIES)
string(APPEND RAFT_COMPONENTS " distributed")
endif()

#-----------------------------------------------------
# Invoke CPM find_package()
#-----------------------------------------------------
rapids_cpm_find(raft ${PKG_VERSION}
GLOBAL_TARGETS raft::raft
BUILD_EXPORT_SET raft-template-exports
INSTALL_EXPORT_SET raft-template-exports
COMPONENTS ${RAFT_COMPONENTS}
CPM_ARGS
GIT_REPOSITORY https://github.com/${PKG_FORK}/raft.git
GIT_TAG ${PKG_PINNED_TAG}
SOURCE_SUBDIR cpp
OPTIONS
"BUILD_TESTS OFF"
"BUILD_BENCH OFF"
"RAFT_NVTX ${ENABLE_NVTX}"
"RAFT_COMPILE_LIBRARY ${PKG_COMPILE_LIBRARY}"
)
endfunction()

# Change pinned tag here to test a commit in CI
# To use a different RAFT locally, set the CMake variable
# CPM_raft_SOURCE=/path/to/local/raft
find_and_configure_raft(VERSION ${RAFT_VERSION}.00
FORK ${RAFT_FORK}
PINNED_TAG ${RAFT_PINNED_TAG}
COMPILE_LIBRARY ON
ENABLE_MNMG_DEPENDENCIES OFF
ENABLE_NVTX OFF
)
Loading

0 comments on commit 9389108

Please sign in to comment.