Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use rapids-cmake new parallel testing feature #1623

Merged
merged 10 commits into from
Jul 21, 2023

Conversation

robertmaynard
Copy link
Contributor

Corrects multiple test issues uncovered by running tests in parallel. Splits the matrix and neighbor tests into multiple executables to improve throughput

Previous performance on 1 GPU: 830sec
New performance on 1 GPU: 692sec
New performance on 2 GPU: 418sec

@robertmaynard robertmaynard added feature request New feature or request non-breaking Non-breaking change labels Jun 29, 2023
@robertmaynard robertmaynard requested review from a team as code owners June 29, 2023 18:33
cpp/test/distance/gram.cu Outdated Show resolved Hide resolved
cpp/test/matrix/select_k.cuh Outdated Show resolved Hide resolved
cpp/test/neighbors/selection.cu Show resolved Hide resolved
cpp/test/neighbors/selection.cu Show resolved Hide resolved
// Copy data to host
RAFT_CUDA_TRY(cudaStreamSynchronize(s)); //ensure stream is done before copy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use a cudaMemcpyAsync below instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we can, and done.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use raft::copy and raft::update_host instead of calling cuda directly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no existing raft calls ( or handles ) in this test. The original version was calling cudaMemcpy on memory being written to by a kernel on a different stream.

Corrects multiple test issues uncovered by running tests in parallel.
Splits the matrix and neighbor tests into multiple executables to
improve throughput

Previous performance on 1 GPU: 830sec
New performance on 1 GPU: 692sec
New performance on 2 GPU: 418sec
@robertmaynard robertmaynard force-pushed the fea/parallel_testing branch from 29e5659 to aa0b622 Compare July 5, 2023 13:50
@robertmaynard robertmaynard removed their assignment Jul 5, 2023
@robertmaynard robertmaynard requested a review from vyasr July 5, 2023 13:51
Copy link
Member

@cjnolet cjnolet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Member

@cjnolet cjnolet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@cjnolet
Copy link
Member

cjnolet commented Jul 21, 2023

/merge

@rapids-bot rapids-bot bot merged commit 9884f69 into rapidsai:branch-23.08 Jul 21, 2023
@robertmaynard robertmaynard deleted the fea/parallel_testing branch November 1, 2024 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci CMake cpp feature request New feature or request non-breaking Non-breaking change
Projects
Development

Successfully merging this pull request may close these issues.

4 participants