Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepare parts of the balanced kmeans for ivf-pq #788

Merged
merged 11 commits into from
Sep 2, 2022
Merged

Prepare parts of the balanced kmeans for ivf-pq #788

merged 11 commits into from
Sep 2, 2022

Conversation

achirkin
Copy link
Contributor

@achirkin achirkin commented Aug 11, 2022 •
edited
Loading

Backport a piece of implementation from cuann/ivf-pq. This include:

  1. Move part of the internal kmeans code to GPU, thus reducing the number of necessary managed memory allocations;
  2. Split some of the functions into smaller reusable pieces;
  3. Move dataset-trainset sampling to the caller side.

@achirkin achirkin requested a review from a team as a code owner August 11, 2022 08:54
@github-actions github-actions bot added the cpp label Aug 11, 2022
@achirkin achirkin added 3 - Ready for Review improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Aug 11, 2022
@tfeher tfeher self-assigned this Aug 11, 2022
@achirkin
Copy link
Contributor Author

rerun tests

@achirkin
Copy link
Contributor Author

rerun tests

@achirkin achirkin mentioned this pull request Aug 15, 2022
Merged
tfeher
tfeher approved these changes Aug 15, 2022
View reviewed changes
Copy link
Contributor

@tfeher tfeher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Artem for the PR, it is nice to get this piece on the GPU! I have two small comments, otherwise it looks good to me. Pre-approving.

achirkin added a commit to achirkin/raft that referenced this pull request Aug 24, 2022
@achirkin
Use the new functions from rapidsai#788
1bae646
@achirkin achirkin requested a review from tfeher August 24, 2022 06:33
@achirkin
Copy link
Contributor Author

achirkin commented Aug 24, 2022 •
edited
Loading

Tamas, thanks for the review! I'm re-requesting the review, because I need to add more changes to reduce duplication and facilitate re-use in both ivf-pq and ivf-flat.

@achirkin achirkin changed the title Port a piece of the balanced kmeans computation to GPU Prepare parts of the balanced kmeans from ivf-pq Aug 24, 2022
@achirkin achirkin changed the title Prepare parts of the balanced kmeans from ivf-pq Prepare parts of the balanced kmeans for ivf-pq Aug 24, 2022
tfeher
tfeher requested changes Aug 30, 2022
View reviewed changes
Copy link
Contributor

@tfeher tfeher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Artem for the update! Looks good, apart from a few minor clarification.

@achirkin achirkin requested a review from tfeher August 30, 2022 11:53
tfeher
tfeher approved these changes Aug 30, 2022
View reviewed changes
Copy link
Contributor

@tfeher tfeher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @achirkin for the clarification. LGTM.

@achirkin achirkin mentioned this pull request Sep 1, 2022
Merged
@cjnolet
Copy link
Member

cjnolet commented Sep 2, 2022

@gpucibot merge

@rapids-bot rapids-bot bot merged commit e972431 into rapidsai:branch-22.10 Sep 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tfeher tfeher tfeher approved these changes

Assignees

@tfeher tfeher

Labels
3 - Ready for Review cpp improvement Improvement / enhancement to an existing function non-breaking Non-breaking change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@achirkin @cjnolet @tfeher