Increase parallelism in allgatherv #525
Conversation
seunghwak
added
3 - Ready for Review
improvement
There was a problem hiding this comment.
I think there's a limit on the number of calls inside a group, quoting the document from nccl:
Also note, that there is a maximum of 2048 NCCL operations that can be inserted between the ncclGroupStart and ncclGroupEnd calls.
Thanks, yeah... I don't think cuGraph will ever hit this limit as we run allgatherv on sub-communicator (unless we will work on multi-million GPUs), but this can definitely happen in other use cases working no the global communicator. I will make an update to accommodate this. |
No, so @trivialfis said, this code can fail if someone runs allgatherv with more than 2048 GPUs. I guess no one will do this in short-term, but better address this now to be future proof. |
|
Maybe just a check since this will not happen in foreseeable future. ;-) |
|
OK, done, now I believe this PR is ready to be merged. |
|
@gpucibot merge |
|
rerun tests |
|
@gpucibot merge |
allgatherv is implemented using multiple NCCL broadcast operations.
Previously, RAFT performed these broadcast operations sequentially creating a hot-spot around the root node in each broadcast operations.
These PR places multiple broadcast operations inside ncclGroupStart and ncclGroupEnd increasing the parallelism and more evenly stressing the communication interconnect.