Bifurcate Dependency Lists [skip-gpuci] #1065
Conversation
ajschmidt8
added
improvement
| packages: | ||
| - c-compiler | ||
| - cxx-compiler | ||
| - rapids-build-env=23.02.* |
There was a problem hiding this comment.
@ajschmidt8 Did you say this package was no longer needed on a different PR?
There was a problem hiding this comment.
Yes. For this PR, the intention is just to replace the manually managed conda/environment files with the dfg. I figured we can remove these packages in the follow-up GHA PR.
If we remove them before, it could break the environment. In the GHA PR, we can remove them and then identify which dependencies we need to re-add after they're removed.
There was a problem hiding this comment.
Okay, that sounds fine to me. We'll clean up later.
| common: | ||
| - output_types: [conda, requirements] | ||
| packages: | ||
| - clang=11.1.0 |
There was a problem hiding this comment.
I think we can drop clang/clang-tools, which means the entire develop list could be deleted. I don't think these are used anywhere. clang-format (the original reason this was added, afaik) is now handled via pre-commit.
There was a problem hiding this comment.
I believe @cjnolet said that they do/plan to run clang.
I notice this script among others https://github.com/rapidsai/raft/blob/branch-23.02/cpp/scripts/run-clang-tidy.py
There was a problem hiding this comment.
I think only clang-tools is required for providing the clang-tidy binary. No scripts are calling clang as far as I can tell.
There was a problem hiding this comment.
We are in the process of updating our code so we can support clang. I'd prefer to leave in the dependency in the meantime (we do have an experimental clang compile script in there that we're working on making build successfully).
| - output_types: [conda] | ||
| packages: | ||
| - doxygen>=1.8.20 | ||
| - rapids-doc-env=23.02.* |
There was a problem hiding this comment.
Likewise, is this still needed?
| common: | ||
| - output_types: [conda] | ||
| packages: | ||
| - rapids-notebook-env=23.02.* |
| - ucx>=1.13.0 | ||
| - ucx-py=0.30.* | ||
| - ucx-proc=*=gpu | ||
| - libfaiss>=1.7.0 |
There was a problem hiding this comment.
I noticed a really annoying issue recently in rapids-compose. It merges the dependency lists / conda channels, and ends up placing the pytorch channel (which is used in cudf tests) above conda-forge. This in turn means that libfaiss is pulled from pytorch and it ignores the faiss-proc=*=cuda package below, which only affects libfaiss from conda-forge. 🙃 Just something to be aware of. We may need to specify libfaiss>=1.7.0=cuda* or something like that, in addition to faiss-proc=*=cuda. Another solution would be to fix the conda-forge packages of pytorch and never use the pytorch channel for any dependencies.
There was a problem hiding this comment.
I just ran into the same issue and fixed it in the old env files, but then I realized this PR was ready to go. I'll make a PR with the additional pinning, which is how I fixed it in my case and it worked fine.
This PR uses the
rapids-dependency-file-generatorto handle sourcing dependencies.Similarly to rapidsai/rmm#1073 and rapidsai/cudf#11674, this PR introduces a GitHub Action that enforces consistency between the new
dependencies.yamlfile and the generated conda environment for developers.