Add support for Python 3.12 #40
Description
Overview
Python 3.12 was released in October 2023. This issue tracks the work to add support for Python 3.12 to RAPIDS.
In #3, RAPIDS added support for Python 3.11, which was released in RAPIDS 24.04. The work to add Python 3.11 was heavily automated, and that could be done again for Python 3.12 to reduce the load on RAPIDS maintainers.
When should we drop Python 3.9?
Typically RAPIDS has kept the matrix of supported Python minor versions to 2 or 3 versions at a time. When adding Python 3.12, we should probably drop Python 3.9 as well.
SPEC 0 recommended dropping support for Python 3.9 in 2023Q4. Meanwhile, NEP 29 recommended dropping support for Python 3.9 as of Apr 05, 2024. Both of these deadlines have passed and several large Python libraries are now moving towards dropping Python 3.9, so it is probably reasonable to drop Python 3.9 around the same time that we add Python 3.12.
Tasks
Each section should be fully completed before moving to the next section.
CI images
- Add Python 3.12 builds of miniforge-cuda. Drop CUDA 11.2 and 11.5, add Python 3.11 and 3.12. miniforge-cuda#55
- Add Python 3.12 CI images (
ci-conda,ci-wheel,citestwheel) Add Python 3.12 ci-imgs#137
CI workflows
Branch Strategy:
- Create a branch on
shared-workflowscalledpython-3.12 - Add Python 3.12 to the build matrix on the
python-3.12branch - Add Python 3.12 to the test matrix on the
python-3.12branch- When adjusting the test matrix, be aware of total GPU resource consumption. Build jobs are CPU only but test jobs require GPUs. We want to keep our GPU consumption roughly the same (don't double the test matrix size), even if it gets a bit sparser in its coverage. We have some rough guidelines for how to decide on the matrix entries to include.
The above tasks are handled by: rapidsai/shared-workflows#213
RAPIDS repositories
This list is intentionally in RAPIDS dependency order... they'll need to be completed in roughly that order.
### Updates for libraries
- [x] pin Python version to 3.11 in `devcontainers` until all repos are migrated (https://github.com/rapidsai/devcontainers/pull/388)
- [x] update `pypi-wheel-scripts` and manually build/publish `cubinlinker` and `ptxcompiler` wheels (https://github.com/rapidsai/pypi-wheel-scripts/pull/17)
- [x] rapids-cmake (https://github.com/rapidsai/rapids-cmake/pull/688)
- [x] rmm (https://github.com/rapidsai/rmm/pull/1666)
- [x] kvikio (https://github.com/rapidsai/kvikio/pull/458)
- [x] pynvjitlink (already supported Python 3.12: https://github.com/rapidsai/pynvjitlink/pull/104#issuecomment-2330079484)
- [x] dask-cuda (https://github.com/rapidsai/dask-cuda/pull/1380, https://github.com/rapidsai/dask-cuda/pull/1382)
- [x] cudf (https://github.com/rapidsai/cudf/pull/16745)
- [x] ucx-py (https://github.com/rapidsai/ucx-py/pull/1067)
- [x] ucxx (https://github.com/rapidsai/ucxx/pull/276)
- [x] raft (https://github.com/rapidsai/raft/pull/2428)
- [x] wholegraph (https://github.com/rapidsai/wholegraph/pull/214)
- [x] cugraph (https://github.com/rapidsai/cugraph/pull/4647)
- [x] cugraph-gnn (https://github.com/rapidsai/cugraph-gnn/pull/41)
- [x] cugraph-pg (pushed Python 3.12 changes to https://github.com/rapidsai/cugraph-pg/pull/25 ... not waiting for that to be merged, as that repo's not being actively maintained yet)
- [x] cuml (https://github.com/rapidsai/cuml/pull/6060)
- [x] cuvs (https://github.com/rapidsai/cuvs/pull/312)
- [x] cuspatial (https://github.com/rapidsai/cuspatial/pull/1453)
- [x] cuxfilter (https://github.com/rapidsai/cuxfilter/pull/625)
- [x] cucim (https://github.com/rapidsai/cucim/pull/773)
- [x] private repos
- [x] integration (https://github.com/rapidsai/integration/pull/719)
- [x] cuopt (https://github.com/rapidsai/cuopt/pull/2006)
For each repo,
- Update
.github/workflows/*.yamlto point to thepython-3.12branch ofshared-workflows - Update
dependencies.yamlto add support for Python 3.12. - Review any
pyproject.tomlfiles for necessary changes (classifiers, etc.) - Update docs (README, etc) that reference a single Python version to point to the latest (3.12).
- Once CI passes, merge the PR.
Most of this is easy to automate with rapids-reviser, and we can copy from this previous migrator for Python 3.11: https://github.com/rapidsai/rapids-reviser/pull/11. We still need to manually review the PRs for missing pieces.
Once all repos are migrated to the python-3.12 branch, the migration is complete. We merge python-3.12 into the development branch on shared-workflows and then open follow-up PRs to each repo to reset the branches to that development branch. This "reset" is simple and should be automated with rapids-reviser.
### Post-migration
- [x] Update the `ci-imgs` repo's `latest` configuration to use Python 3.12. (https://github.com/rapidsai/ci-imgs/pull/188)
- [x] Update the `docker` repo's build and test matrices, and references in docs (https://github.com/rapidsai/docker/pull/711)
- [x] Update the build matrix in `pypi-wheel-scripts` so that Python 3.12 wheels are uploaded (https://github.com/rapidsai/pypi-wheel-scripts/pull/17)
- [x] Update `rapidsai/dask-build-environment` (https://github.com/rapidsai/dask-build-environment/pull/102)
- [x] update release selector in `docs` (https://github.com/rapidsai/docs/pull/539)
- [x] update all the repos to point at `branch-24.10` on `shared-workflows` again (blocked by https://github.com/rapidsai/cuopt/pull/2006)
- [x] move `devcontainers` Python version pin to Python 3.12 (https://github.com/rapidsai/devcontainers/pull/391)
- [x] (after updating all the repos) delete the `python-3.12` branch in `shared-workflows`
- [x] merge any remaining `rapids-reviser` changes (https://github.com/rapidsai/rapids-reviser/pull/52)
Notes:
The :latest image from ci-imgs is frequently used by CI jobs for building docs and testing notebooks. Be aware that issues may arise in those jobs.