add CUDA 12.5 images #689

jameslamb · 2024-07-16T20:30:03Z

Contributes to rapidsai/build-planning#73

Proposes the following:

adding CUDA 12.5 images
PR builds:
- +1 test job covering (cuda=12.5, arch=arm64)
branch builds:
- +1 test job covering (cuda=12.5, arch=x86_64)
- -1 test job covering (cuda=12.2, arch=arm64)

context: rapidsai/build-planning#73 (comment)

Notes for Reviewers

Per offline discussion with @raydouglass , this would be accompanied by a deprecation notice in the RAPIDS 24.08 release stating that the CUDA 12.2 images will be removed in some future release (future release not yet determined).

jakirkham

Thanks James! 🙏

Am curious what you think about swapping where the x86_64 and aarch64 jobs are

For more context CUDA 12.2 added some interesting things like cuFile (for ARM) and HMM, which are helpful to test on CUDA 12.2 ARM specifically. Am not aware of a similar change in CUDA 12.5 (though could be missing it)

Given there are fewer ARM machines than x86_64, it may make sense to use x86_64 primarily in PRs and run ARM more on branches. That would mean less ARM jobs are needed during PR development

Anyways please let me know what you think 🙂

matrix-test.yaml

Co-authored-by: jakirkham <[email protected]>

matrix-test.yaml

jameslamb · 2024-07-16T22:02:43Z

Thanks @jakirkham ! That's helpful context I didn't have. I just applied your suggestions.

jakirkham · 2024-07-18T18:31:25Z

Updating to pull in PR ( #690 ) now that is in branch-24.08

jameslamb · 2024-07-18T19:24:50Z

See this

9.818 Running dfg on cuspatial
9.939 Traceback (most recent call last):
9.939   File "/opt/conda/bin/rapids-dependency-file-generator", line 10, in <module>
9.939     sys.exit(main())
9.939              ^^^^^^
9.939   File "/opt/conda/lib/python3.11/site-packages/rapids_dependency_file_generator/_cli.py", line 121, in main
9.939     make_dependency_files(
9.939   File "/opt/conda/lib/python3.11/site-packages/rapids_dependency_file_generator/_rapids_dependency_file_generator.py", line 421, in make_dependency_files
9.940     raise ValueError(f"No matching matrix found in '{include}' for: {matrix_combo}")
9.940 ValueError: No matching matrix found in 'cuda_version' for: {'cuda': '12.5', 'arch': 'x86_64', 'py': '3.9'}

(build link)

This will be broken until rapidsai/cuspatial#1405 is merged. And that is blocked until CI in cuspatial is fixed (which could be done by merging rapidsai/cuspatial#1407).

KyleFromNVIDIA

Just one quick question. Approved so it doesn't block merging.

Dockerfile

jakirkham · 2024-07-19T20:08:07Z

Dockerfile

 ARG LINUX_DISTRO=ubuntu
 ARG LINUX_DISTRO_VER=22.04


This need not be done in this PR. Just the discussion above ( #689 (comment) ) made me wonder

Should we do the same thing for the Linux version?

Suggested change

ARG LINUX_DISTRO=ubuntu

ARG LINUX_DISTRO_VER=22.04

ARG LINUX_DISTRO=unset

ARG LINUX_DISTRO_VER=unset

If this sounds reasonable, happy to send a 2nd PR with this change. Also ok not doing this if we don't think it makes sense

Yes in my opinion, this is a pattern that should be used in any Dockerfile where it's unacceptable to silently fall back to default values if configuration is not passed.

I just didn't want to go any further with it in this PR and grow the diff here.

I'd support a 2nd PR with this change.

jakirkham · 2024-07-19T22:47:49Z

One of the 11.8.0 tests failed on CI. Restarted it

That said, we are not changing 11.8.0 builds here and all other builds pass

Edit: Actually that won't work since the images get deleted after

jakirkham · 2024-07-20T00:20:41Z

Going to try restarting from scratch. If that doesn't work, we can discuss next steps on Monday

raydouglass · 2024-07-20T03:34:53Z

/merge

jakirkham · 2024-07-20T03:58:22Z

Thanks all! 🙏

Contributes to #832 Removes reliance on https://hub.docker.com/r/rapidsai/ci-conda in this repo. It was only being used in 2 places, both of which could be run in simpler environments: * in a `compute-matrix` job that just needs `bash`, `jq` and `yq` --> regular old `ubuntu-latest` GitHub Actions VM is fine * in a part of a multi-stage build that just needs `bash`, `git`, `pip`, and a Python interpreter --> `python:{version}` container is fine Other changes based on PR reviewers: * puts versions to be managed by renovate in a `versions.yaml` ([borrowing the approach from rapidsai/ci-imgs](https://github.com/rapidsai/ci-imgs/blob/28b8b5ac127bcf5cd6679cd8ee40a53ce7240421/ci/compute-build-args.sh#L36)) * updates shell code in Dockerfiles to match our preferred style (including the array format for package lists from rapidsai/ci-imgs#343) * sets all empty arg defaults to `notset` (ref: #689 (comment)) - *weakly prefer `notset` to `unset` because `unset` is a valid shell command* * alphabetizes build args in Dockerfiles ## Notes for Reviewers ### This is only the beginning To be clear, this project still heavily relies on `rapidsai/miniforge-cuda` (the base image for `rapidsai/ci-conda`. So there's more work to do for #832. But this at least means we no longer need to care about anything in these lines breaking the things in this repo 😁 : https://github.com/rapidsai/ci-imgs/blob/55b24734b7d594e768599771dcae2e718af52f9a/ci-conda.Dockerfile#L183-L313 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #834

add CUDA 12.5 images

677117d

jameslamb added the 2 - In Progress Currenty a work in progress label Jul 16, 2024

jameslamb requested review from KyleFromNVIDIA, jakirkham and raydouglass July 16, 2024 20:30

jameslamb requested a review from a team as a code owner July 16, 2024 20:30

jameslamb mentioned this pull request Jul 16, 2024

Update latest CUDA version for build/test to 12.5 rapidsai/build-planning#73

Closed

13 tasks

jakirkham reviewed Jul 16, 2024

View reviewed changes

matrix-test.yaml Outdated Show resolved Hide resolved

matrix-test.yaml Outdated Show resolved Hide resolved

Apply suggestions from code review

d2e18fb

Co-authored-by: jakirkham <[email protected]>

jameslamb commented Jul 16, 2024

View reviewed changes

matrix-test.yaml Outdated Show resolved Hide resolved

Update matrix-test.yaml

adf7816

Merge branch 'branch-24.08' into cuda-12.5.1

1f8045f

jameslamb mentioned this pull request Jul 18, 2024

fix return-before-assignment bug in notebook test script, add pre-commit, drop cuspatial notebooks #690

Merged

Merge branch 'branch-24.08' into cuda-12.5.1

58af07a

raydouglass approved these changes Jul 19, 2024

View reviewed changes

KyleFromNVIDIA approved these changes Jul 19, 2024

View reviewed changes

Dockerfile Show resolved Hide resolved

jakirkham reviewed Jul 19, 2024

View reviewed changes

jakirkham approved these changes Jul 20, 2024

View reviewed changes

rapids-bot bot merged commit d240fc3 into rapidsai:branch-24.08 Jul 20, 2024

jameslamb deleted the cuda-12.5.1 branch July 22, 2024 04:15

jameslamb mentioned this pull request Jul 24, 2024

start using CUDA 12.5, Python 3.11 images in nightly docs rapidsai/deployment#398

Merged

This was referenced Aug 27, 2024

install selector: Drop python 3.9 for nightlies, default to CUDA 12.5 for stable and nightly rapidsai/docs#534

Merged

Build args in dockerfiles should not use realistic values rapidsai/dask-build-environment#100

Closed

jameslamb mentioned this pull request Jul 14, 2025

Add nightly container support NVIDIA/cuopt#180

Merged

8 tasks

jameslamb mentioned this pull request Jan 7, 2026

remove reliance on rapidsai/ci-conda, use a versions.yaml #834

Merged

add CUDA 12.5 images #689

add CUDA 12.5 images #689

Uh oh!

Conversation

jameslamb commented Jul 16, 2024

Notes for Reviewers

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jameslamb commented Jul 16, 2024

Uh oh!

jakirkham commented Jul 18, 2024

Uh oh!

jameslamb commented Jul 18, 2024

Uh oh!

KyleFromNVIDIA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakirkham Jul 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jameslamb Jul 19, 2024

Choose a reason for hiding this comment

Uh oh!

jakirkham commented Jul 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakirkham commented Jul 20, 2024

Uh oh!

raydouglass commented Jul 20, 2024

Uh oh!

jakirkham commented Jul 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jakirkham Jul 19, 2024 •

edited

Loading

jakirkham commented Jul 19, 2024 •

edited

Loading