Simplify caching mechanisms for CI and PROD images (canary) #45266

potiuk · 2024-12-29T11:51:53Z

For a long time we had used a sophisticated mechanism to speed up our CI jobs by building the images in "pull_request_target" workflow and pushing them to GitHub registry. That however had several drawbacks:

CI image was complex when it comes to layer setup (we had to pre- cache installed dependencies by installing them from branch tip
The pull_request_target is a very dangerous workflow, we had a number of security problems with it (and it's difficult to debug)
Caching of pip and uv was not used because it increased size of the image significantly

This PR significantly improves the caching mechanisms for the images building of several advacements that were not possible before:

The upload-artifacts@v4 action and improved stash action developed by @assignUser and published in "apache/infrastructure-actions" allows us to store all images (8GB per run) in artifacts rather than in registry - so we can do the image build once and share it with all the jobs.
The uv speed is "enough" to allow occasional installation of Airlfow locally. This allows to utilize cache-mount and locally build uv cache, rather than rely on "remote" cache when we are building local images for breeze. The first time you build local breeze image it will take 2-5 more minutes (depending on your network speed, but because we can utilise cache mounts, every subsequent build should be very fast - even if all dependencies change. Using uv also allows to "always" reinstall airflow when you build the image even if single source file changed, because with cache it takes sub-seconds to reinstall airflow and all dependencies.
the cache mounts are not included in the image size, and since we can export and import images in CI in artifacts and we do not need to rebuild them, the images shared as compressed artifacts are relatively small (2GB) - cache of uv is around 4GB on top of that so sharing image built in the "build image" job with other jobs in the same workflow is fast.
we are still using registry cache for the "non-python" parts of the image - both CI and breeze image build speed benefit from using the image cache for system dependencies, database clients etc.
documentation has been updated to reflect the new CI setup. The diagrams showing the workflows of ours are no longer needed as the workflows are quite straightforward when they are looked at.

Fixes: #42999
Fixes: #43268

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

.github/actions/checkout_target_commit/action.yml

.github/actions/prepare_all_ci_images/action.yml

.github/workflows/additional-ci-image-checks.yml

.github/workflows/build-images.yml

.github/workflows/ci-image-build.yml

.github/workflows/ci.yml

.github/workflows/k8s-tests.yml

dev/breeze/doc/06_managing_docker_images.rst

dev/breeze/doc/ci/06_diagrams.md

scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh

scripts/ci/images/ci_stop_arm_instance.sh

scripts/docker/common.sh

scripts/docker/entrypoint_ci.sh

scripts/docker/install_airflow.sh

This is a bulk change that synchronizes dev/ci scripts for v2-10-test branch with main #45266 - including follow-ups. Rather than cherry-picking relevant PRs, this one gets the latest version of the scripts from main and updates the branch with some changes to adapt them to v2-10-test (such as bringing back python 3.8 support, removing some providers checks after the bulk move of providers and making sure all tests are passing. This is far easier than cherry-picking the changes, because for the v2-10-test we stopped cherry-picking CI changes which was deemed unnecessary (we used to do it for all previous branches) but this made it far more difficult (if not impossible) to cherry-pick individual changes. Fortunately, the CI scripts are maintained in the way that their latest version **should** in principle work for a v2-* branch and hopefully after just a few adjustments we should be able to synchronize the changes from main by updating all relevant CI/DEV scripts, dockerfile images, workflows, pre-commits etc. Add actions in codeql workflows to scan github workflow actions (#45534) * add actions in codeql workflows to scan github workflow actions * add actions in codeql workflows to scan github workflow actions CodeQL scanning can run always on all code (#45541) The CodeQL scannig is fast and having custom configuration to select which scanning to run should be run makes it unnecessarily complex We can just run all CodeQL scans always. This has been suggested by actions codeql scan itself. Add explicit permissions for all workflow-run workflows (#45548) Those workflows inherit permissions from the calling workflows but it's good to add explicit permissions to indicate what is needed and in case we will also use the workflows for other purposes in the future - default permissions for older repos might be write so it's best to be explicit about the permissions. Found by CodeQL scanning Remove contents: write permission from generate-constraints (#45558) The write permission cannot be set for PRs from forks in the call workflow - so we have to come back to implicit permissions and make explicit permissions passing a bit differently. (cherry picked from commit ae32ebc) Bump trove-classifiers from 2025.1.7.14 to 2025.1.10.15 (#45561) Bumps [trove-classifiers](https://github.com/pypa/trove-classifiers) from 2025.1.7.14 to 2025.1.10.15. - [Release notes](https://github.com/pypa/trove-classifiers/releases) - [Commits](pypa/trove-classifiers@2025.1.7.14...2025.1.10.15) --- updated-dependencies: - dependency-name: trove-classifiers dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit f3fd262) Add optional --image-file-dir to store loaded files elsewhere (#45564) While backorting the "pull_request_target" removal to v2-10-test branches it turned out that there is not enough disk space on Public runner to load all 5 images and keep the file dump at the same time in the same filesystem. This PR allows to choose where the load/save files will be stored and in the github runner environment we store the files in "/mnt" wnich is a separate folder with 40GB free. (cherry picked from commit 6628049)

This is a bulk change that synchronizes dev/ci scripts for v2-10-test branch with main #45266 - including follow-ups. Rather than cherry-picking relevant PRs, this one gets the latest version of the scripts from main and updates the branch with some changes to adapt them to v2-10-test (such as bringing back python 3.8 support, removing some providers checks after the bulk move of providers and making sure all tests are passing. This is far easier than cherry-picking the changes, because for the v2-10-test we stopped cherry-picking CI changes which was deemed unnecessary (we used to do it for all previous branches) but this made it far more difficult (if not impossible) to cherry-pick individual changes. Fortunately, the CI scripts are maintained in the way that their latest version **should** in principle work for a v2-* branch and hopefully after just a few adjustments we should be able to synchronize the changes from main by updating all relevant CI/DEV scripts, dockerfile images, workflows, pre-commits etc. Add actions in codeql workflows to scan github workflow actions (#45534) * add actions in codeql workflows to scan github workflow actions * add actions in codeql workflows to scan github workflow actions CodeQL scanning can run always on all code (#45541) The CodeQL scannig is fast and having custom configuration to select which scanning to run should be run makes it unnecessarily complex We can just run all CodeQL scans always. This has been suggested by actions codeql scan itself. Add explicit permissions for all workflow-run workflows (#45548) Those workflows inherit permissions from the calling workflows but it's good to add explicit permissions to indicate what is needed and in case we will also use the workflows for other purposes in the future - default permissions for older repos might be write so it's best to be explicit about the permissions. Found by CodeQL scanning Remove contents: write permission from generate-constraints (#45558) The write permission cannot be set for PRs from forks in the call workflow - so we have to come back to implicit permissions and make explicit permissions passing a bit differently. (cherry picked from commit ae32ebc) Bump trove-classifiers from 2025.1.7.14 to 2025.1.10.15 (#45561) Bumps [trove-classifiers](https://github.com/pypa/trove-classifiers) from 2025.1.7.14 to 2025.1.10.15. - [Release notes](https://github.com/pypa/trove-classifiers/releases) - [Commits](pypa/trove-classifiers@2025.1.7.14...2025.1.10.15) --- updated-dependencies: - dependency-name: trove-classifiers dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit f3fd262) Add optional --image-file-dir to store loaded files elsewhere (#45564) While backorting the "pull_request_target" removal to v2-10-test branches it turned out that there is not enough disk space on Public runner to load all 5 images and keep the file dump at the same time in the same filesystem. This PR allows to choose where the load/save files will be stored and in the github runner environment we store the files in "/mnt" wnich is a separate folder with 40GB free. (cherry picked from commit 6628049) Fix --from-pr feature for image load and stabilize help This is a follow-up after #45564 - it fixes the `--from-pr` and `--from-run` to work (it was failing with file does not exist). Also found out that gettempdir might return different directory depending on which is your designated tmp directory (for example in MacOS this is is a longer path in /var/.....) - so we have to force the default during help generation to always return "/tmp" so that the --help images do not change depending on which system you are and what your tmp directory is.

@assignUser

For a long time we had used a sophisticated mechanism to speed up our CI jobs by building the images in "pull_request_target" workflow and pushing them to GitHub registry. That however had several drawbacks: * CI image was complex when it comes to layer setup (we had to pre- cache installed dependencies by installing them from branch tip * The pull_request_target is a very dangerous workflow, we had a number of security problems with it (and it's difficult to debug) * Caching of `pip` and `uv` was not used because it increased size of the image significantly This PR significantly improves the caching mechanisms for the images building of several advacements that were not possible before: * The upload-artifacts@v4 action and improved stash action developed by @assignUser and published in "apache/infrastructure-actions" allows us to store all images (8GB per run) in artifacts rather than in registry - so we can do the image build once and share it with all the jobs. * The uv speed is "enough" to allow occasional installation of Airlfow locally. This allows to utilize cache-mount and locally build uv cache, rather than rely on "remote" cache when we are building local images for breeze. The first time you build local breeze image it will take 2-5 more minutes (depending on your network speed, but because we can utilise cache mounts, every subsequent build should be very fast - even if all dependencies change. Using uv also allows to "always" reinstall airflow when you build the image even if single source file changed, because with cache it takes sub-seconds to reinstall airflow and all dependencies. * the cache mounts are not included in the image size, and since we can export and import images in CI in artifacts and we do not need to rebuild them, the images shared as compressed artifacts are relatively small (2GB) - cache of `uv` is around 4GB on top of that so sharing image built in the "build image" job with other jobs in the same workflow is fast. * we are still using registry cache for the "non-python" parts of the image - both CI and breeze image build speed benefit from using the image cache for system dependencies, database clients etc. this helps with faster rebuilds of the images for local development environment * documentation has been updated to reflect the new CI setup. The diagrams showing the workflows of ours are no longer needed as the workflows are quite straightforward when they are looked at. Fixes: apache#42999 Fixes: apache#43268

…#45281) In very simple changes we should skip building ci image altogether. This had been missing condition in one of the earlier refactors. Follow-up after apache#45266

Packaging tools (uv and pip) do not have to use mounted cache as they are always reinstalled when their versions got updated in the Dockerfile, and changing sources of airflow does not invalidate the installation step (COPY . is made after the installation). This will add two more layers being cached in github registry when the image is built. Follow-up after apache#45266

We are using various caches in our build and so far - due to the way how "standard" caching works, PRs from forks could not effectively use the cache from main Airflow repository - because caches are not shared with other repositories - so the PRs builds could only use cache effectively when they were rebased and continued running from the same fork. This PR improves caching strategy using "stash" action from the ASF. Unlike `cache` - the action uses artifacts to store cache, and that makes it possible for the stash action to use such cache uploaded from `main` canary builds in PRs coming from the fork. As part of this change all the places where setup-python was used and breeze installed afterwards were reviewed and updated to use only breeze installation action (it already installs python) and this action has been improved to use UV caching effectively. Overall this PR should decrease setup overhead for many jobs across the CI workflow. Follow-up after apache#45266

Prod image build sometimes (quite often) is not needed. When removing `pull_request_target` in apache#45266 `wait-for=prod-images` had the condition that prevented it from running (and the `build-prod-images` step depended on it) - but this condition is gone now. Instead of preventing the whole composite workflow from running, we are adding it to "build-prod-packages" so that the whole workflow can complete as prerequisite to "finalize-tests" which should be executed regardless from `prod-image-build` being executed.

@assignUser

For a long time we had used a sophisticated mechanism to speed up our CI jobs by building the images in "pull_request_target" workflow and pushing them to GitHub registry. That however had several drawbacks: * CI image was complex when it comes to layer setup (we had to pre- cache installed dependencies by installing them from branch tip * The pull_request_target is a very dangerous workflow, we had a number of security problems with it (and it's difficult to debug) * Caching of `pip` and `uv` was not used because it increased size of the image significantly This PR significantly improves the caching mechanisms for the images building of several advacements that were not possible before: * The upload-artifacts@v4 action and improved stash action developed by @assignUser and published in "apache/infrastructure-actions" allows us to store all images (8GB per run) in artifacts rather than in registry - so we can do the image build once and share it with all the jobs. * The uv speed is "enough" to allow occasional installation of Airlfow locally. This allows to utilize cache-mount and locally build uv cache, rather than rely on "remote" cache when we are building local images for breeze. The first time you build local breeze image it will take 2-5 more minutes (depending on your network speed, but because we can utilise cache mounts, every subsequent build should be very fast - even if all dependencies change. Using uv also allows to "always" reinstall airflow when you build the image even if single source file changed, because with cache it takes sub-seconds to reinstall airflow and all dependencies. * the cache mounts are not included in the image size, and since we can export and import images in CI in artifacts and we do not need to rebuild them, the images shared as compressed artifacts are relatively small (2GB) - cache of `uv` is around 4GB on top of that so sharing image built in the "build image" job with other jobs in the same workflow is fast. * we are still using registry cache for the "non-python" parts of the image - both CI and breeze image build speed benefit from using the image cache for system dependencies, database clients etc. this helps with faster rebuilds of the images for local development environment * documentation has been updated to reflect the new CI setup. The diagrams showing the workflows of ours are no longer needed as the workflows are quite straightforward when they are looked at. Fixes: apache#42999 Fixes: apache#43268

…#45281) In very simple changes we should skip building ci image altogether. This had been missing condition in one of the earlier refactors. Follow-up after apache#45266

Packaging tools (uv and pip) do not have to use mounted cache as they are always reinstalled when their versions got updated in the Dockerfile, and changing sources of airflow does not invalidate the installation step (COPY . is made after the installation). This will add two more layers being cached in github registry when the image is built. Follow-up after apache#45266

We are using various caches in our build and so far - due to the way how "standard" caching works, PRs from forks could not effectively use the cache from main Airflow repository - because caches are not shared with other repositories - so the PRs builds could only use cache effectively when they were rebased and continued running from the same fork. This PR improves caching strategy using "stash" action from the ASF. Unlike `cache` - the action uses artifacts to store cache, and that makes it possible for the stash action to use such cache uploaded from `main` canary builds in PRs coming from the fork. As part of this change all the places where setup-python was used and breeze installed afterwards were reviewed and updated to use only breeze installation action (it already installs python) and this action has been improved to use UV caching effectively. Overall this PR should decrease setup overhead for many jobs across the CI workflow. Follow-up after apache#45266

Prod image build sometimes (quite often) is not needed. When removing `pull_request_target` in apache#45266 `wait-for=prod-images` had the condition that prevented it from running (and the `build-prod-images` step depended on it) - but this condition is gone now. Instead of preventing the whole composite workflow from running, we are adding it to "build-prod-packages" so that the whole workflow can complete as prerequisite to "finalize-tests" which should be executed regardless from `prod-image-build` being executed.

potiuk added full tests needed We need to run full set of tests for this PR to merge canary When set on PR running from apache repo - behave as canary run all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs labels Dec 29, 2024

boring-cyborg bot added area:dev-tools area:production-image Production image improvements and fixes labels Dec 29, 2024

potiuk mentioned this pull request Dec 29, 2024

Simplify caching mechanisms for CI and PROD images (regular PR) #45261

Closed

potiuk force-pushed the redesign-image-caching branch from 4f266c6 to 2682082 Compare December 29, 2024 11:59

potiuk changed the title ~~Simplify caching mechanisms for CI and PROD images~~ Simplify caching mechanisms for CI and PROD images (canary) Dec 29, 2024

potiuk force-pushed the redesign-image-caching branch 4 times, most recently from 1e4fe9b to 374cc5d Compare December 29, 2024 14:29