Skip to content

Add CI cache pruning#17664

Merged
glenn-jocher merged 17 commits intomainfrom
uv-cache-prune
Dec 8, 2024
Merged

Add CI cache pruning#17664
glenn-jocher merged 17 commits intomainfrom
uv-cache-prune

Conversation

@Burhan-Q
Copy link
Contributor

@Burhan-Q Burhan-Q commented Nov 20, 2024

As recommended in uv documentation

Results

  • GPU CI: Removed 785 files (6.6MiB)
  • Tests (ubuntu-latest, 3.11, latest): Removed 48101 files (3.0GiB)
  • Tests (macos-14, 3.11, latest): Removed 41819 files (2.0GiB)
  • Tests (windows-latest, 3.11, latest): Removed 40539 files (3.1GiB)
  • Tests (ubuntu-latest, 3.8, latest): Removed 34847 files (2.6GiB)

🛠️ PR Summary

Made with ❤️ by Ultralytics Actions

🌟 Summary

Improved CI efficiency by adding automated cache management and enhancing token handling for workflows. 🚀

📊 Key Changes

  • 🧹 Added uv cache prune --ci to multiple workflow jobs (CI benchmarks, tests, and publishing) for automated cache cleanup.
  • 🔧 Improved token handling in formatting workflow, enabling fallback to a secondary token if the primary isn't available.

🎯 Purpose & Impact

  • Efficiency: Automated cache pruning reduces unnecessary storage usage and potential cache-related issues, streamlining CI processes.
  • 💡 Resilience: Improved token fallback logic ensures uninterrupted workflow execution even if the primary secret is unavailable.
  • 🌱 Scalability: Enhances maintainability and reliability of workflows for long-term project benefits.

@Burhan-Q Burhan-Q added the devops GitHub Devops or MLops label Nov 20, 2024
@UltralyticsAssistant UltralyticsAssistant added the enhancement New feature or request label Nov 20, 2024
@UltralyticsAssistant
Copy link
Member

👋 Hello @Burhan-Q, thank you for submitting your PR to the ultralytics/ultralytics 🚀 repository! This is an automated response, but rest assured, an Ultralytics engineer will assist you soon. In the meantime, please review the following checklist to ensure everything is in order for your contribution:

  • Clarify Purpose: Make sure to clearly define the purpose and impact of your changes in the PR description. If applicable, link to any relevant issues to provide context.
  • Stay Updated: Confirm that your branch is in sync with the ultralytics/ultralytics main branch. If there are any conflicts, resolve them by updating your branch with the latest changes.
  • Pass CI Checks: Ensure all Continuous Integration (CI) checks pass successfully. If there are any issues, please address them promptly.
  • Documentation: Update the documentation as needed to reflect your changes, especially if there are new or modified features.
  • Testing: If relevant, add or update tests for your modifications and verify that all tests are passing.
  • Contributor License Agreement: Please sign the Contributor License Agreement (CLA) if this is your first contribution to Ultralytics by stating "I have read the CLA Document and I sign the CLA" in a new comment on this PR.
  • Keep It Concise: Limit the changes in your PR to the minimum necessary to accomplish your objective. "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." — Antoine de Saint-Exupéry

For additional guidance, refer to our Contributing Guide. Feel free to ask questions or seek clarification by leaving a comment. We appreciate your effort to enhance the CI workflow, making it more efficient with cache pruning! 🚀💾

@Burhan-Q Burhan-Q added the TODO High priority items label Nov 20, 2024
@codecov
Copy link

codecov bot commented Nov 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.01%. Comparing base (bf63170) to head (f77547a).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #17664      +/-   ##
==========================================
+ Coverage   73.97%   74.01%   +0.03%     
==========================================
  Files         128      128              
  Lines       17138    17138              
==========================================
+ Hits        12678    12684       +6     
+ Misses       4460     4454       -6     
Flag Coverage Δ
Benchmarks 35.02% <ø> (+0.03%) ⬆️
GPU 38.47% <ø> (ø)
Tests 67.75% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@glenn-jocher
Copy link
Member

@Burhan-Q can we combine these into the uv pip install step?

@Burhan-Q
Copy link
Contributor Author

Burhan-Q commented Dec 7, 2024

@glenn-jocher I can try moving this but was following the guidance of the uv docs to place after all tests have run. I'm guessing that since it removes all pre built packages, moving it forward might cause problems, but it's worth testing out.

@glenn-jocher
Copy link
Member

glenn-jocher commented Dec 7, 2024

@Burhan-Q yes I just read them and they recommend this as a final step.

I read some more on their README though and it seems like there is an argument there for cache pruning that is already set to default true, so I'm a little confused if since we are using the action and not just the CLI commands if this already covers us?

https://github.com/astral-sh/setup-uv/blob/1e4d4ea9ff6b65a6b126d250fc05cf16a2adf619/action.yml#L35-L37

@Burhan-Q
Copy link
Contributor Author

Burhan-Q commented Dec 7, 2024

Oh I see. That's curious bc adding the cache prune --ci step does remove items, but maybe that was in the "clean up" step by default and I just moved this forward... You're right the GHA has it enabled by default

@Burhan-Q
Copy link
Contributor Author

Burhan-Q commented Dec 7, 2024

I wonder if the cache prune command only runs by default if the cache is enabled for the action? I poked around the issues on the setup-uv repo and saw someone posted their logs, which show "Pruning cache" in the post setup-uv job clean up step, which isn't shown for the Ultralytics GHA runs. This user's workflow file does enable caching, so I'm wondering if that's why it doesn't run for ours and enabling the CLI command, that this PR introduces, ensures this runs.

Signed-off-by: Glenn Jocher <[email protected]>
Signed-off-by: Glenn Jocher <[email protected]>
Signed-off-by: Glenn Jocher <[email protected]>
@glenn-jocher glenn-jocher merged commit 118edde into main Dec 8, 2024
@glenn-jocher glenn-jocher deleted the uv-cache-prune branch December 8, 2024 00:20
@glenn-jocher glenn-jocher removed the TODO High priority items label Dec 8, 2024
@glenn-jocher
Copy link
Member

@Burhan-Q merged!

dariussingh pushed a commit to dariussingh/ultralytics that referenced this pull request Dec 11, 2024
Signed-off-by: Glenn Jocher <[email protected]>
Co-authored-by: Ultralytics Assistant <[email protected]>
Co-authored-by: Glenn Jocher <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devops GitHub Devops or MLops enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants