Skip to content

uv pip install for Benchmarks#17749

Merged
glenn-jocher merged 29 commits intomainfrom
uv-benchmarks
Dec 2, 2024
Merged

uv pip install for Benchmarks#17749
glenn-jocher merged 29 commits intomainfrom
uv-benchmarks

Conversation

@glenn-jocher
Copy link
Member

@glenn-jocher glenn-jocher commented Nov 24, 2024

🛠️ PR Summary

Made with ❤️ by Ultralytics Actions

🌟 Summary

Streamlined GitHub Actions by integrating astral-sh/setup-uv, enhancing consistency and efficiency in the CI pipeline. 🚀

📊 Key Changes

  • Added astral-sh/setup-uv@v3 for unified uv tool installation and management.
  • Modified dependency installation to use uv pip for improved package handling, replacing plain pip.
  • Updated commands to use uv for listing installed packages.
  • Removed manual caching for pip dependencies.

🎯 Purpose & Impact

  • Improved Automation: The uv tool simplifies dependency management and standardizes CI workflows. 🔧
  • Enhanced Reliability: Addresses potential issues like numpy errors by refining the installation process. ✅
  • Efficiency Boost: Streamlines setup, reducing manual steps and improving CI consistency for developers and contributors. 🏎️

@UltralyticsAssistant UltralyticsAssistant added dependencies Dependencies and packages devops GitHub Devops or MLops enhancement New feature or request labels Nov 24, 2024
@UltralyticsAssistant
Copy link
Member

👋 Hello @glenn-jocher, thank you for submitting an Ultralytics pull request! 🚀 To ensure a seamless integration of your changes into the ultralytics/ultralytics repository, please check the following:

  • Define a Purpose: Clearly explain the purpose of your changes in your PR description. You've done a great job highlighting the enhancements with the UV toolset! If this PR relates to any existing issues, please link them in your description.
  • Synchronize with Source: Make sure your PR is up-to-date with the main branch of the ultralytics/ultralytics repository. If necessary, update it by clicking the 'Update branch' button or by using git pull and git merge main locally.
  • Ensure CI Checks Pass: Check that all Ultralytics CI checks are successful. If any issues arise, addressing them promptly will help keep the merge process smooth.
  • Update Documentation: If your changes alter any functionalities, consider updating the documentation to reflect your improvements.
  • Add Tests: Where relevant, include new tests or update existing ones to verify your changes, and confirm all tests pass successfully.
  • Sign the CLA: If this is your first contribution to Ultralytics, please ensure you've signed the Contributor License Agreement by stating "I have read the CLA Document and I sign the CLA" in the comments.
  • Minimize Changes: As Bruce Lee wisely said, "It is not daily increase but daily decrease, hack away the unessential."

For additional guidance, you can refer to our Contributing Guide. Feel free to ask if you have any questions or need assistance. An Ultralytics engineer will review your PR soon. Thanks for your valuable contribution to Ultralytics! 🌟

@codecov
Copy link

codecov bot commented Nov 24, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.01%. Comparing base (69c12a4) to head (c8f77e6).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #17749      +/-   ##
==========================================
- Coverage   78.64%   74.01%   -4.64%     
==========================================
  Files         127      127              
  Lines       17099    17099              
==========================================
- Hits        13447    12655     -792     
- Misses       3652     4444     +792     
Flag Coverage Δ
Benchmarks 35.00% <ø> (-0.04%) ⬇️
GPU 39.13% <ø> (-2.64%) ⬇️
Tests 67.78% <ø> (-4.78%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@inisis
Copy link
Contributor

inisis commented Nov 25, 2024

Hi, @glenn-jocher what's the advantage of using uv?

@glenn-jocher
Copy link
Member Author

Faster, but it looks like it's causing problems here unfortunately.

@Laughing-q
Copy link
Member

@glenn-jocher After several tries I figured CI failing mostly because ofuv package has a different strategy(walking through all the dependencies and find proper versions for all the packages and install them) of handling dependencies. And it selected tensorflowjs==3.9.0 based on our dependency of tensorflowjs==3.9.0, however actually tensorflowjs==3.9.0 is not compatible with numpy>=1.20 hence the error. The original pip somehow would select tensorflowjs==4.15.0 which is a version compatible with numpy>=1.20 hence all CI tests passing.

I found the versions of tensorflowjs that compatible with numpy>=1.20 and tried pining tensorflowjs>=4.5.0 but then strange errors thorown from uv.

  × No solution found when resolving dependencies:
  ╰─▶ Because tensorflowjs>=4.5.0,<=4.10.0 depends on packaging>=20.9,<21.dev0
      and only the following versions of tensorflowjs are available:
          tensorflowjs<=4.11.0
          tensorflowjs==4.12.0
          tensorflowjs>=4.13.0
      we can conclude that all of:
          tensorflowjs>=4.5.0,<4.11.0
          tensorflowjs>4.11.0,<4.12.0
          tensorflowjs>4.12.0,<4.13.0
      depend on packaging>=20.9,<21.dev0.
      And because all of:
          tensorflowjs==4.11.0
          tensorflowjs==4.12.0
          tensorflowjs>=4.13.0
      depend on packaging>=23.1,<24.dev0 and only packaging>=22.0 is
      available, we can conclude that tensorflowjs>=4.5.0 depends on
      packaging>=23.1,<24.dev0.
      And because only the following versions of packaging are available:
          packaging<23.1
          packaging>24.dev0
      and ultralytics[export]==8.3.38 depends on tensorflowjs>=4.5.0, we can
      conclude that ultralytics[export]==8.3.38 cannot be used.
      And because only ultralytics[export]==8.3.38 is available and you
      require ultralytics[export], we can conclude that your requirements
      are unsatisfiable.

      hint: Pre-releases are available for tensorflowjs in the requested
      range (e.g., 4.13.0rc0), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

      hint: An index URL (https://download.pytorch.org/whl/cpu) could not
      be queried due to a lack of valid authentication credentials (403
      Forbidden).

      hint: packaging was requested with a pre-release marker (e.g.,
      packaging>=20.9,<21.dev0), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

      hint: `packaging` was found on https://download.pytorch.org/whl/cpu,
      but not at the requested version (packaging>=20.9,<21.dev0). A
      compatible version may be available on a subsequent index (e.g.,
      https://pypi.org/simple). By default, uv will only consider versions
      that are published on the first index that contains a given package, to
      avoid dependency confusion attacks. If all indexes are equally trusted,
      use `--index-strategy unsafe-best-match` to consider all versions from
      all indexes, regardless of the order in which they were defined.

@Laughing-q
Copy link
Member

@glenn-jocher ok reported a issue to the official uv repo and seek for help: astral-sh/uv#9462

@Y-T-G
Copy link
Collaborator

Y-T-G commented Nov 28, 2024

@glenn-jocher @Laughing-q GPU CI is also not working correctly right now. It passes, but it's running tests on the container installed Ultralytics package instead of the local/PR changes. Visible from the logs that shows the install type to be pip instead of git.
image

A fix was tested on this issue: #17781 (comment)

@Laughing-q
Copy link
Member

@Y-T-G @Skillnoob Hey I saw the error in that PR and thanks guys for the fix there! Probably it'd be better to open another PR to fix the issue there instead of directly modifying the user's PR since it's a common issue than a PR issue. Can you guys open another PR for it? THanks

@Skillnoob
Copy link
Collaborator

@Laughing-q Created #17883

@Laughing-q
Copy link
Member

@Skillnoob nice work! Thanks!

@Laughing-q
Copy link
Member

@glenn-jocher After several tries I figured CI failing mostly because ofuv package has a different strategy(walking through all the dependencies and find proper versions for all the packages and install them) of handling dependencies. And it selected tensorflowjs==3.9.0 based on our dependency of tensorflowjs==3.9.0, however actually tensorflowjs==3.9.0 is not compatible with numpy>=1.20 hence the error. The original pip somehow would select tensorflowjs==4.15.0 which is a version compatible with numpy>=1.20 hence all CI tests passing.

I found the versions of tensorflowjs that compatible with numpy>=1.20 and tried pining tensorflowjs>=4.5.0 but then strange errors thorown from uv.

  × No solution found when resolving dependencies:
  ╰─▶ Because tensorflowjs>=4.5.0,<=4.10.0 depends on packaging>=20.9,<21.dev0
      and only the following versions of tensorflowjs are available:
          tensorflowjs<=4.11.0
          tensorflowjs==4.12.0
          tensorflowjs>=4.13.0
      we can conclude that all of:
          tensorflowjs>=4.5.0,<4.11.0
          tensorflowjs>4.11.0,<4.12.0
          tensorflowjs>4.12.0,<4.13.0
      depend on packaging>=20.9,<21.dev0.
      And because all of:
          tensorflowjs==4.11.0
          tensorflowjs==4.12.0
          tensorflowjs>=4.13.0
      depend on packaging>=23.1,<24.dev0 and only packaging>=22.0 is
      available, we can conclude that tensorflowjs>=4.5.0 depends on
      packaging>=23.1,<24.dev0.
      And because only the following versions of packaging are available:
          packaging<23.1
          packaging>24.dev0
      and ultralytics[export]==8.3.38 depends on tensorflowjs>=4.5.0, we can
      conclude that ultralytics[export]==8.3.38 cannot be used.
      And because only ultralytics[export]==8.3.38 is available and you
      require ultralytics[export], we can conclude that your requirements
      are unsatisfiable.

      hint: Pre-releases are available for tensorflowjs in the requested
      range (e.g., 4.13.0rc0), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

      hint: An index URL (https://download.pytorch.org/whl/cpu) could not
      be queried due to a lack of valid authentication credentials (403
      Forbidden).

      hint: packaging was requested with a pre-release marker (e.g.,
      packaging>=20.9,<21.dev0), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

      hint: `packaging` was found on https://download.pytorch.org/whl/cpu,
      but not at the requested version (packaging>=20.9,<21.dev0). A
      compatible version may be available on a subsequent index (e.g.,
      https://pypi.org/simple). By default, uv will only consider versions
      that are published on the first index that contains a given package, to
      avoid dependency confusion attacks. If all indexes are equally trusted,
      use `--index-strategy unsafe-best-match` to consider all versions from
      all indexes, regardless of the order in which they were defined.

@glenn-jocher uv currently is working properly with our benchmark now. I solved the issue by adding --index-strategy unsafe-first-match, based on the suggestion from the uv repo maintainer. It's because uv has different strategy when it works with --extra-index-url, see https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes

@glenn-jocher
Copy link
Member Author

@Skillnoob @Y-T-G #17883 merged, updating this one now, nice work!

@glenn-jocher glenn-jocher changed the title Try uv pip install for Benchmarks uv pip install for Benchmarks Dec 2, 2024
@glenn-jocher glenn-jocher merged commit 5015d1e into main Dec 2, 2024
@glenn-jocher glenn-jocher deleted the uv-benchmarks branch December 2, 2024 14:48
@glenn-jocher
Copy link
Member Author

@Laughing-q PR merged, great work!

dariussingh pushed a commit to dariussingh/ultralytics that referenced this pull request Dec 11, 2024
Co-authored-by: Ultralytics Assistant <[email protected]>
Co-authored-by: Laughing <[email protected]>
Co-authored-by: Laughing-q <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Dependencies and packages devops GitHub Devops or MLops enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants