Yes, I like asdf and I mention it in an article Mac Python where I suggest using it to handle multiple languages. I think I might recommend mise rather than asdf but I haven’t yet tried mise with Python.
I used mise when it was rtx, and it worked great!
I have since stopped using it, through no fault of its own – I just haven’t been writing much python lately.
I had problems on my mac with using homebrew-installed python, such as scripts trying to use wrong python version or not finding the python by full path at all. Instead of proper investigation, I did removed the homebrew python(s), installed asdf and used it to install and resolve per-project any problems with python versions. That is not python-way as the one described in the article, but more general, high level way of managing versions of tooling on your machine, and I highly recommend asdf both for production and development workflows.
Yup, it’s great. The only annoying thing is that now it requires a separate tool for the command line completion (which can be installed using mise itself).
I’ve used (and recommend) asdf for projects that combine Ruby and JavaScript. When you use asdf with Python do you use pip to install packages? That works out okay, so it’s project based? I had the same problems with the Homebrew brew install python. It worked okay to install pyenv for a choice of Python versions but I still needed to use Venv. In the end Rye seems simpler because it’s project-based and just one tool.
Yes, I guess some python-specific tooling like pyencv or venv will be needed for the python development, but for just a mere managing of python projects (e.g. installing, running scripts etc) asdf seems to be enough — and more general, as it covers lots of other tools. I don’t have enought expertise to say if asdf+internal tooling can cover all the possible problems (e.g. what would be happening with caches of pip used in different contexts?), but it was and is enough for me. Thanks for pointing to that seam between “general” and “internal” tooling version control!
Yes, I think experienced Python developers overlook the difference between installing Python tools/utilities/applications like Youtube-dl or Ruff, where one can install Pipx and be done, and Python programming where one should use a version manager (Pyenv), package manager (Pip), and environment manager (Venv) – or use Rye. Very different use cases and confusing when READMEs just give instructions to pip install and beginners end up with Command not found: pip.
As a Rubyist who began using Python recently, I was frustrated by Python’s tooling until I found Rye, which is an all-in-one tool that replaces Pyenv, Pip, and Venv, for a more project-centered approached to development (like Ruby or JavaScript or Rust). As someone who habitually writes tutorials for beginners, I took it upon myself to write a freeCodeCamp tutorial advocating setting up Python projects using Rye.
Posting this here for feedback from those more experienced than I. Anyone else who shares the opinion that Python standard tooling is a hindrance? Or am I leading beginners down the wrong path?
You are on the right path (or maybe one of current right paths?) Armin built Rye on his own and handed it off to Astral. I that Armin knows what needed to be done, but the use of Rust and hand-off to Astral doesn’t feel right with some people in the Python community. There has also been a number of other individual-led projects which have come to prominence with people feeling they were going to break through and solve problems only for expectations to be broken.
It is mostly grumbling with comments like “I’m going to wait to see what happens because I can get by with what I have/know.” or “I don’t like what’s been happening with open source companies recently cf. Hashicorp and this is more fundamental to how I work.” This diagram may not show all of the tools people are using which feels like a lot.
I didn’t know and it’s helpful to hear this. I wonder if I’m overly discounting the community discomfort. Still, from a devex perspective, the current state leaves a lot to be desired, so I think efforts like Rye are a good idea.
It’s a bit weird to infer if Xcode tools are installed from Python being available at a given path. A more explicit option is: xcode-select -p which either succeeds with the path or fails when not installed.
My assumption is that a beginner installing Python uses the default Mac zsh. But you have a point: Someone installing Python for the first time might be an experienced developer who has set a different shell. Thanks for opening my eyes to this.
Not to be that guy, but if you install Nix running any version of Python is as simple as i.e. nix-shell -p python311. Being a Nix user really made my life way simpler.
I give the average nix-darwin installer about a week before they have the thought “hmmmmmmm, if I’m using Nix to grab Python, I might as well use it to get my dependencies too, right?”, start learning to make Python shell.nix/devShells for their projects, and then they’re just goners.
For folks already into pkgsrc (please note hat and sprinkle grains of salt accordingly), it’s pretty easy to bootstrap multiple prefixes and give each its own Python. For a solo project I might just do that. For team projects, I’d probably want us to converge on asdf or mise wherever possible (and I maintain pkgsrc’s packages of both).
$ ./bootstrap --unprivileged ~/pkg/ancient-project
$ oldpath="$PATH"
$ PATH="~/pkg/ancient-project/bin:~/pkg/ancient-project/sbin:$PATH"
$ cd ../lang/python27 && bmake install
$ # also install anything else needed for ancient-project
$ PATH="$oldpath"
$ ./bootstrap --unprivileged ~/pkg/recent-project
$ oldpath="$PATH"
$ PATH="~/pkg/recent-project/bin:~/pkg/recent-project/sbin:$PATH"
$ cd ../lang/python312 && bmake install
$ # also install anything else needed for recent-project
$ PATH="$oldpath"
Then when you’re working on one of these projects, set PATH accordingly, maybe with direnv. (Assuming a non-project-specific general-purpose pkgsrc installation already in $PATH, install direnv in it.)
I looked at that approach and it’s solid and well-supported. Just a choice or multiple tools versus Rye as a single tool. Though you can say multiple tools are more Unix-like.
If folks are choosing a way, and rye is their chosen way, these arguments apply to Linux and Windows also, right? I’d say that you should test these on those two other systems and then leave off the “on a Mac” part.
As long as there’s been an xkcd about the hell that is Python version management (https://xkcd.com/1987/), there isn’t a reason not to suggest a way that could work for folks. I go with os Python for everything that is packaged, homebrew Python for things I pip install over just to make something work this week and then blow it away in a few months when it’s been destroyed by bad package hygiene, and anything that I’m producing is built inside a container that then allows it to be reproduced by others. This feels more robust than rye with venv because development is identical to where it will run, and CI can make sure that dependencies are updated properly with dependabot so that you don’t end up with a “this worked two years ago, but all the specified versions have been withdrawn for security reasons or dependencies’ security reasons”.
One thing I would add is a template or how to at the bottom for building containers out of your project. It’s so important to have a low impedance when transitioning to a container, so I don’t feel like tutorials that ends with ‘it now works on your machine’ is sufficient. It’s interesting that rye suggests not to use itself once it comes to shipping your code: https://rye-up.com/guide/docker/
I see the humor :-). Seriously, I think it’s because Python has been around a long time (since 1991) and has a healthy grassroots community (multiple choices and difficult to achieve consensus). Maybe the challenges of the shift from Python 2 to 3 held back devex improvements? Of course I like Ruby (1995). Some will remember Ruby’s early growing pains around devex but rvm and Bundler made early inroads and gained acceptance. For sure, I’d rather use Python than JavaScript (1995). Finally, the Rust devex is way better but it’s a much newer language (2010).
You would definitely get a lot of comments under an article about how to use CMake or whatever, so the age of the ecosystem is probably one factor, but I think the bigger issue is that there’s no leadership around dependencies in Python. JavaScript is technically run by the ECMA committee, but in practice, NPM made up a solution for dependencies, then Yarn and PNPM copied and extended it, and now Deno and Bun have to shoehorn in compatibility with it. With Python, there are some standards around setup.py and pyproject.toml, but they have been weirdly frozen for the last decade plus.
On macOS? Prior to macOS 12.3, Macs came with Python 2.7 pre-installed. Macs don’t come with Python now. Until you install Xcode Command Line Tools which includes Python 3.9.6. That’s an older version, too. And using pip install for any package with the system Python will install dependencies in a user folder that is prone to dependency conflicts because there is no isolation of projects. Unless you use Venv to create virtual environments. Which is why I think it’s simpler to use Rye and manage Python projects within a project folder with an all-in-one tool. But I’m interested in hearing if it’s just me that finds standard Python tooling confusing.
It’s not straightforward, but… “pip installs packages relative to the used python file, or for the user if you use –user; venv creates a local python file link; multiple versions can’t exist in the same environment” explains 99% of use cases people will run into. Almost every “package installed in the wrong place”, “I don’t have permissions”, “versions are conflicting”, etc. issue can be understood through those 3 ideas.
But yeah, don’t use the system Python. It can change during the system upgrade and your code may not be ready for that.
There was also an article here a few months back telling you not to use the one from Homebrew because it’s intended as a dependency for Python things in Homebrew and so may get updated at random times. Using pipenv seemed to work for me for Python things on macOS, but I do very few Python things.
I run a team that does the devex for a larger group of Python developers. I had a tab open with this article because I had to tell yet another dev to not use the brew python.
The vanilla python.org pythons, after much pain with other ideas, is what we wound up settling on.
I saw that article and it was informative. I’m curious why (or if?) you recommend the python.org installer. I’m not sure I like to see Python versions installed to /Library/Frameworks/Python.framework/ with symlinks. I like that Rye installs everything to a ~/.rye/ folder with one change needed to the PATH. In any case, I think using the official python.org installer just creates another global Python with a continued need for either Pipx (for standalone tools and scripts) or Venv/Pip for programming in virtual environments. Rye gives me a single tool, instead. What’s the downside? You’ve done more work with this than I have.
In theory, I like it. But Rye isn’t Lindy. In fact, Rye just became part of Astral, with the idea that uv will take over the Rye functionality. Maybe uv will catch fire the way Ruff has, and then I will be very tempted to use it.
I agree all the global pythons aren’t ideal. At least, it isn’t elegant. But at least this way, I know that if I do deactivate && python3.n, I always know what python I am using. After a lot of lecturing, an init script that refuses to proceed if it detects homebrew python being used, more lecturing, I finally have most of the dev org using vanilla python.org pythons. It’s gotten rid of a bunch of problems.
I face a similar issue with Node. I’ve settled on using n, which is the most like installing vanilla python.org pythons. But it has an option to install under ~, and I expect to migrate to that if/when multiple globally installed nodes becomes a problem. So far, they haven’t been.
I just get grumpy about how difficult it is to manage python installs. Honestly I just use ASDF. But containers work too. I do use python when I have absolutely no other choice.
Step 0: install nix and start cursing everything - at least your python might then be the correct version
s/Python problems/Nix problems/Please also check
asdf, the multiple runtime version manager.It is easier to maintain a lot of installations without cluttering a whole bunch of shell rc scripts.
Yes, I like
asdfand I mention it in an article Mac Python where I suggest using it to handle multiple languages. I think I might recommend mise rather thanasdfbut I haven’t yet tried mise with Python.I used mise when it was rtx, and it worked great!
I have since stopped using it, through no fault of its own – I just haven’t been writing much python lately.
Especially if you’re using more than one language. asdf solves a lot of issues in that case.
Also, ASDF supports UV as a plugin in itself https://github.com/asdf-community/asdf-uv
I had problems on my mac with using homebrew-installed python, such as scripts trying to use wrong python version or not finding the python by full path at all. Instead of proper investigation, I did removed the homebrew python(s), installed
asdfand used it to install and resolve per-project any problems with python versions. That is not python-way as the one described in the article, but more general, high level way of managing versions of tooling on your machine, and I highly recommendasdfboth for production and development workflows.This is an article that I would read if I was python-centric: https://justinmayer.com/posts/homebrew-python-is-not-for-you/
Mise, formerly rtx, is a great asdf alternative.
https://mise.jdx.dev/
Have you used mise with Python? I haven’t tried it yet. Any hiccups?
Yup, it’s great. The only annoying thing is that now it requires a separate tool for the command line completion (which can be installed using mise itself).
Good article.
I’ve used (and recommend)
asdffor projects that combine Ruby and JavaScript. When you useasdfwith Python do you usepipto install packages? That works out okay, so it’s project based? I had the same problems with the Homebrew brew install python. It worked okay to install pyenv for a choice of Python versions but I still needed to use Venv. In the end Rye seems simpler because it’s project-based and just one tool.Yes, I guess some python-specific tooling like pyencv or venv will be needed for the python development, but for just a mere managing of python projects (e.g. installing, running scripts etc) asdf seems to be enough — and more general, as it covers lots of other tools. I don’t have enought expertise to say if asdf+internal tooling can cover all the possible problems (e.g. what would be happening with caches of pip used in different contexts?), but it was and is enough for me. Thanks for pointing to that seam between “general” and “internal” tooling version control!
Yes, I think experienced Python developers overlook the difference between installing Python tools/utilities/applications like Youtube-dl or Ruff, where one can install Pipx and be done, and Python programming where one should use a version manager (Pyenv), package manager (Pip), and environment manager (Venv) – or use Rye. Very different use cases and confusing when READMEs just give instructions to
pip installand beginners end up with Command not found: pip.As a Rubyist who began using Python recently, I was frustrated by Python’s tooling until I found Rye, which is an all-in-one tool that replaces Pyenv, Pip, and Venv, for a more project-centered approached to development (like Ruby or JavaScript or Rust). As someone who habitually writes tutorials for beginners, I took it upon myself to write a freeCodeCamp tutorial advocating setting up Python projects using Rye.
Posting this here for feedback from those more experienced than I. Anyone else who shares the opinion that Python standard tooling is a hindrance? Or am I leading beginners down the wrong path?
You are on the right path (or maybe one of current right paths?) Armin built Rye on his own and handed it off to Astral. I that Armin knows what needed to be done, but the use of Rust and hand-off to Astral doesn’t feel right with some people in the Python community. There has also been a number of other individual-led projects which have come to prominence with people feeling they were going to break through and solve problems only for expectations to be broken.
I hadn’t seen the pushback against the hand-off to Astral, what are the concerns there?
It is mostly grumbling with comments like “I’m going to wait to see what happens because I can get by with what I have/know.” or “I don’t like what’s been happening with open source companies recently cf. Hashicorp and this is more fundamental to how I work.” This diagram may not show all of the tools people are using which feels like a lot.
Great diagram! And disturbing :-0 lol. It looks like PYFlow is in the same sweet spot as Rye, any opinions? I didn’t know about it.
I didn’t know and it’s helpful to hear this. I wonder if I’m overly discounting the community discomfort. Still, from a devex perspective, the current state leaves a lot to be desired, so I think efforts like Rye are a good idea.
It’s a bit weird to infer if Xcode tools are installed from Python being available at a given path. A more explicit option is:
xcode-select -pwhich either succeeds with the path or fails when not installed.Good point. I’ve mentioned this in an article Update Python based on the freeCodeCamp article but I’ll go back and revise the original as you suggest.
Does not really acknowledge that shells other than
zshexist, which might make it confusing for someone who’s using fish, for instance.My assumption is that a beginner installing Python uses the default Mac zsh. But you have a point: Someone installing Python for the first time might be an experienced developer who has set a different shell. Thanks for opening my eyes to this.
Not to be that guy, but if you install Nix running any version of Python is as simple as i.e.
nix-shell -p python311. Being a Nix user really made my life way simpler.I give the average nix-darwin installer about a week before they have the thought “hmmmmmmm, if I’m using Nix to grab Python, I might as well use it to get my dependencies too, right?”, start learning to make Python shell.nix/devShells for their projects, and then they’re just goners.
For folks already into pkgsrc (please note hat and sprinkle grains of salt accordingly), it’s pretty easy to bootstrap multiple prefixes and give each its own Python. For a solo project I might just do that. For team projects, I’d probably want us to converge on
asdformisewherever possible (and I maintain pkgsrc’s packages of both).Out of curiosity, what does that look like?
Something like so:
Then when you’re working on one of these projects, set
PATHaccordingly, maybe withdirenv. (Assuming a non-project-specific general-purpose pkgsrc installation already in$PATH, installdirenvin it.)I use pyenv via homebrew, and poetry to manage dependency management. Works out fine, for me at least.
I looked at that approach and it’s solid and well-supported. Just a choice or multiple tools versus Rye as a single tool. Though you can say multiple tools are more Unix-like.
I’d suggest following the method here: https://blog.glyph.im/2023/08/get-your-mac-python-from-python-dot-org.html
If folks are choosing a way, and rye is their chosen way, these arguments apply to Linux and Windows also, right? I’d say that you should test these on those two other systems and then leave off the “on a Mac” part.
As long as there’s been an xkcd about the hell that is Python version management (https://xkcd.com/1987/), there isn’t a reason not to suggest a way that could work for folks. I go with os Python for everything that is packaged, homebrew Python for things I pip install over just to make something work this week and then blow it away in a few months when it’s been destroyed by bad package hygiene, and anything that I’m producing is built inside a container that then allows it to be reproduced by others. This feels more robust than rye with venv because development is identical to where it will run, and CI can make sure that dependencies are updated properly with dependabot so that you don’t end up with a “this worked two years ago, but all the specified versions have been withdrawn for security reasons or dependencies’ security reasons”.
One thing I would add is a template or how to at the bottom for building containers out of your project. It’s so important to have a low impedance when transitioning to a container, so I don’t feel like tutorials that ends with ‘it now works on your machine’ is sufficient. It’s interesting that rye suggests not to use itself once it comes to shipping your code: https://rye-up.com/guide/docker/
I wonder why other languages don’t have this kind of article with 36 comments under it.
I see the humor :-). Seriously, I think it’s because Python has been around a long time (since 1991) and has a healthy grassroots community (multiple choices and difficult to achieve consensus). Maybe the challenges of the shift from Python 2 to 3 held back devex improvements? Of course I like Ruby (1995). Some will remember Ruby’s early growing pains around devex but rvm and Bundler made early inroads and gained acceptance. For sure, I’d rather use Python than JavaScript (1995). Finally, the Rust devex is way better but it’s a much newer language (2010).
You would definitely get a lot of comments under an article about how to use CMake or whatever, so the age of the ecosystem is probably one factor, but I think the bigger issue is that there’s no leadership around dependencies in Python. JavaScript is technically run by the ECMA committee, but in practice, NPM made up a solution for dependencies, then Yarn and PNPM copied and extended it, and now Deno and Bun have to shoehorn in compatibility with it. With Python, there are some standards around setup.py and pyproject.toml, but they have been weirdly frozen for the last decade plus.
pyproject.toml has a spec that was last updated in 2020, the spec is not weirdly frozen. There is no spec for lock files yet, which is a shame.
It pains me to say this, but 2020 was four years ago.
Go was considered fairly late to introduce a lock file system. It was added back in 2018.
Everytime I use python, it’s a pain.
How about – don’t? Use the built-in one and KISS.
On macOS? Prior to macOS 12.3, Macs came with Python 2.7 pre-installed. Macs don’t come with Python now. Until you install Xcode Command Line Tools which includes Python 3.9.6. That’s an older version, too. And using pip install for any package with the system Python will install dependencies in a user folder that is prone to dependency conflicts because there is no isolation of projects. Unless you use Venv to create virtual environments. Which is why I think it’s simpler to use Rye and manage Python projects within a project folder with an all-in-one tool. But I’m interested in hearing if it’s just me that finds standard Python tooling confusing.
I think the confusion is a meme now. https://blog.viraptor.info/post/python-dependency-management-difficulty-is-an-unhelpful-meme
It’s not straightforward, but… “pip installs packages relative to the used python file, or for the user if you use –user; venv creates a local python file link; multiple versions can’t exist in the same environment” explains 99% of use cases people will run into. Almost every “package installed in the wrong place”, “I don’t have permissions”, “versions are conflicting”, etc. issue can be understood through those 3 ideas.
But yeah, don’t use the system Python. It can change during the system upgrade and your code may not be ready for that.
I can’t find the source anymore, but it is not recommended to and Apple plans to not make it available to the end user anymore in the future.
There was also an article here a few months back telling you not to use the one from Homebrew because it’s intended as a dependency for Python things in Homebrew and so may get updated at random times. Using
pipenvseemed to work for me for Python things on macOS, but I do very few Python things.https://justinmayer.com/posts/homebrew-python-is-not-for-you/
That article winds up recommending asdf.
I run a team that does the devex for a larger group of Python developers. I had a tab open with this article because I had to tell yet another dev to not use the brew python.
The vanilla python.org pythons, after much pain with other ideas, is what we wound up settling on.
I saw that article and it was informative. I’m curious why (or if?) you recommend the python.org installer. I’m not sure I like to see Python versions installed to
/Library/Frameworks/Python.framework/with symlinks. I like that Rye installs everything to a~/.rye/folder with one change needed to the PATH. In any case, I think using the official python.org installer just creates another global Python with a continued need for either Pipx (for standalone tools and scripts) or Venv/Pip for programming in virtual environments. Rye gives me a single tool, instead. What’s the downside? You’ve done more work with this than I have.In theory, I like it. But Rye isn’t Lindy. In fact, Rye just became part of Astral, with the idea that
uvwill take over the Rye functionality. Maybeuvwill catch fire the way Ruff has, and then I will be very tempted to use it.I agree all the global pythons aren’t ideal. At least, it isn’t elegant. But at least this way, I know that if I do
deactivate && python3.n, I always know what python I am using. After a lot of lecturing, an init script that refuses to proceed if it detects homebrew python being used, more lecturing, I finally have most of the dev org using vanilla python.org pythons. It’s gotten rid of a bunch of problems.I face a similar issue with Node. I’ve settled on using
n, which is the most like installing vanilla python.org pythons. But it has an option to install under~, and I expect to migrate to that if/when multiple globally installed nodes becomes a problem. So far, they haven’t been.Thanks!
Maybe the best way to install python on a Mac is not to install python on a Mac!
I just get grumpy about how difficult it is to manage python installs. Honestly I just use ASDF. But containers work too. I do use python when I have absolutely no other choice.
(chuckles) But seriously, are you saying it’s easier to use Python on Linux or Windows? Or are you saying don’t use Python?
[Comment removed by author]