Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent greenlet from building on free-threaded Python #423

Open
jamadden opened this issue Sep 10, 2024 · 5 comments
Open

Prevent greenlet from building on free-threaded Python #423

jamadden opened this issue Sep 10, 2024 · 5 comments
Labels
Feature Request Feature request Help Wanted Internal Discussions on processes, etc, that don't directly face end users

Comments

@jamadden
Copy link
Contributor

We don't set the flag that indicates we're compatible with free-threading, so loading us would automatically enable the GIL according to the Python documentation. It is thus misleading that we can even build on a free-threaded Python and get the "t" suffix, e.g., "313t".

The make-manylinux script has some support for skipping free-threaded Python's, but it's neither complete nor sufficient.

The trouble is that CPython doesn't document a way to detect at compile time that it is built on free-threading. At least, not that I could see in the "Porting to 3.13" and "C API Changes" sections of the release notes.

@jamadden jamadden added Help Wanted Internal Discussions on processes, etc, that don't directly face end users Feature Request Feature request labels Sep 10, 2024
@LeroyK111
Copy link

0.06s - pydevd: GEVENT_SUPPORT is set but gevent is not available in the environment. Please unset GEVENT_SUPPORT from the environment variables or install gevent. Traceback (most recent call last): File "c:\Users\Administrator\.vscode\extensions\ms-python.debugpy-2024.13.2024101501-win32-x64\bundled\libs\debugpy\_vendored\pydevd\pydevd.py", line 166, in <module> from _pydevd_bundle import pydevd_gevent_integration File "c:\Users\Administrator\.vscode\extensions\ms-python.debugpy-2024.13.2024101501-win32-x64\bundled\libs\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_gevent_integration.py", line 2, in <module> import greenlet File "d:\python\Lib\site-packages\greenlet\__init__.py", line 29, in <module> from ._greenlet import _C_API # pylint:disable=no-name-in-module ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'greenlet._greenlet'

So, I configured the launch.json file in VSCode and set the "gevent" field to false. After that, I was able to use Python's free-threaded version, which disables the GIL (Global Interpreter Lock) without any issues.

@jamadden
Copy link
Contributor Author

If you load the extension, it's going to initialize the GIL. If you force Python to not initialize the GIL, the extension will be broken. Do not do this.

@ngoldbaum
Copy link

The trouble is that CPython doesn't document a way to detect at compile time that it is built on free-threading. At least, not that I could see in the "Porting to 3.13" and "C API Changes" sections of the release notes.

You can check if Py_GIL_DISABLED is set.

Thinking further afield - is there an issue for supporting the free-threaded build? I'm actively working on adding support for free-threaded Python in community packages and can offer help and advice. There's also https://py-free-threading.github.io which has some advice for porting extensions.

@jamadden
Copy link
Contributor Author

You can check if Py_GIL_DISABLED is set.

So I just need:

// greenlet.h 
#ifdef Py_GIL_DISABLED
#error Nope right out of that.
#endif       

is there an issue for supporting the free-threaded build?

Unfortunately yes. greenlet itself implicitly depends on the GIL to keep its internal structures consistent. We could add a bunch of locks internally to take care of that, but that leaves bigger issues. I'm not saying they can't be dealt with, but they are certainly factors that make trying to support GILless CPython risky.

Bigger issue #1 is that greenlet copies and mutates internal CPython state and then restores it, and that data must be completely consistent. That state should all belong to one thread, and shouldn't be manipulated by other threads, but that's a lot of wishful thinking that I don't know is actually true, especially when it comes to internal CPython data structures that can be manipulated as side-effects (e.g., there has to be a list of threads and thread creating and destruction has to manipulate that list; if something we copy/restore during greenlet switching refers to that list, concurrent manipulation of it could be deadly). Extensive testing would be required (and its hard to test for race conditions.)

Bigger issue #2 is effectively the same issue that the Python community as a whole has to deal with --- race conditions that have been hidden until now --- only writ large. Because greenlets are fully cooperative, users are used to ignoring race conditions altogether. Already when you try to mix threads with greenlets things get dicy --- gevent has a lot of code to try to make this safe and workable, but it's not perfect --- and actually adding true concurrency on top of that is only going to magnify the problem. (Oh, and also, gevent depends on the GIL for some of its invariants as well, so gevent would have to be reworked too.) Again, not an insurmountable problem (and technically not one that greenlet can really do anything about), but a factor to consider before deciding if it's worth spending the effort.

@ngoldbaum
Copy link

ngoldbaum commented Nov 27, 2024

So I just need:

You need to #include "python.h" so if greenlet.h does that then yes.

Thanks for the context!

I'm not sure whether pip will continue to install the cp313t wheels on pypi if there is a newer version without cp313t wheels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature Request Feature request Help Wanted Internal Discussions on processes, etc, that don't directly face end users
Projects
None yet
Development

No branches or pull requests

3 participants