Skip to content

Comments

gh-123471: Make concurrent iteration over itertools.repeat safe under free-threading#131247

Merged
kumaraditya303 merged 3 commits intopython:mainfrom
eendebakpt:ft_itertools
Apr 13, 2025
Merged

gh-123471: Make concurrent iteration over itertools.repeat safe under free-threading#131247
kumaraditya303 merged 3 commits intopython:mainfrom
eendebakpt:ft_itertools

Conversation

@eendebakpt
Copy link
Contributor

@eendebakpt eendebakpt commented Mar 14, 2025 •
edited
Loading

@eendebakpt eendebakpt requested a review from rhettinger as a code owner March 14, 2025 14:17
@bedevere-app bedevere-app bot mentioned this pull request Mar 14, 2025
Open
@eendebakpt eendebakpt marked this pull request as draft March 14, 2025 14:22
@eendebakpt eendebakpt marked this pull request as ready for review March 14, 2025 14:41
@rhettinger
Copy link
Contributor

rhettinger commented Mar 14, 2025

Is it possible to create tests demonstrating code the previously failed that can now be counted on to succeed?

@eendebakpt eendebakpt closed this Mar 14, 2025
@eendebakpt eendebakpt reopened this Mar 14, 2025
@eendebakpt
Copy link
Contributor Author

eendebakpt commented Mar 14, 2025

@rhettinger The issue that this PR solves is the following: when defining the iterator as repeat(data, 245) and concurrent iteration with multiple threads in this part of the code:

    if (ro->cnt > 0)
        ro->cnt--;

we can have multiple threads passing the condition ro->cnt > 0 (if ro->cnt is 1), and then decrementing ro->cnt to a negative value. The iterator will then nevert be exhausted (even though this is not the intend of the user).

A test for this is:

import unittest
from threading import Thread, Barrier
from itertools import repeat
from test.support import threading_helper


threading_helper.requires_working_threading(module=True)

class ItertoolsThreading(unittest.TestCase):

    @threading_helper.reap_threads
    def test_repeat(self):
        number_of_threads = 20
        number_of_iterations = 700_000
        barrier = Barrier(number_of_threads)
        def work(it):
            barrier.wait()
            while True:
                try:
                    _ = next(it)
                except StopIteration:
                    break

        data = 3
        for it in range(number_of_iterations):
            print(f'iteration {it}')
            repeat_iterator = repeat(data, 245)
            worker_threads = []
            for ii in range(number_of_threads):
                worker_threads.append(
                    Thread(target=work, args=[repeat_iterator]))

            with threading_helper.start_threads(worker_threads):
                pass

            barrier.reset()


if __name__ == "__main__":
    unittest.main()

On my system this creates a deadlock (typically after a couple of 1000) iterations in a free-threading debug build.

@kumaraditya303 kumaraditya303 merged commit 9d127e8 into python:main Apr 13, 2025
83 of 87 checks passed
@bedevere-bot
Copy link

bedevere-bot commented Apr 13, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot iOS ARM64 Simulator 3.x (tier-3) has failed when building commit 9d127e8.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1380/builds/3222) and take a look at the build logs.
  4. Check if the failure is related to this commit (9d127e8) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1380/builds/3222

Summary of the results of the build (if available):

==

Click to see traceback logs 
Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/7817F54E-F383-435E-A96B-7848C5D4910E/data/Containers/Bundle/Application/6FD8422A-BA2D-42A0-9F8A-786BED06BB0E/iOSTestbed.app/python/lib/python3.14/threading.py", line 1079, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/7817F54E-F383-435E-A96B-7848C5D4910E/data/Containers/Bundle/Application/6FD8422A-BA2D-42A0-9F8A-786BED06BB0E/iOSTestbed.app/python/lib/python3.14/threading.py", line 1021, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/7817F54E-F383-435E-A96B-7848C5D4910E/data/Containers/Bundle/Application/6FD8422A-BA2D-42A0-9F8A-786BED06BB0E/iOSTestbed.app/python/lib/python3.14/test/test_interpreters/test_stress.py", line 30, in task
    interp = interpreters.create()
  File "/Users/buildbot/Library/Developer/XCTestDevices/7817F54E-F383-435E-A96B-7848C5D4910E/data/Containers/Bundle/Application/6FD8422A-BA2D-42A0-9F8A-786BED06BB0E/iOSTestbed.app/python/lib/python3.14/test/support/interpreters/__init__.py", line 76, in create
    id = _interpreters.create(reqrefs=True)
interpreters.InterpreterError: interpreter creation failed
k

@freakboy3742
Copy link
Contributor

freakboy3742 commented Apr 17, 2025

The iOS failure is caused by #127108

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kumaraditya303 kumaraditya303 kumaraditya303 approved these changes

@rhettinger rhettinger Awaiting requested review from rhettinger rhettinger is a code owner

Assignees

No one assigned

Labels

topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@eendebakpt @rhettinger @bedevere-bot @freakboy3742 @kumaraditya303