Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: first concurrency issue in Python 3.13t #1310

Merged
merged 2 commits into from
Oct 7, 2024
Merged

fix: first concurrency issue in Python 3.13t #1310

merged 2 commits into from
Oct 7, 2024

Conversation

jpivarski
Copy link
Member

This:

executor = ThreadPoolExecutor(max_workers=16)

for array in uproot.iterate(
    [
        "/home/jpivarski/Downloads/Run2012B_DoubleMuParked.root:Events",  # https://opendata.cern.ch/record/12365
        "/home/jpivarski/Downloads/Run2012C_DoubleMuParked.root:Events",  # https://opendata.cern.ch/record/12366
    ],
    decompression_executor=executor,
    interpretation_executor=executor,   #  <--- especially this one
    handler=uproot.MemmapSource,
    step_size="1 GB",
):
    print(array)

was causing this:

  File "/home/jpivarski/storage/data/python-3.13-uproot/uproot/interpretation/numerical.py", line 108, in final_array
    basket_array = basket_arrays[basket_num]
                   ~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 290

in Python 3.13 with free-threading. But this PR fixes it.

Amusingly, it introduces a global lock. But it's a global lock on basket_array dicts, not all objects everywhere.

I've been monitoring the performance and this is even better than ignoring the problem (i.e. don't update the output array—output random bytes—in the rare cases in which the key is missing). This PR is faster and preserves correctness.

jpivarski added a commit to jpivarski-talks/2024-10-23-chep2024-gil-free-uproot that referenced this pull request Oct 3, 2024
@jpivarski
update measurements for scikit-hep/uproot5#1310
eadccec
@jpivarski jpivarski merged commit a8f674e into main Oct 7, 2024
26 checks passed
@jpivarski jpivarski deleted the jpivarski/concurrency-issue-in-313t branch October 7, 2024 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jpivarski