Multiprocessing Pool Crash #846
Description
Attempting to profile programs with multiprocessing yields the following error:
Error in program being profiled:
Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
I was able to reproduce the error using the example from #809 (comment)
import multiprocessing as mp
def fun(args):
x = 0
for i in range(100):
x += 1
return args
if __name__ == "__main__":
with mp.Pool(mp.cpu_count()) as p:
r = p.map(fun, range(1000000))
print(sum(x for x in r))
print(((1000000-1) * 1000000) / 2)Expected behavior
No crash?
Screenshots
Desktop (please complete the following information):
- OS: Windows 11
- Browser Firefox
- Version 129
- Python: CPython 3.12.3
- Scalene: Repository version
Error Text
Error in program being profiled:
Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Traceback (most recent call last):
File "C:\Users\||||||||\scratch\py312\Lib\site-packages\scalene\scalene_profiler.py", line 1791, in profile_code
exec(code, the_globals, the_locals)
File "C:\Users\||||||||\scratch\scalene_mp.py", line 10, in <module>
with mp.Pool(mp.cpu_count()) as p:
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 215, in __init__
self._repopulate_pool()
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 306, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
w.start()
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\context.py", line 337, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\popen_spawn_win32.py", line 95, in __init__
reduction.dump(process_obj, to_child)
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Scalene: The specified code did not run for long enough to profile.
By default, Scalene only profiles code in the file executed and its subdirectories.
To track the time spent in all files, use the `--profile-all` option.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\spawn.py", line 113, in spawn_main
new_handle = reduction.duplicate(pipe_handle,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\reduction.py", line 79, in duplicate
return _winapi.DuplicateHandle(
^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 6] The handle is invalid