Skip to content

threads deadlock caught locally: missing safe region in _jl_mutex_wait #56638

Open
@vtjnash

Description

(rr trace available upon request)

Looks like _jl_mutex_wait fails to enter a gc-safe region inside when jl_running_under_rr is true, causing deadlocks in this situation:

# this thread (tid 50) wants GC to start (waiting on tid 1 to enter gc-safe)
unknown function (ip: 0x7fbe26a5d115) at /lib/x86_64-linux-gnu/libc.so.6                                                                                                                               
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)                                                                                                                                    
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822                                                                                                                                          
jl_gc_wait_for_the_world at /home/vtjnash/julia1/src/safepoint.c:154                                                                                                                                   
ijl_gc_collect at /home/vtjnash/julia1/src/gc-stock.c:3362                                                                                                                                             
gc at ./gcutils.jl:133 [inlined]                                                                                                                                                                       
cl at /home/vtjnash/julia1/test/threads.jl:466                                                                                                                                                         
#70 at /home/vtjnash/julia1/test/threads.jl:488                                                                                                                                                        
unknown function (ip: 0x7fbe0cb75dd5) at (unknown file)                                                                                                                                                
unknown function (ip: 0x7fbe26a60ac2) at /lib/x86_64-linux-gnu/libc.so.6                                                                                                                               
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)                                                                                                                                                
unknown function (ip: (nil)) at (unknown file)                                                                                                                                                         
unknown function (ip: 0x7fbe26a5d115) at /lib/x86_64-linux-gnu/libc.so.6                                                                                                                               
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)                                                                                                                                    
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822                                                                                                                                          
ijl_task_get_next at /home/vtjnash/julia1/src/scheduler.c:520                                      
poptask at ./task.jl:1163                                                                          
wait at ./task.jl:1172                                                                             
#wait#550 at ./condition.jl:141                                                                    
wait at ./condition.jl:136 [inlined]                                                               
take_buffered at ./channels.jl:533                                                                 
take! at ./channels.jl:527 [inlined]                                                               
#56 at /home/vtjnash/julia1/test/threads.jl:446                                                    
init_perthread at ./lock.jl:789                                                                    
getindex at ./lock.jl:746 [inlined]                                                                
OncePerThread at ./lock.jl:739 [inlined]                                                           
cl at /home/vtjnash/julia1/test/threads.jl:469                                                     
#70 at /home/vtjnash/julia1/test/threads.jl:488                                                    
unknown function (ip: 0x7fbe0cb75dd5) at (unknown file)                                            
unknown function (ip: 0x7fbe26a60ac2) at /lib/x86_64-linux-gnu/libc.so.6                           
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)                                            
unknown function (ip: (nil)) at (unknown file)                                                     

# this thread (tid 1) wants the iolock to finish running finalizers (waiting on tid 0 to release)
unknown function (ip: 0x7fbe26a5d115) at /lib/x86_64-linux-gnu/libc.so.6                                                                                                                               
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)                                                                                                                                    
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822                                                                                                                                          
_jl_mutex_wait at /home/vtjnash/julia1/src/threading.c:925                                                                                                                                             
_jl_mutex_lock at /home/vtjnash/julia1/src/threading.c:957                                                                                                                                             
jl_mutex_lock at /home/vtjnash/julia1/src/julia_locks.h:65 [inlined]                                                                                                                                   
JL_UV_LOCK at /home/vtjnash/julia1/src/jl_uv.c:137 [inlined]                                                                                                                                           
ijl_iolock_begin at /home/vtjnash/julia1/src/jl_uv.c:147                                                                                                                                               
iolock_begin at ./libuv.jl:48 [inlined]                                                                                                                                                                
uvfinalize at ./asyncevent.jl:200                                                                                                                                                                      
jfptr_uvfinalize_97523 at /home/vtjnash/julia1/usr/lib/julia/sys.so (unknown line)                                                                                                                     
_jl_invoke at /home/vtjnash/julia1/src/gf.c:3306 [inlined]                                                                                                                                             
ijl_apply_generic at /home/vtjnash/julia1/src/gf.c:3494                                                                                                                                                
run_finalizer at /home/vtjnash/julia1/src/gc-common.c:180                                                                                                                                              
jl_gc_run_finalizers_in_list at /home/vtjnash/julia1/src/gc-common.c:270                                                                                                                               
run_finalizers at /home/vtjnash/julia1/src/gc-common.c:316                                                                                                                                             
ijl_gc_collect at /home/vtjnash/julia1/src/gc-stock.c:3400                                                                                                                                             
gc at ./gcutils.jl:133 [inlined]                                                                                                                                                                       
cl at /home/vtjnash/julia1/test/threads.jl:466                                                                                                                                                         
#70 at /home/vtjnash/julia1/test/threads.jl:488                                                                                                                                                        
unknown function (ip: 0x7fbe0cb75dd5) at (unknown file)                                                                                                                                                
unknown function (ip: 0x7fbe26a60ac2) at /lib/x86_64-linux-gnu/libc.so.6                                                                                                                               
clone at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)                                                                                                                                                
unknown function (ip: (nil)) at (unknown file)

# this thread (tid 0) is holding the iolock waiting to hear that gc is finished (waiting on tid 50)
unknown function (ip: 0x7fbe26a5d115) at /lib/x86_64-linux-gnu/libc.so.6                                                                                                                               
pthread_cond_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)                                                                                                                                    
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822                                                                                                                                          
jl_safepoint_wait_gc at /home/vtjnash/julia1/src/safepoint.c:246                                                                                                                                       
jl_set_gc_and_wait at /home/vtjnash/julia1/src/safepoint.c:226                                                                                                                                         
segv_handler at /home/vtjnash/julia1/src/signals-unix.c:395 [inlined]                                                                                                                                  
segv_handler at /home/vtjnash/julia1/src/signals-unix.c:381                                                                                                                                            
unknown function (ip: 0x7fbe26a0e51f) at /lib/x86_64-linux-gnu/libc.so.6                           
ijl_gc_safepoint at /home/vtjnash/julia1/src/jlapi.c:740                                           
ijl_task_get_next at /home/vtjnash/julia1/src/scheduler.c:456                                      
poptask at ./task.jl:1163                                                                          
wait at ./task.jl:1172                                                                             
#wait#550 at ./condition.jl:141                                                                    
wait at ./condition.jl:136 [inlined]                                                               
take_buffered at ./channels.jl:533                                                                 
take! at ./channels.jl:527                                                                         
unknown function (ip: 0x7fbe0cb707f2) at (unknown file)                                            
_jl_invoke at /home/vtjnash/julia1/src/gf.c:3306 [inlined]                                         
ijl_apply_generic at /home/vtjnash/julia1/src/gf.c:3494                                            
jl_apply at /home/vtjnash/julia1/src/julia.h:2240 [inlined]                                        
do_call at /home/vtjnash/julia1/src/interpreter.c:125                                              
eval_value at /home/vtjnash/julia1/src/interpreter.c:223                                           
eval_body at /home/vtjnash/julia1/src/interpreter.c:567                                            
eval_body at /home/vtjnash/julia1/src/interpreter.c:544                                            
jl_interpret_toplevel_thunk at /home/vtjnash/julia1/src/interpreter.c:894                          
jl_toplevel_eval_flex at /home/vtjnash/julia1/src/toplevel.c:1065                                  
jl_toplevel_eval_flex at /home/vtjnash/julia1/src/toplevel.c:1005                                  
ijl_toplevel_eval at /home/vtjnash/julia1/src/toplevel.c:1076                                      
ijl_toplevel_eval_in at /home/vtjnash/julia1/src/toplevel.c:1118                                   
eval at ./boot.jl:460                            
include_string at ./loading.jl:2839              
_jl_invoke at /home/vtjnash/julia1/src/gf.c:3306 [inlined]                                         
ijl_apply_generic at /home/vtjnash/julia1/src/gf.c:3494                                            
_include at ./loading.jl:2899                    
include at ./Base.jl:300                         
macro expansion at /home/vtjnash/julia1/test/testdefs.jl:33 [inlined]                              
macro expansion at /home/vtjnash/julia1/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:1700 [inlined]                                                                                                   
macro expansion at /home/vtjnash/julia1/test/testdefs.jl:26 [inlined]                              

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    backport 1.11Change should be backported to release-1.11ciContinuous integrationmultithreadingBase.Threads and related functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions