Calling from Python multiprocessing causes hanging

[curtd]

Hello! I'm currently trying to invoke a julia function via pyjulia, which works great in the serial python case, but causes hanging for some reason when invoked in python multiprocessing pool (through a map operation, say). Has anyone else run into this problem and, if so, have they found any solution to this issue?

[tkf]

from multiprocessing import Pool


def printpid(i):
    # from julia import Julia
    # Julia(debug=True)

    from julia.Base import println, getpid
    println("pid = ", getpid(), "i = ", i)


pool = Pool()
pool.map(printpid, range(4))

works for me. Tested with master.

[tkf]

Note that multiprocessing uses threads behind the scene and julia is not thread-safe. It may segfault if you are not careful. (edit: Actually, I'm not sure about this. If Julia objects are not used in different threads, I guess this should be fine.) If you want RPC in Julia, it perhaps better to use Distributed (which is like multiprocessing in Julia).

See, e.g., https://docs.julialang.org/en/v1/stdlib/Distributed/index.html#Distributed.pmap

[curtd]

Thanks for your reply. I ended up remedying this situation by not calling pyjulia from the main Python thread but instead only calling it from the worker processes.

It's a bit of a specific scenario I'm involved in, as I'm trying to use

• a PackageCompiler compiled Julia image (with the attendant code I'm interested in running)
• pyjulia being invoked indirectly in each subprocess through a parallel pool, e.g.,

import multiprocessing as mp

def example_mp(args_list):
   with mp.Pool(4) as pool:
        results = pool.map(example_func_which_invokes_pyjulia, args_list)
   return results

I'm currently experiencing segfaults when the parallel pool is closed (the results themselves seem fine).
I think that it may have to do potentially with the issues of how Julia handles interruptions (e.g., #211), but this is only a hunch.

[tkf]

Yeah, multiprocessing could be using SIGINT for termination and it could be the problem (though I need to check the code). But if that's the case, I think you'd see something like

fatal: error thrown and no exception handler available.
InterruptException()

instead of segfault.

By "the results themselves seem fine", you mean the returned results are correct inside the with block but the segfault happens while closing the pool? If so, maybe it's something to do with interaction of GC and thread. But I'm only guessing.

[curtd]

Yes that's correct, the results computed by my function seem to be correct (in line with running them serially outside of pool). I also will occasionally receive a stacktrace like

signal (15): Terminated
in expression starting at no file:0
do_futex_wait.constprop.1 at /lib64/libpthread.so.0 (unknown line)
__new_sem_wait_slow.constprop.0 at /lib64/libpthread.so.0 (unknown line)
sem_wait at /lib64/libpthread.so.0 (unknown line)
PyThread_acquire_lock_timed at /Python-3.6.4/Python/thread_pthread.h:354
_enter_buffered_busy at /Python-3.6.4/./Modules/_io/bufferedio.c:292 [inlined]
buffered_flush at /Python-3.6.4/./Modules/_io/bufferedio.c:852
_PyCFunction_FastCallDict at /Python-3.6.4/Objects/methodobject.c:192
_PyObject_FastCallDict at /Python-3.6.4/Objects/abstract.c:2313
callmethod at /Python-3.6.4/Objects/abstract.c:2583 [inlined]
_PyObject_CallMethodId_SizeT at /Python-3.6.4/Objects/abstract.c:2685
_PyCFunction_FastCallDict at /Python-3.6.4/Objects/methodobject.c:192
call_function at /Python-3.6.4/Python/ceval.c:4824
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
_PyFunction_FastCall at /Python-3.6.4/Python/ceval.c:4906
fast_function at /Python-3.6.4/Python/ceval.c:4941 [inlined]
call_function at /Python-3.6.4/Python/ceval.c:4845
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
_PyFunction_FastCall at /Python-3.6.4/Python/ceval.c:4906
fast_function at /Python-3.6.4/Python/ceval.c:4941 [inlined]
call_function at /Python-3.6.4/Python/ceval.c:4845
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
_PyFunction_FastCall at /Python-3.6.4/Python/ceval.c:4906
_PyFunction_FastCallDict at /Python-3.6.4/Python/ceval.c:5008
_PyObject_FastCallDict at /Python-3.6.4/Objects/abstract.c:2310
_PyObject_Call_Prepend at /Python-3.6.4/Objects/abstract.c:2373
PyObject_Call at /Python-3.6.4/Objects/abstract.c:2261
slot_tp_init at /Python-3.6.4/Objects/typeobject.c:6407
type_call at /Python-3.6.4/Objects/typeobject.c:915
_PyObject_FastCallDict at /Python-3.6.4/Objects/abstract.c:2331
call_function at /Python-3.6.4/Python/ceval.c:4848
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
_PyFunction_FastCall at /Python-3.6.4/Python/ceval.c:4906
fast_function at /Python-3.6.4/Python/ceval.c:4941 [inlined]
call_function at /Python-3.6.4/Python/ceval.c:4845
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
...
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
_PyEval_EvalCodeWithName at /Python-3.6.4/Python/ceval.c:4153
PyEval_EvalCodeEx at /Python-3.6.4/Python/ceval.c:4174
PyEval_EvalCode at /Python-3.6.4/Python/ceval.c:730
run_mod at /Python-3.6.4/Python/pythonrun.c:1025 [inlined]
PyRun_FileExFlags at /Python-3.6.4/Python/pythonrun.c:978
PyRun_SimpleFileExFlags at /Python-3.6.4/Python/pythonrun.c:420
run_file at /Python-3.6.4/Modules/main.c:338 [inlined]
Py_Main at /Python-3.6.4/Modules/main.c:809
main at /Python-3.6.4/./Programs/python.c:69
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at /usr/local/bin/python3.6 (unknown line)
unknown function (ip: 0xffffffffffffffff)
Allocations: 21513853 (Pool: 21499003; Big: 14850); GC: 277

signal (11): Segmentation fault
in expression starting at no file:0
tupledealloc at /Python-3.6.4/Objects/tupleobject.c:239
BaseException_clear at /Python-3.6.4/Objects/exceptions.c:75 [inlined]
BaseException_dealloc at /Python-3.6.4/Objects/exceptions.c:86
pydecref_ at /julia/code/packages/PyCall/0jMpb/src/PyCall.jl:102 [inlined]
pydecref at /julia/code/packages/PyCall/0jMpb/src/PyCall.jl:107
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:115
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:210
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:245
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:260
jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:604
jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:342
unknown function (ip: 0xffffffffffffffff)

If that's helpful

[tkf]

These lines

pydecref_ at /julia/code/packages/PyCall/0jMpb/src/PyCall.jl:102 [inlined]
pydecref at /julia/code/packages/PyCall/0jMpb/src/PyCall.jl:107

could mean it's GC related.

But this simple script

import os
import threading
import time


def my_ident(_):
    time.sleep(0.001)
    return (os.getpid(), threading.get_ident(), threading.main_thread().ident)


if __name__ == "__main__":
    from multiprocessing import Pool
    import pprint
    num_pool = 4
    pool = Pool(num_pool)
    results = pool.map(my_ident, range(1000))
    pprint.pprint(sorted(set(results)))
    assert all(r[1] == r[2] for r in results)

shows that the function is executed in the main thread. So I'm not sure why multiprocessing could cause a problem.

Does example_func_which_invokes_pyjulia return any objects from julia like an array? If that's the case, (deep)coping it may solve the issue. (Yet another random guess.)

[tkf]

Using Distributed to fake multiprocessing is not super hard. Here is a quick hack:

# distributed.py
import os


def worker(i):
    return (i, os.getpid())


def main():
    from julia import Main
    Main.eval("""
    using Distributed
    @everywhere using PyCall
    @everywhere pushfirst!(PyVector(pyimport("sys")["path"]), @__DIR__)
    @everywhere function runpython(fullname, arg)
        m, f = rsplit(fullname, ".", limit=2)
        return pyimport(m)[f](arg)
    end

    function mappy(fullname, arglist)
        return Distributed.pmap(runpython, Iterators.cycle([fullname]), arglist)
    end
    """)
    results = Main.mappy("distributed.worker", range(3))
    print(results)


if __name__ == "__main__":
    from julia.Distributed import addprocs
    addprocs(3)
    main()

[curtd]

Unfortunately in example_func_which_invokes_pyjulia, there's an extensive python preamble and postamble surrounding the call to Julia, and the main driver program is written in python, which prevents me from porting everything to Julia (as I'd prefer to do, thus avoiding these headaches). Maybe I'll have to look into disabling garbage collection for Julia, I'm not exactly sure what the way forward should be to avoid this

[tkf]

In my example, each Julia worker calls Python function distributed.worker. You can run Python code inside it.

Side note: To use PyJulia inside worker function, use Julia(init_julia=False) to initialize it without extra overhead.

def worker(i):
    from julia import Julia
    Julia(init_julia=False)

    from julia import ...
    # do Python & Julia stuff

[tkf]

JuliaPy/PyCall.jl#603 may solve this issue

[curtd]

I will give it a try, thank you!

[tkf]

Also try pyjulia master now that #216 is merged.

[curtd]

Hmm unfortunately both of those changes still result in similar stacktrace dumps

signal (15): Terminated

signal (15): Terminated
in expression starting at no file:0
do_futex_wait.constprop.1 at /lib64/libpthread.so.0 (unknown line)
__new_sem_wait_slow.constprop.0 at /lib64/libpthread.so.0 (unknown line)
sem_wait at /lib64/libpthread.so.0 (unknown line)
PyThread_acquire_lock_timed at /Python-3.6.4/Python/thread_pthread.h:354
_enter_buffered_busy at /Python-3.6.4/./Modules/_io/bufferedio.c:292 [inlined]
buffered_flush at /Python-3.6.4/./Modules/_io/bufferedio.c:852
_PyCFunction_FastCallDict at /Python-3.6.4/Objects/methodobject.c:192
_PyObject_FastCallDict at /Python-3.6.4/Objects/abstract.c:2313
callmethod at /Python-3.6.4/Objects/abstract.c:2583 [inlined]
_PyObject_CallMethodId_SizeT at /Python-3.6.4/Objects/abstract.c:2685
_PyCFunction_FastCallDict at /Python-3.6.4/Objects/methodobject.c:192
call_function at /Python-3.6.4/Python/ceval.c:4824
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
_PyFunction_FastCall at /Python-3.6.4/Python/ceval.c:4906
fast_function at /Python-3.6.4/Python/ceval.c:4941 [inlined]
call_function at /Python-3.6.4/Python/ceval.c:4845
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
_PyFunction_FastCall at /Python-3.6.4/Python/ceval.c:4906
fast_function at /Python-3.6.4/Python/ceval.c:4941 [inlined]
call_function at /Python-3.6.4/Python/ceval.c:4845
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
_PyFunction_FastCall at /Python-3.6.4/Python/ceval.c:4906
_PyFunction_FastCallDict at /Python-3.6.4/Python/ceval.c:5008
_PyObject_FastCallDict at /Python-3.6.4/Objects/abstract.c:2310
_PyObject_Call_Prepend at /Python-3.6.4/Objects/abstract.c:2373
PyObject_Call at /Python-3.6.4/Objects/abstract.c:2261
slot_tp_init at /Python-3.6.4/Objects/typeobject.c:6407
type_call at /Python-3.6.4/Objects/typeobject.c:915
_PyObject_FastCallDict at /Python-3.6.4/Objects/abstract.c:2331
call_function at /Python-3.6.4/Python/ceval.c:4848
_PyEval_EvalFrameDefault at /Python-3.6.4/Python/ceval.c:3322
_PyFunction_FastCall at /Python-3.6.4/Python/ceval.c:4906
...
PyEval_EvalCodeEx at /Python-3.6.4/Python/ceval.c:4174
PyEval_EvalCode at /Python-3.6.4/Python/ceval.c:730
run_mod at /Python-3.6.4/Python/pythonrun.c:1025 [inlined]
PyRun_FileExFlags at /Python-3.6.4/Python/pythonrun.c:978
PyRun_SimpleFileExFlags at /Python-3.6.4/Python/pythonrun.c:420
run_file at /Python-3.6.4/Modules/main.c:338 [inlined]
Py_Main at /Python-3.6.4/Modules/main.c:809
main at /Python-3.6.4/./Programs/python.c:69
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at /usr/local/bin/python3.6 (unknown line)
unknown function (ip: 0xffffffffffffffff)
Allocations: 30387807 (Pool: 30360075; Big: 27732); GC: 568

signal (11): Segmentation fault
in expression starting at no file:0
tupledealloc at /Python-3.6.4/Objects/tupleobject.c:239
BaseException_clear at /Python-3.6.4/Objects/exceptions.c:75 [inlined]
BaseException_dealloc at /Python-3.6.4/Objects/exceptions.c:86
pydecref_ at /julia/code/packages/PyCall/0jMpb/src/PyCall.jl:102 [inlined]
pydecref at /julia/code/packages/PyCall/0jMpb/src/PyCall.jl:107
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:115
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:210
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:245
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:260
jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:604
jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:342
unknown function (ip: 0xffffffffffffffff)
Allocations: 30395394 (Pool: 30367660; Big: 27734); GC: 568

[tkf]

From the path component PyCall/0jMpb, I suppose you are using PyCall v1.18.5. Did you try the combination of #216 and JuliaPy/PyCall.jl#603?