What does it take for pyjulia to drop GIL?

[Moelf]

Numba has a nogil annotation, I'm wondering if we can somehow drop the GIL and what would that imply.

[mkitti]

I think we would need to reformulate the binding as a C or Rust based extension rather than using ctypes

[yuyichao]

It seems that using CDLL rather than PyDLL will release the Gil.

[mkitti]

We do use CDLL for unix like systems:

pyjulia/src/julia/find_libpython.py

Line 83 in 69b734f

libdl = ctypes.CDLL(ctypes.util.find_library("dl"))

[yuyichao]

If I understand correctly, it’s the one for libjulia that matters.

pyjulia/src/julia/libjulia.py

Line 220 in 69b734f

self.libjulia = ctypes.PyDLL(libjulia_path, ctypes.RTLD_GLOBAL)

[Moelf]

yeah, I think because you're calling Julia via the libjulia, so whether or not GIL is dropped during call depends on how you loaded libjulia?

[Moelf]

maybe it's worth getting perspective from @stevengj and @MilesCranmer ?

1. Can we by default drop GIL (i.e. use CDLL)?
2. Does it actually drop GIL and speed things up?
3. Does it create corretness issues?
4. Given 3, how can we drop GIL call by call

[MilesCranmer]

I would love to drop GIL. This would be great so that I can do other things in Python (like printing updates from another thread) while some large Julia processing is happening in the background.

Regarding 3, I'm not sure what correctness issues it would cause, if any. It's unclear to me if you'd be able to mutate any Julia variables while processing is already taking place, so maybe it's safe as-is...? Maybe we should just try it out and see.

[Moelf]

yeah I'd like to try it, but I would prefer if we can present it as an option. In this case, it would be better if it can be a per-call site option, but a per-session option would do.

One case we know definitely won't work is if Julia needs to call Python stuff as call back, or mutate Python variable, but I feel like that's minor usage

[yuyichao]

I assume just having two libjulia handle should allow having this as a per call option with the ctypes api.

[Moelf]

I can't quite figure out how to have a local testable pyjulia, clone and pip install -e . didn't quite work, or maybe I'm looking at the wrong place to print debug

[Moelf]

I guess for one I can't figure out which handle is used when I do

import julia
#vs.
from julia import Base

I can't get any print statement to work -- these two must share the same Julia handle right? but which one / where I can't quite figure out

[stevengj]

See also JuliaPy/PythonCall.jl#343

[Moelf]

I think the main use case is when calling Julia code from Python, releasing the GIL would be useful.