PyJulia does not work with Julia 1.0 & 0.7 with Python installed via Conda/Ubuntu/(what else?)

[tkf]

I'm writing down the issue so that it is easy to find by the users.

At the moment, for PyJulia to load Julia >= 0.7, two conditions have to be met:

1. Python interpreter configured for PyCall.jl and Python interpreter used to import PyJulia have to use identical libpython (once Support virtualenv/venv without separate cache #190 is merged) have exactly identical paths.

2. The Python interpreter have to be dynamically linked (which is usually the case in Windows and macOS according to this comment)

To check if condition 2 is the case, run:

$ ldd /usr/bin/python | grep libpython
        libpython3.7m.so.1.0 => /usr/lib/libpython3.7m.so.1.0 (0x00007f17c12c4000)

where /usr/bin/python is the appropriate Python interpreter.

If it does not print anything, you can't use PyJulia with this Python interpreter.

[mprogram]

This was very helpful; thank you, @tkf

I must say I was very-very surprised by not finding libpython3 in my ldd $(which python3) output.

Debian/Ubuntu packagers are the known strong advocates against linking things statically; but doing so in python/3?!?!

[tkf]

Yeah, I was surprised too. I thought it was just conda (in which static compilation maybe makes sense).

[Hong-Xiang]

@tkf Thanks for your reply. But there are still some questions:

• Currently we do not have any method to call Julia in conda based/Ubuntu/Debian python3, right? Even dirty/tricky method would be helpful.
• Could we use custom compiled python for it? If true, is this a practical solution?
• Will this issue be solved in near future or we should not expect that?

Thanks.

[tkf]

I think there are three ways to use Python with Julia 1.0 on Ubuntu/Debian/etc. at the moment:

• If you just want to use your Python module, you can import it from PyCall.jl and call the functions in it. See, e.g., https://github.com/tkf/IPython.jl/blob/d5b27ac6e38f1b96ba42099da1098975cdd8b9be/src/core.jl#L31 and https://github.com/tkf/IPython.jl/blob/d5b27ac6e38f1b96ba42099da1098975cdd8b9be/src/core.jl#L20 in IPython.jl/src/core.jl.
• If you just want an interactive Python session, you can use my IPython.jl to drop into IPython through Julia REPL.
• If you want fully functional PyJulia in Python, one way to do it is though docker. Check out: https://github.com/tkf/docker-ipyjulia

Will this issue be solved in near future

I think I know how to fix it #173. But likely not in next weeks. I can't grant anything :) If I were to do it I want to do it after dropping Julia 0.6 (which requires PyJulia to be released soon).

[tkf]

I made an experimental package julia-venv: https://github.com/tkf/julia-venv

It implements the private DEPOT_PATH hack mentioned in #173 but does so per-virtualenv basis. This let us use different Python executables without recompiling PyCall.jl all the time. On top of creating a private DEPOT_PATH, it installs a CLI julia-venv which acts like the normal julia interpreter (by invoking it via C API from Python). This way, we can run PyCall.jl always in "PyJulia-mode" so that the issue with Python in Ubuntu/Debian and Conda can be avoided (https://github.com/tkf/julia-venv#how-it-works). As a nice side-effect, you can also use PyCall.jl with different Python executable and different set of Python libraries.

Edit: Actually, julia-venv probably still doesn't work with Conda (and probably also with Ubuntu): tkf/julia-venv#2 Edit^2: It works now.

[Hong-Xiang]

@tkf ，thanks for your awesome work. The new package works on Ubuntu 16.04 with anaconda. It really helps.

There are some minor suggestions (or some misunderstanding of me).

third-party packages need to be re-compiled/re-installed

for test purpose, I test from julia import NPZ, while NPZ is already added in original julia environment. But it failed.

and the following code works:

from julia import Pkg
Pkg.add("NPZ")
#... some install outputs
from julia import NPZ
data = NPZ.npzread("path")

It seems that pyjulia and julia of system is using different package cache? I think re-install packages is not a problem and it might be a valuable feature. I only suggest add some doc for this

convenient broadcast call

broadcast is commonly used in julia, but Main.abs.([1,2,3]) is a SyntaxError in python.
in README.md of pyjulia, there is

Main.eval("sin.(xs)")

which seems that pyjulia will not expose broadcast method to python level. I'm not sure if I am right, but if it is true, I'm thinking is it possible to adding some special syntax like Main.abs.broadcast([1,2,3]) or in short Main.abs.b([1,2,3]) (although may introduce some confusing to sum_b for python version sum!).

InterruptException() handling

When I press ctrl-C in python REPL, there would be a small problem.
If I'm using a pure python REPL, it should work like this

>>> # presss ctrl-C here
KeyboardInterrupt
>>> 

But in a python REPL which loaded julia, it would work like this:

>>> # press ctrl-C here

and nothing happens, after that if I enter next command

>>> # Ctrl - C, and nothing happens
>>> Main.xs = [1,2,3] # Enter, change line and type new command and Enter I got:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 140, in __setattr__
    self._julia.eval(setter)(value)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 606, in eval
    ans = self._call(src)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 562, in _call
    self.check_exception(src)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 589, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'InterruptException' occurred while calling julia code:

            PyCall.pyfunctionret(
                (x) -> eval(Main, :(xs = $x)),
                Any,
                PyCall.PyAny)

It seems the InterruptException is send to julia in one delayed statement?

[tkf]

@Hong-Xiang Hey, thanks for the feedback. Are you using julia-venv in Ubuntu or Debian? Did it actually work?

I think re-install packages is not a problem and it might be a valuable feature.

Yeah, something like virtualenv's --system-site-packages would be nice. I actually did that in first versions but stopped doing that since I encountered with some quirks in julia's package handling. It it actually as "easy" as replacing depots = filter(!isequal(default_depot1), __py_julia_venv_orig_DEPOT_PATH) with depots = copy(__py_julia_venv_orig_DEPOT_PATH).

convenient broadcast call

I started experimenting more numpy-like interface on top of PyJulia (i.e., broadcast operations on arrays by default):

https://ipyjulia-hacks.readthedocs.io/en/latest/

Not sure if it is suitable for PyJulia, though.

But if you want to leverage Julia, I think it's better to just use Julia syntax. If you don't want to use global variable (of course), you can do:

Main.eval("(xs) -> sin.(exp.(xs) .+ 1)")(xs)

or equivalently

Main.eval("(xs) -> @. sin(exp(xs) + 1)")(xs)

Unlike numpy.exp etc., it doesn't allocate intermediate arrays.

InterruptException

Hmm... I didn't notice that. But yes, I can reproduce it. Actually, I just noticed that it's reported in #189. Let's track the issue there.

It looks like IPython 7 (development version) handle ctrl-C differently and it doesn't have this problem. I've been using it so I didn't notice the problem.

[Hong-Xiang]

@tkf my system/environment info is:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.5 LTS
Release:	16.04
Codename:	xenial

and actually it's Ubuntu Mate.

conda:

$ conda --version
conda 4.3.30

python(in conda virutal env, with julia-venv)

$ source activate juliaenv

(juliaenv) $ which python
(home path)/anaconda3/envs/juliaenv/bin/python

(juliaenv)  $ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia_venv import get_julia
>>> get_julia()
<julia.core.Julia object at 0x7f66c6cc7f28>
>>> from julia import Main

it worked, without error.

python(in conda virtual env, without julia-venv)

(juliaenv)  $ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia import Main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 172, in load_module
    JuliaMainModule(self, fullname))
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 77, in __init__
    self._julia = loader.julia
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 164, in julia
    self.__class__.julia = julia = Julia()
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 496, in __init__
    "\"lib\", \"pyjulia%s-v$(VERSION.major).$(VERSION.minor)\"))" % sys.version_info[0])
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 562, in _call
    self.check_exception(src)
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 589, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'UndefVarError' occurred while calling julia code:
unshift!(Base.LOAD_CACHE_PATH, abspath(Pkg.Dir._pkgroot(),"lib", "pyjulia3-v$(VERSION.major).$(VERSION.minor)"))
>>> 

tests like using standard libraries, basic array operations and even third-party packages like NPZ is performed, and it seems just worked, as mentioned above.
Is there any further tests I can do for this?

[tkf]

Thanks. Interesting... Actually, the problem I thought I had was already fixed tkf/julia-venv#2. My guess was that it was some kind of IO caching issue (compilation file was not properly written before exiting the problem). It then maybe explains why it's machine-dependent.

Anyway, I'm glad that it worked in your machine!

[tkf]

For Ubuntu/Debian users, probably the cleanest and relatively easy way to use PyJulia at the moment is to install (build) Python using pyenv. But you need to pass --enable-shared manually.

$ PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.6
Downloading Python-3.6.6.tar.xz...
-> https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz
Installing Python-3.6.6...
Installed Python-3.6.6 to /home/takafumi/.pyenv/versions/3.6.6

  200.54s user 16.69s system 271% cpu 1:20.10 total
$ ldd ~/.pyenv/versions/3.6.6/bin/python3.6
        linux-vdso.so.1 =>  (0x00007fff6933c000)
        libpython3.6m.so.1.0 => /home/takafumi/.pyenv/versions/3.6.6/lib/libpython3.6m.so.1.0 (0x00007fca44c8b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fca44a6e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fca446a4000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fca444a0000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fca4429d000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fca43f94000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fca451c0000)
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

[tkf]

Or you can switch to Arch Linux 😆

It uses --enable-shared: https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python&id=68e188a1f975dcc6798072bcda4ef35abfe162ed#n57

[tkf]

@ExpandingMan @mprogram So it turned out static linking is just the default behavior of Python's ./configure script (see my message above #185 (comment)). I guess Debian is not doing anything particularly evil here.

[mprogram]

Thanks, @tkf for your last two comments. I'm a fan of deb-based system and my thought would be that Debian needs to look at those. If I'll be up to recompile Python (for the sake of optimization with MKL, for example as Julia does), I would include the --enable-shared option in my build.

[tkf]

@mprogram It looks like pyenv does something more than passing --enable-shared to ./configure:

• https://github.com/pyenv/pyenv/tree/master/plugins/python-build#building-with---enable-shared
• https://github.com/pyenv/pyenv/blob/5603eb51c93bf053f2b03e3d80a9d0be62dcdeec/plugins/python-build/bin/python-build#L2056

I didn't check if ./configure --enable-shared was enough when building without pyenv.

[ExpandingMan]

This problem is not going to go away, Debian is not going to change it and lots and lots of people are going to try this as debian distros are the most popular (though I prefer Arch I still have to use them often myself). My first question is how other languages solve this problem. We can't be the first people ever to want to use a dynamically linked Python executable. It would be good to see how Scala and Java solve this, though I haven't started looking yet.

I think at a minimum what we need is a very clear set of the simplest possible steps that one would take to get pyjulia running on debian in a very visible location right in the packages readme or documentation. Thanks to @tkf to taking the first steps to this with his pyenv setup, I'm going to try it myself when I get a chance.

[tkf]

Yeah "switch to Arch Linux" was rather a joke... 😉 It definitely is the biggest TODO in this package to support Debian family out-of-the-box.

My first question is how other languages solve this problem.

I don't think the question is how to solve the problem. In principle there is a straightforward solution: just compile different cache of PyCall.jl and all other packages depending on it to different files. For example, here is an idea to implement it: JuliaLang/julia#27418 (comment). The question is when that happens. Or maybe it's more like if we can convince Julia core devs to include enough features required by PyJulia.

Until then, here is yet another solution: #200. The idea is to bundle python-jl executable in PyJulia package which can be used to run your Python script instead of the usual python executable.

[ExpandingMan]

I think I'm rather confused about what the issue is here. I thought the problem was that since the debian Python binary is statically linked, using libpython doesn't do you any good because you can't interact with the process started by python. What I don't understand is what this has to do with the pre-compile process on the Julia side. I would think that pyjulia is just talking to the julia process with libjulia and that you'd be perfectly able to use everything that was already precompiled for your regular Julia process. Care to clarify? (reading this thread did not clear it up for me)

[tkf]

I thought the problem was that since the debian Python binary is statically linked, using libpython doesn't do you any good because you can't interact with the process started by python.

You can interact with C API of statically linked python. For example, Python itself provides ctypes.pythonapi for such purpose. Actually, python statically linked to libpython worked in julia 0.6 because it was easier to monkey-patch precompilation mechanism of julia 0.6.

The reason why it's related to Julia's precompilation mechanism is explained in julia/fake-julia/README. If you look at this if block in src/startup.jl in PyCall.jl, you can see that it generates different macros depending on libpython is statically linked or not (libpython === nothing or not). This if block is run at compile time and those macros are used all over the places so the functions in PyCall.jl becomes incompatible for the program statically linked to libpython (e.g., python in Ubuntu) and the program dynamically linked libpython (julia). Apparently defining those different set of macros was important for improving the load time of PyCall.jl (JuliaPy/PyCall.jl#167) so I guess we can't do the branching at runtime in PyCall.jl (I'm mentioning it since that was my question: JuliaPy/PyCall.jl#528).

[tkf]

I merged a workaround for statically linked Python #200. Basically, you "just" need to run python-jl script.py (notice the -jl suffix) instead of python script.py if you use Debian or conda. You can try it by:

pip install https://github.com/JuliaPy/pyjulia/archive/master.zip#egg=julia

[ExpandingMan]

Awesome job! Thanks so much for your work on this. This was a really awkward problem in which we were basically just getting screwed by the distros (and sadly some of the most important ones) and there was really never going to be a perfect solution, this seems to be about the best we'll be able to do with the situation we've been put in.

If you could, I think it would be really great if you could explain this as much as possible in the README (I won't offer to do it myself since I'm not sure I fully understand what you did yet).

Thanks again!

[tkf]

Yep, README is the only blocker before the release ATM.

[bstellato]

Guys I might have found a solution to this! If you install python from conda-forge there is no need for python-jl. Try this

conda install -c conda-forge python

python -c "import julia.Main"

It should work. See #212 (comment)

[ExpandingMan]

How is that a better solution than python-jl? Personally I'd prefer python-jl running on the system Python binaries, but I suppose it's nice to have as many options as possible.

[bstellato]

If you are using Conda you are already not using the system Python binaries. In the other cases, I agree there is still need for python-jl. Also, there can still be issues with python-jl #208.

[ExpandingMan]

Agreed, let's just aim to document as many options as possible and people will be able to select whatever solution works best for them.

[tkf]

@bstellato Thanks!. This is a good news! So I guess conda-forge distributes a Python executable dynamically linked to libpython. I'll add it to the documentation.

@ExpandingMan Yeah, I think python-jl should stay. Another benefit of python-jl is that signal handler is slightly well behaved this way (libjulia "steals" signal handler from python #211).

[tkf]

It's a bit strange. 3.7.0 from conda-forge is statically linked:

$ conda create --prefix conda-forge-python conda-forge::python
...
The following NEW packages will be INSTALLED:
...
    python:          3.7.0-h5001a0f_4        conda-forge
...
$ conda-forge-python/bin/python
Python 3.7.0 | packaged by conda-forge | (default, Sep 30 2018, 14:56:18)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ ldd conda-forge-python/bin/python
        linux-vdso.so.1 (0x00007ffec5dce000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f86eb290000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f86eb28b000)
        libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f86eb286000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f86eb27c000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f86eb0f7000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f86eaf33000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f86eb2e4000)
$ conda-forge-python/bin/python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
None

But conda-forge::python=3.6 is dynamically linked.

$ rm -r conda-forge-python
$ conda create --prefix conda-forge-python conda-forge::python=3.6
...
The following NEW packages will be INSTALLED:
...
    python:          3.6.6-h5001a0f_3        conda-forge
...
$ ldd conda-forge-python/bin/python
        linux-vdso.so.1 (0x00007ffff0f7e000)
        libpython3.6m.so.1.0 => /home/takafumi/repos/pyjulia/conda-forge-python/bin/../lib/libpython3.6m.so.1.0 (0x00007f1f758a8000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1f75856000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f1f75851000)
        libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f1f7584c000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f1f75842000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f1f756bd000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1f754f7000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1f75de7000)
$ conda-forge-python/bin/python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
/home/takafumi/repos/pyjulia/conda-forge-python/lib/libpython3.6m.so.1.0

[bstellato]

Version 3.7.0 is still dynamically linked for me on mac os.

conda create -n testpyjulia37 conda-forge::python
conda activate testpyjulia37
otool -L $(which python)

I get

/Users/xxx/miniconda3/envs/testpyjulia37/bin/python:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1153.18.0)
	@rpath/libpython3.7m.dylib (compatibility version 3.7.0, current version 3.7.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Also,

> python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
/Users/xxxx/miniconda3/envs/testpyjulia37/lib/libpython3.7m.dylib

[tkf]

Interesting. I asked the question in conda-forge: conda-forge/python-feedstock#222

[tkf]

I wrote README in #210 and you can read it here: https://github.com/tkf/pyjulia/blob/readme/README.md

@ExpandingMan If you are interested, have a look at Troubleshooting and pre-compilation mechanism sections:
https://github.com/tkf/pyjulia/blob/readme/README.md#troubleshooting
https://github.com/tkf/pyjulia/blob/readme/README.md#pre-compilation-mechanism-in-julia-10

[tkf]

I just merged #256 to master. I think PyJulia is now usable with many CPython executables if you can compile a custom system image with python3 -m julia.sysimage command. The bonus point is that this makes PyJulia startup virtually instantaneous. See: https://pyjulia.readthedocs.io/en/latest/sysimage.html

[tk3369]

Hi @tkf - it's annoying having to deal with this issue. Do you know which Linux distro (ideally docker container) that provides a dynamically linked Python?

[tkf]

It works in Arch Linux :)

IIRC, the official python docker works. It's a bit old, but here is an example for setting up PyJulia in Docker: https://github.com/tkf/docker-pyjulia

Also, just FYI, I started working on compile-time preference integration for PyCall JuliaPy/PyCall.jl#835. If it works as I expected, PyJulia should just work without any hacks in Julia 1.6 and above. Obviously, there are still some unknowns as I haven't finished implementing all. So, let's see ;)

[Ge0rges]

What's the status of this issue? I am trying to run PyJulia in Ubuntu 22 but python is statically linked.

[EmanuelCastanho]

Is it possible to build my own Python with --enable-shared in Conda?

[mkitti]

I think that might be a question for conda-forge. If you are going to compile it yourself, do you actually need conda though?

Perhaps we could use https://github.com/JuliaBinaryWrappers/Python_jll.jl

[traversaro]

Just FYI, it seems that also python-build-standalone is considering switching to static builds of python for performance gains: astral-sh/python-build-standalone#540 .

[traversaro]

Just FYI, it seems that also python-build-standalone is considering switching to static builds of python for performance gains: astral-sh/python-build-standalone#540 .

Ah, it seems that this package is now deprecated, so it may not be a big problem: #556 .