Skip to content

Note multiprocessing start method compatibility #529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Note multiprocessing start method compatibility #529

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4a99ba3
Note multiprocessing start method compatibility
Mark2000 Apr 4, 2023
af85977
Update docs/source/limitations.rst
Mark2000 Apr 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/source/limitations.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,23 @@ run:
...: a
Out[3]: array([1., 1., 1., 2., 2., 2., 3., 3., 4., 4.])

Multiprocessing
~~~~~~~~~~~~~~~

While not supported or guaranteed to be stable, PyJulia tends to work
with the multiprocessing library if the correct `start method
<https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods>`_
is selected: *fork* (default on Unix) segfaults with with `unknown
function` errors, while *spawn* (default on Windows and MacOS) generally,
but not always, runs without memory allocation issues. To select *spawn*,
include the line
Comment on lines +76 to +79
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
is selected: *fork* (default on Unix) segfaults with with `unknown
function` errors, while *spawn* (default on Windows and MacOS) generally,
but not always, runs without memory allocation issues. To select *spawn*,
include the line
is selected. *fork* (default on Unix) segfaults with with `unknown
function` errors. *spawn* (default on Windows and MacOS) generally,
but not always, runs without memory allocation issues. To select *spawn*,
include the line

Just using . here simplifies sentence structure. When does using spawn run with memory allocation issues? Could you elaborate on that point?

Copy link
Author

@Mark2000 Mark2000 Apr 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish I could, but I haven't done extensive enough testing to say why it happens. I've just noted that it does occasionally happen in my use case. @davidavdav is this something you've also experienced?


.. code:: python

multiprocessing.set_start_method('spawn')

once in your code prior to using the multiprocessing library.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
once in your code prior to using the multiprocessing library.
once in your code prior to using the multiprocessing library.
Also, consider using the *spawn* or *forkserver* methods with ``multiprocessing.get_context``:
.. code:: python
context = multiprocessing.get_context('fork_server')
queue = context.Queue()
process = context.Process(target=foo, args=(queue,))

@davidavdav does that sound right?


PyJulia does not release GIL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down