Numba threading issues with v0.5.0 and above

[tmillenaar]

PySE master cannot be used over multiple threads.

I feel like this is a recent development since I was able to run pyse in multiple threads before (in v0.4.0 to be exact). Note that in the following example already has allow_multiprocessing set to False. This is to avoid confusion as to where the threading problem comes from but also to avoid the issue described here: #158

import threading

from sourcefinder.accessors import open as pyse_open
from sourcefinder.accessors import sourcefinder_image_from_accessor
from sourcefinder.config import Conf

def process_image():
    path = "test/data/GRB201006A_final_2min_srcs-t0002-image-pb_cutout.fits"
    im = pyse_open(path)
    pyse_im = sourcefinder_image_from_accessor(im, conf=Conf({"allow_multiprocessing":False},{}))
    results = pyse_im.extract(det=5,anl=5)

if __name__ == "__main__":
    thread_1 = threading.Thread(target=process_image)
    thread_2 = threading.Thread(target=process_image)

    thread_1.start()
    thread_2.start()

    thread_1.join()
    thread_2.join()

When I run this I get the following error:

Numba workqueue threading layer is terminating: Concurrent access has been detected.

 - The workqueue threading layer is not threadsafe and may not be accessed concurrently by multiple threads. Concurrent access typically occurs through a nested parallel region launch or by calling Numba parallel=True functions from multiple Python threads.
 - Try using the TBB threading layer as an alternative, as it is, itself, threadsafe. Docs: https://numba.readthedocs.io/en/stable/user/threading-layer.html

If you read the docs that the error points to it will tell you to use another package that can be used as a threadsafe runtime like tbb or openmp. While this works, I would prefer not to add an extra multithreaded runtime on top of the one I am already running. Besides, I was able to get it to run on my laptop but not on my desktop with errors like "ValueError: No threading layer could be loaded.", which occurs because I apparently have some outdated software installed unrelated to pip. I am not eager to dive into this conflict since I would prefer not to have tbb or openmp in the first place.

I can get it to run if I set all target="parallel" to target="cpu". That way numba does not try to spin up a multithreaded runtime. I understand we want to keep the parallel option as a default for people who use PySE from the command line, but can we maybe add the option to run with target set to 'cpu'?

Cheers,
Timo

P.S.
If you want to replicate the example in version v0.4.0 you need to use results = pyse_im.extract(5,5) since it does not yet use the config object for all parameters. When that runs I do however get the warning:

/media/timo/samsung_870_evo/software/astron/pyse/venv/lib/python3.11/site-packages/numba/np/ufunc/parallel.py:371: NumbaWarning: The TBB threading layer requires TBB version 2021 update 6 or later i.e., TBB_INTERFACE_VERSION >= 12060. Found TBB_INTERFACE_VERSION = 12050. The TBB threading layer is disabled.

[HannoSpreeuw]

I can get it to run if I set all target="parallel" to target="cpu".

I guess you mean here, right?
Yeah, setting that to "parallel" makes a considerable speed difference.

If you want to replicate the example in version v0.4.0 you need to use results = pyse_im.extract(5,5) since it does not yet use the config object for all parameters.

That should no longer be the case for the current master, after PR #157 , i.e. conf sets detection and analysis thresholds.

[HannoSpreeuw]

You can exclude that this has something to do with Python version, right?

[tmillenaar]

You can exclude that this has something to do with Python version, right?

Yeah indeed, I just tried a fresh install of python3.11 and there it fails as well. So this issue is indeed not nesecarily related to the python version, though it seems that getting tbb to work is sometimes python version dependent.

I realize now however, in an attempt to address warnings we got before regarding the numba threading layer I forced a workqueue here: f88d088

But now we use the parallel targets this seems I shot myself in the foot.

[tmillenaar]

As a suggestion, to avoid setting the target on import of the module, what do you think of something like the following:

def interp_per_row(grid_row, y_initial, y_sought, interp_row, numba_target="parallel"):
    interp_per_row_compiled = guvectorize(
        _interp_per_row,
        ["void(float32[:], float32[:], float32[:], float32[:])"],
        "(n),(n),(k)->(k)",
        target=numba_target,
        nopython=True,
        cache=True,
    )
    interp_per_row_compiled(grid_row, y_initial, y_sought, interp_row)

def _interp_per_row(grid_row, y_initial, y_sought, interp_row):
    interp_row[:] = np.interp(y_sought, y_initial, grid_row)

And similar for other places where we set target="parallel".
This would allow us to call the function with an argument to specify the target and we could derive this value from the config is so desired. Do you think this would be a viable approach?

[HannoSpreeuw]

But now we use the parallel targets this seems I shot myself in the foot.

Perhaps I should try to play with numba_config.THREADING_LAYER first to see if it can solve our problems.

[HannoSpreeuw]

This would allow us to call the function with an argument to specify the target and we could derive this value from the config is so desired. Do you think this would be a viable approach?

Yes, it could, but I am hesitant because of the complexity it adds. Actually, I'd rather not go down that alley.

All of this never occurs when running PySE standalone, so I am not sure what level of priority this should be given.

[tmillenaar]

Yes, it could, but I am hesitant because of the complexity it adds. Actually, I'd rather not go down that alley.

Yeah that is understandable. We can see what it takes to make it work on the TraP end and compare it to a potential change in PySE. At the very least though to make it work as long as parallel=True somewhere in the codebase we would need to remove the forced workqueue as threading layer: #160

[HannoSpreeuw]

At the very least though to make it work as long as parallel=True somewhere in the codebase we would need to remove the forced workqueue as threading layer: #160

@tmillenaar With #160 merged, can this issue be closed?

[tmillenaar]

Yeah let's close it. With that merged I can maybe do some gymnastics on the trap end to work around the parallel calls. I will re-open an issue if I cannot make it work but I assume we should be fine from here.

Thanks for baring with me on this one ^^