Pathos still giving pickle error

[NeelKanwal]

Hi,

I switches to Pathos.multiprocessing instead of multiprocessing but still getting a similar error with pickle dump.

Here is simplifed version of the code

def create_patches(location, file, path, 
    patch_folder, workers=1, patch_size=224, mask_overlap= 95.0):
    # Calculate something
    # pass input to next function.

  def extract_and_save_patch(x_cord, y_cord, file_path=file_pth, file_name=file, mask_path=mask_path,
                                 patch_folder=patch_folder, patch_size=patch_size, mask_overlap=mask_overlap):
          # extract a small peice of image based on x_cord and y_cord from big iage
          # save the image

        
    def multiprocess_passer(x_cord, y_cord):
        print(f"creating patches for ({x_cord},{y_cord})")
        extract_and_save_patch(x_cord, y_cord)


     list_cord = [(x, y) for x in range(0, n_across) for y in range(0, n_down)]
     with Pool(processes=workers) as p:
            p.map(extract_and_save_patch, list_cord)

create_patches(location, file, path, patch_folder, workers=num_workers)

I want to pass a tuple to function every time and leave other argument default which load the files and mask. But still getting the same error which appears in multiprocesisng library.

This the error trace.

Traceback (most recent call last):
  File "multiprocess.py", line 137, in <module>
    create_patches(location, file, path, patch_folder, workers=num_workers)
  File "multiprocess.py", line 130, in create_patches
    p.map(multiprocess_passer, list_cord)
  File "/home/neel/miniconda3/envs/process/lib/python3.7/site-packages/pathos/multiprocessing.py", line 135, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/neel/miniconda3/envs/process/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'starargs.<locals>.<lambda>'

Anyone faced a similar issue or solved it?

Thanks

[mmckerns]

What version of pathos are you using? What version of multiprocess and dill are you using? If I look at the line numbers in your traceback, they don't correspond with the current code for pathos. I also don't see how your code is falling back to multiprocessing instead of multiprocess. multiprocessing doesn't use dill, it uses pickle... and thus you'd see the picking error. Is multiprocess (and dill) installed? I don't see why your code should be using multiprocessing unless multiprocess isn't installed. Also, it'd be good if you could post a minimal working example so that I can actually test it and see if I get the same behavior. If I can reproduce what you have, it's easier to solve why it's occurring. Without a minimal working example, it's much harder to diagnose what is going on.

[dougwood]

I am also getting a can't pickle an object error using pathos ProcessPool . But my example is pretty complex, so it is very hard to create something smaller and reproducible.

I am using pathos 0.3.0 and multiprocess 0.70.14

The failure I am having is in trying to pickle an class from the module pylibczirw

I have included the traceback below if that is any help.

I do notice from the traceback that the root exception is coming from line 388 in _dill and that looks like some sort of attempt to try using StockPIckler.save(), so it would appear that dill does use pickle in some situations.

File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 578, in save
rv = reduce(self.proto)
TypeError: cannot pickle '_pylibCZIrw.czi_reader' object

and if you look at the code in _dill.py near line 388 (last line in the snippet below) you can see:

    if GENERATOR_FAIL and type(obj) == GeneratorType:
        msg = "Can't pickle %s: attribute lookup builtins.generator failed" % GeneratorType
        raise PicklingError(msg)
    StockPickler.save(self, obj, save_persistent_id)

Here is a more complete traceback:

File "C:\dev\ultistacker.ai\ultistacker\Stacker.py", line 923, in process_tiles
self.process_tiles_parallel(round_index, def_field, A_global)
File "C:\dev\ultistacker.ai\ultistacker\Stacker.py", line 1247, in process_tiles_parallel
res = pp.map(self._process_one_tile, args)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\pathos\multiprocessing.py", line 135, in map
return _pool.map(star(f), zip(*args)) # chunksize
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\pool.py", line 771, in get
raise self._value
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\pool.py", line 537, in _handle_tasks
put(task)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\connection.py", line 214, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\multiprocess\reduction.py", line 54, in dumps
cls(buf, protocol, *args, **kwds).dump(obj)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 394, in dump
StockPickler.dump(self, obj)
File "C:\Program Files\Python39\lib\pickle.py", line 487, in dump
self.save(obj)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python39\lib\pickle.py", line 901, in save_tuple
save(element)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python39\lib\pickle.py", line 886, in save_tuple
save(element)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python39\lib\pickle.py", line 886, in save_tuple
save(element)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1824, in save_function
_save_with_postproc(pickler, (_create_function, (
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1089, in _save_with_postproc
pickler.save_reduce(*reduction)
File "C:\Program Files\Python39\lib\pickle.py", line 692, in save_reduce
save(args)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python39\lib\pickle.py", line 886, in save_tuple
save(element)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1427, in save_instancemethod0
pickler.save_reduce(MethodType, (obj.func, obj.self), obj=obj)
File "C:\Program Files\Python39\lib\pickle.py", line 692, in save_reduce
save(args)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python39\lib\pickle.py", line 886, in save_tuple
save(element)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "C:\Program Files\Python39\lib\pickle.py", line 717, in save_reduce
save(state)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "C:\Program Files\Python39\lib\pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "C:\Program Files\Python39\lib\pickle.py", line 997, in _batch_setitems
save(v)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Program Files\Python39\lib\pickle.py", line 931, in save_list
self._batch_appends(obj)
File "C:\Program Files\Python39\lib\pickle.py", line 955, in _batch_appends
save(x)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "C:\Program Files\Python39\lib\pickle.py", line 717, in save_reduce
save(state)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "C:\Program Files\Python39\lib\pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "C:\Program Files\Python39\lib\pickle.py", line 997, in _batch_setitems
save(v)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "C:\Program Files\Python39\lib\pickle.py", line 717, in save_reduce
save(state)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "C:\Program Files\Python39\lib\pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "C:\Program Files\Python39\lib\pickle.py", line 997, in _batch_setitems
save(v)
File "C:\Users\Doug\AppData\Local\pypoetry\Cache\virtualenvs\ultistacker-ai-wqxucbEU-py3.9\lib\site-packages\dill_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "C:\Program Files\Python39\lib\pickle.py", line 578, in save
rv = reduce(self.proto)
TypeError: cannot pickle '_pylibCZIrw.czi_reader' object
python-BaseException

Thank you for your help with this!
-Doug

[mmckerns]

so it would appear that dill does use pickle in some situations.

yes, that's true. What I said was multiprocessing uses pickle not dill, so the OP's traceback with multiprocessing in it is unexpected unless multiprocess was not installed.

@dougwood: so this seems like a different issue than the OP, so probably belongs in it's own issue.
Again, there's limited ability to debug without a minimal example that demonstrates the same error.
However, it looks like you are running into an object that dill nor pickle knows how to serialize. You can try a few things: (1) try the dill serialization variant dill.settings['recurse'] = True, (2) try to diagnose why it fails, or how it gets to the failure point, but using the pickle debugger dill.detect.trace(True). You'd want to post the results of the trace so we can help interpreting it. Think about opening a new ticket.