TypeError: can't pickle _abc_data objects

[AlJohri]

On Python 3.7, I'm getting this error:

Traceback (most recent call last):
  File "../scripts/make_preprocessor.py", line 101, in <module>
    pickle.dump(preprocessor, f)
  File "/home/ubuntu/repo/.venv/lib/python3.7/site-packages/dill/_dill.py", line 287, in dump
    pik.dump(obj)
  File "/usr/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/repo/.venv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 816, in save_list
    self._batch_appends(obj)
  File "/usr/lib/python3.7/pickle.py", line 840, in _batch_appends
    save(x)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.7/pickle.py", line 633, in save_reduce
    save(cls)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/repo/.venv/lib/python3.7/site-packages/dill/_dill.py", line 1323, in save_type
    obj.__bases__, _dict), obj=obj)
  File "/usr/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.7/pickle.py", line 786, in save_tuple
    save(element)
  File "/usr/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/ubuntu/repo/.venv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _abc_data objects

I believe it may be related to this issue: cloudpipe/cloudpickle#180

[AlJohri]

@mmckerns here is an minimal, reproducible example:

import dill as pickle

from abc import ABCMeta

class TestClass(metaclass=ABCMeta):
    pass

with open('test.pickle', 'wb') as f:
    pickle.dump(TestClass, f)

[AlJohri]

And more real world example...

from sklearn.compose import ColumnTransformer

class DataFrameTransformer(ColumnTransformer):

    def get_feature_names(self):
        """
        enable get_feature_names when using remainder passthrough
        """
        feature_names = []
        for name, transformer, columns in self.transformers_:
            if hasattr(transformer, "get_feature_names"):
                feature_names += [f"{name}_{x}" for x in transformer.get_feature_names()]
            elif transformer == "drop":
                continue
            elif name == "remainder" and transformer == "passthrough":
                feature_names += self._df_columns.take(columns).tolist()
        return feature_names

with open('test.pickle', 'wb') as f:
    pickle.dump(DataFrameTransformer, f)

[Zahlii]

I think this is a blocking issue. Cloudpickle handles it fine. I'm not able to migrate production code using dill to the newer python versions since I'm using abstract classes, hence I will need to consider switching to cloudpickle.

[LarsHill]

I still have the exact same problem. Python 3.8.5, multiprocess 0.70.11.1, dill 0.3.3.
Is there any planned fix? It is also blocking in our situation.

Update: The above minimal example runs through with Python 3.6 but throws the above error in Python 3.7, 3.8 and 3.9.

[iirekm]

+1

and it's even not easy to debug which object is the cause of the problem

[iirekm]

silly workaround (but worked for me): remove all ABCs from every class extending from ABC

[emfdavid]

Any thoughts on how to fix this... or where to start?
I would be willing to contribute but need some pointers to get started.

[mmckerns]

I did not realize this was still an issue... it's basically, the _abc_impl attribute which is an _abc_data object, which is coded in C, that's causing the issue. There's a method, _abc._get_dump that provides the serialization for the object. I'm not sure how the load works yet... I'll look into it.

[emfdavid]

Would it be something like registering those methods you reference similar to the save_XXX methods in dill/_dill.py?
The dill code is a bit hard to jump into... but I would be glad to help get this done if it isn't more trouble for you than I am worth.

[MKlubhouse]

+1 long running convo it seems.

[emfdavid]

I will take a crack at this this week.
Anybody interested in jumping in with me?

[emfdavid]

@mmckerns I was able to hack in a solution that solves my broken tests...
emfdavid@770c5a0
But I am still a bit lost. I looked at what cloudpickle does, which also works with /clsdict/_dict/ in _dill, but once I realized that the built in pickle seems to handle ABC objects, it seemed simpler to kick it back to StockPickler and more inline with your approach elsewhere.

Anyway, not sure I have this in the right place... but it does fix my test cases. Further input would be welcome!