How is dill different from Python's pickle module?

Question

I have a large object in my Python3 code which, when tried to be pickled with the pickle module throws the following error:

TypeError: cannot serialize '_io.BufferedReader' object

However, dill.dump() and dill.load() are able to save and restore the object seamlessly.

1. What causes the trouble for the pickle module?
2. Now that dill saves and reconstructs the object without any error, is there any way to verify if the pickling and unpickling with dill went well?
3. How's it possible that pickle fails, but dill succeeds?

Answer 1

I'm the dill author.

1) Easiest thing to do is look at this file: https://github.com/uqfoundation/dill/blob/master/dill/_objects.py, it lists what pickle can serialize, and what dill can serialize.

2) you can try dill.copy and dill.check and dill.pickles to check different levels of pickling and unpickling. dill also more includes utilities for detecting and diagnosing serialization issues in dill.detect and dill.pointers.

3) dill is built on pickle, and augments it by registering new serialization functions.

4) dill includes serialization variants which enable the user to choose from different object dependency serialization strategies (in dill.settings) -- including source code extraction and object reconstitution with dill.source (and extension of the stdlib inspect module).