How to save tf.data.Dataset object?

Question

As above. I tried pickling it, but I got this error:

maybe_arr = self._cpu_nograd()._numpy() # pylint: disable=protected-access

RuntimeError: Tensorflow type 21 not convertible to numpy dtype.

Answer 1

tf.data.Dataset is a more abstract object whose job is to define the data pipeline. If you want to save intermediate results to speed up your data pipeline you can use tf.data.Dataset.cache() or tf.data.Dataset.prefetch() (more on it here)

If you are interested in saving the sequence of operations in your data pipeline, I assume there is no such thing and you need to keep the code for data pipeline. I am not aware of any method that can extract the graph of data pipeline of Dataset API. If anyone is aware of that please add to the answer.