Unable to load a previously dumped pickle file in Python

Question

The implemented algorithm which I use is quite heavy and has three parts. Thus, I used pickle to dump everything in between various stages in order to do testing on each stage separately.

Although the first dump always works fine, the second one behaves as if it is size dependent. It will work for a smaller dataset but not for a somewhat larger one. (The same actually also happens with a heatmap I try to create but that's a different question) The dumped file is about 10MB so it's nothing really large.

The dump which creates the problem contains a whole class which in turn contains methods, dictionaries, lists and variables.

I actually tried dumping both from inside and outside the class but both failed. The code I'm using looks like this:

data = pickle.load(open("./data/tmp/data.pck", 'rb')) #Reads from the previous stage dump and works fine.
dataEvol = data.evol_detect(prevTimeslots, xLablNum) #Export the class to dataEvol
dataEvolPck = open("./data/tmp/dataEvol.pck", "wb") #open works fine
pickle.dump(dataEvol, dataEvolPck, protocol = 2) #dump works fine
dataEvolPck.close()

and even tried this:

dataPck = open("./data/tmp/dataFull.pck", "wb")
pickle.dump(self, dataPck, protocol=2) #self here is the dataEvol in the previous part of code
dataPck.close()

The problem appears when i try to load the class using this part:

dataEvol = pickle.load(open("./data/tmp/dataEvol.pck", 'rb'))

The error in hand is:

Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
dataEvol = pickle.load(open("./data/tmp/dataEvol.pck", 'rb'))
ValueError: itemsize cannot be zero

Any ideas? I'm using Python 3.3 on a 64-bit Win-7 computer. Please forgive me if I'm missing anything essential as this is my first question.

---

Answer:

The problem was an empty numpy string in one of the dictionaries. Thanks Janne!!!

Answer 1

It is a NumPy bug that has been fixed recently in this pull request. To reproduce it, try:

import cPickle
import numpy as np
cPickle.loads(cPickle.dumps(np.string_('')))