"OSError: cannot write object arrays to a file in binary mode"

Question

It must be one of those days. I have always been able to use function tofile to save data. But for some reason, today it's not working :)

import numpy as np

blah.....

print(type(blist))
npdata = np.array(blist)
print(type(npdata))
npdata.tofile('myfile.dat')

Gets me the following results:

Traceback (most recent call last):
File "C:/context.py", line 67, in <module>
npdata.tofile('myfile.dat')
OSError: cannot write object arrays to a file in binary mode
<class 'list'>
<class 'numpy.ndarray'>

So it says I have the file open in binary mode. But I am not opening it in binary mode as far as I know.

EDIT (After Problem solved): I was assuming that blist was a list of integers when I posted this question. Instead it was a list of lists of integers. Problem was that when I created it I was getting a dtype=object instead of dtype=int32 that I was expecting.

Morale: Make sure to use np.append/np.extend correctly and always set the dtype explicitly.

Answer 1

According to tofile docs, if the sep is the default value, it writes the array in binary mode.

In [714]: x
Out[714]: array([[1, 2, 3], [1, 2]], dtype=object)

In [715]: x.tofile('test')
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-715-0ff8f3c688ad> in <module>()
----> 1 x.tofile('test')

OSError: cannot write object arrays to a file in binary mode

It opened the file in binary mode, but then found that the array is an object array, which it can't write that way. The default binary mode is for quickly writing numbers to the file, not general objects.

Specifying a sep, makes it write in text mode, which in this case works:

In [716]: x.tofile('test',sep=',')

In [717]: cat test
[1, 2, 3],[1, 2]

I have another object array (from another question) that contains a generator.

In [719]: g.tofile('test',sep=',')

In [720]: cat test
<generator object <genexpr> at 0xb266632c>

So in text mode, tofile writes the str(x) representation of the array to the file.

np.save is better at handling object arrays. It uses pickle to encode the object that it can't write as regular arrays. np.load can reload such a file.