Convert list to numpy array without using much RAM

Question

I would like to convert a list with shape (1200, 140, 150, 130) to a numpy array, but the standard numpydata = np.array(mylist) uses to much memory.

Is there any less memory consuming way to do this?

Answer 1

If there's memory for the final result, but np.array internals is using too much memory, you might get around that processing the list in blocks. For example:

In [236]: res = np.zeros((10,3,4),int)                                                         
In [237]: alist = np.random.randint(0,10,(10,3,4)).tolist()                                    
In [238]: for i,row in enumerate(alist): 
     ...:     res[i] = row 
In [240]: np.allclose(res, np.array(alist))                                                    
Out[240]: True

For small arrays this iteration will be slower, but with large ones, memory management issues might out weight the iteration costs.