A RAM error of big array

Question

I need to get the numbers of one line randomly, and put each line in other array, then get the numbers of one col.

I have a big file, more than 400M. In that file, there are 13496*13496 number, means 13496 rows and 13496 cols. I want to read them to a array. This is my code:

_L1 = [[0 for col in range(13496)] for row in range(13496)]
_L1file = open('distanceCMD.function.txt')
while (i<13496):
    print "i="+str(i)
    _strlf = _L1file.readline()
    _strlf = _strlf.split('\t')
    _strlf = _strlf[:-1]
    _L1[i] = _strlf
    i += 1
_L1file.close()

And this is my error message:

MemoryError:
File "D:\research\space-function\ART3.py", line 30, in <module>
  _strlf = _strlf.split('\t')

Answer 1

you might want to approach your problem in another way. Process the file line by line. I don't see a need to store the whole big file into array. Otherwise, you might want to tell us what you are actually trying to do.

for line in open("400MB_file"):
     # do something with line.

Or

f=open("file")
for linenum,line in enumerate(f):
    if linenum+1 in [2,3,10]:
         print "there are ", len(line.split())," columns" #assuming you want to split on spaces
         print "100th column value is: ", line.split()[99]
    if linenum+1>10:
         break # break if you want to stop after the 10th line
f.close()