Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

read a very very big file with python

Tags:

python

file

text

What is the best solution to process each line of a text file whose size is about 500 MB?

The proposal to which I had thought :

def files(mon_fichier):
    while True:
        data = mon_fichier.read(1024)
        if not data:
            break
        yield data

fichier = open('tonfichier.txt', 'r')
for bloc in files(fichier):
    print bloc

Thank you in advance

like image 577
user653861 Avatar asked Mar 16 '26 20:03

user653861


2 Answers

with open('myfile.txt') as inf:
    for line in inf:
        # do something
        pass
like image 58
Hugh Bothwell Avatar answered Mar 19 '26 10:03

Hugh Bothwell


Just using the standard file operations should work as long as you keep away from readlines and instead just use readline.

like image 26
filmor Avatar answered Mar 19 '26 10:03

filmor



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!