Python: How to read from stdin by byte chunks until EOF?

Question

I want to read from standard input chunk by chunk until EOF. For example, I could have a very large file, and I want to read in and process 1024 bytes at a time from STDIN until EOF is encountered. I've seen sys.stdin.read() which saves everything in memory at once. This isn't feasible because there might not be enough space available to store the entire file. There is also for "line in sys.stdin", but that separates the input by newline only, which is not what I'm looking for. Is there any way to accomplish this in Python?

Answer 1

The read() method of a file object accepts an optional size parameter.

If you specify size, at most size bytes are read and returned. If the end of the file has been reached, f.read() will return an empty string ('').

See the io docs and open() docs.

Pseudo code:

with open('file') as f:
    while True:
        buffer = f.read(1024) # Returns *at most* 1024 bytes, maybe less
        if buffer = '':
            break
        process_data(buffer)

Answer 2

You can read stdin (or any file) in chunks using f.read(n), where n is the integer number of bytes you want to read as an argument. It will return the empty string if there is nothing left in the file.