Re-reading file when cycling over a file object in Python

Question

From this context:

import itertools
lines = itertools.cycle(open('filename'))

I'm wondering how I can implement that same 'feature' but rereading the file when it reaches the end, so if the file got changed since first iteration it's reloaded before starts another cycle. (hope I've explained well)

Thank you in advance! :)

Answer 1

I'd use:

itertools.chain.from_iterable(itertools.starmap(open, itertools.repeat(("filename",))))

or:

itertools.chain.from_iterable(itertools.starmap(lambda: open("filename"), itertools.repeat(())))

You can also write a generator comprehension (I think I like this best!):

(line for _ in itertools.repeat(()) for line in open("filename"))

Here's the imperative (statement-based) equivalent:

def cycle_file(filename):
    while True:
        for line in open(filename):
            yield line

Or, with Python 3.3 (using PEP 380 subgenerator delegation):

def cycle_file(filename):
    while True:
        yield from open(filename)

One problem with all of these is that (on a GC platform e.g. Jython) the file will not be closed until the file object is GCed, which could happen some time later. To prevent the open file leaking, you have to call close on it or use a contextmanager (with statement). This comes out naturally in the imperative form:

def cycle_file(filename):
    while True:
        with open(filename) as f:
            for line in f:
                yield line

or

def cycle_file(filename):
    while True:
        with open(filename) as f:
            yield from f

Trying to close the file with a generator comprehension becomes highly contrived:

(line for f in (itertools.chain(f, (f for f in (f,) if f.close() and False))
                for f in (open("filename") for _ in itertools.repeat(())))
 for line in f)

It'd be nice if Python had a way to specify that an opened file should close itself upon reaching the end of the file, or a way to tell a contextmanager-iterator to close itself on StopIteration.

Answer 2

Something like

def cycle_file(f):
    while True:
        ln = f.readline()
        if ln == "":
            f.seek(0)
            continue
        yield ln

Except that it might be nice to put in a check for empty files, which I'll leave to you.