How to properly redirect Python script output to file?

Question

I am trying to run a script remotely on a server and I intend to use something along the following lines: nohup ./script.py > runtime.out 2> runtime.err & and monitor the script's progress with tail -f runtiime.out. The problem I am having is that the redirect doesn't seem to work as expected. For the purposes of my problem my problem can be reproduced as described below:

script.py:

#!/usr/bin/env python3

import time
if __name__=='__main__':
    for i in range(1000):
        print("hi")
        time.sleep(1)

Then in shell run ./print.py > a.out &. This will give the PID of the proccess and will exit as expected. However a.out is empty despite the program running. Also if i do ./print.py > a.out without the '&' the a.out file remains empty until I Ctrl-C the command. Then it displays all expected output until the termination of the script.

I thought the ">" redirected continuously the stdout and stderr and not only at command completion.

Answer 1

The simplest way to do that is just by using -u flag of the python command. It should look like that:

nohup python3 -u script.py > runtime.out 2> runtime.err &

According to the python3 --help:

-u : force the stdout and stderr streams to be unbuffered; this option has no effect on stdin; also PYTHONUNBUFFERED=x

Answer 2

Using print("hi", flush=True) will keep forcing the stream to flush contents, so it will continuously update the output file. I don't have enough information about your program to suggest alternatives, but I would look for a better method if possible.