Piping commands into a Python interpreter

Question

I want to control a Python interpreter by sending commands to it one by one via a pipe, e. g. make the Python interpreter print "hello", wait 3 secs, then make the Python interpreter print "world".

Why does the following code in a bash:

{ echo 'from sys import stdout' ; echo 'stdout.write("hello\n")' ; echo 'stdout.flush()' ; sleep 3 ; echo 'stdout.write("world\n")' ; echo 'stdout.flush()' ; } | python3.5 -u

first waits 3 secs and then prints:

hello
world

whereas the following code:

{ echo 'from sys import stdout' ; echo 'stdout.write("hello\n")' ; echo 'stdout.flush()' ; sleep 3 ; echo 'stdout.write("world\n")' ; echo 'stdout.flush()' ; } | while read -r l ; do echo $l ; done

first prints:

from sys import stdout
stdout.write("hello\n")
stdout.flush()

then waits 3 secs and finally prints:

stdout.write("world\n")
stdout.flush()

?

In the end, I want to control a second Python interpreter from a first Python script like in the example above (print "hello", wait 3 secs, print "world"). Is this possible with the "subprocess" Python module?

The following code:

import logging
from select import select
from subprocess import Popen, PIPE, TimeoutExpired
from time import sleep
from threading import Thread

logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s %(message)s")
logger = logging.getLogger()

def read_stdout(proc):
    stdout_closed, stderr_closed = False, False

    while True:
        rlist, wlist, xlist = select([proc.stdout, proc.stderr], [], [], 1.2)
        if len(rlist) == 0:
            logger.debug("timeout")
            continue
        if proc.stdout in rlist:
            new_data = proc.stdout.read(1024)
            logger.debug("new_data on stdout: %r" % new_data)
            if len(new_data) == 0:
                stdout_closed = True
        if proc.stderr in rlist:
            new_data = proc.stderr.read(1024)
            logger.debug("new_data on stderr: %r" % new_data)
            if len(new_data) == 0:
                stderr_closed = True
        if stdout_closed and stderr_closed:
            break

logger.debug("start")
proc = Popen(
    #["bash", "-c", 'while read -r l ; do echo $l ; done'],
    ["python", "-u"],
    stdin=PIPE,
    stdout=PIPE,
    stderr=PIPE,
    bufsize=0)
thread = Thread(target=read_stdout, args=(proc,))
thread.start()
proc.stdin.write(b'from sys import stdout\n')
proc.stdin.write(b'stdout.write("hello\\n")\n')
proc.stdin.write(b'stdout.flush()\n')
proc.stdin.flush()
sleep(3)
proc.stdin.write(b'stdout.write("world\\n")\n')
proc.stdin.write(b'stdout.flush()\n')
proc.stdin.flush()
proc.stdin.close()
thread.join()

results in the same problem as the bash code above:

When Popen() calls "python -u", then the logging output is:

2016-07-27 00:46:05,208 start
2016-07-27 00:46:06,411 timeout
2016-07-27 00:46:07,612 timeout
2016-07-27 00:46:08,213 new_data on stdout: b'hello\nworld\n'
2016-07-27 00:46:08,216 new_data on stdout: b''
2016-07-27 00:46:08,216 new_data on stderr: b''

whereas when Popen() calls "bash -c 'while read -r l ; do echo $l ; done'", then the logging output is:

2016-07-27 00:48:37,237 start
2016-07-27 00:48:37,239 new_data on stdout: b'from sys import stdout\n'
2016-07-27 00:48:37,239 new_data on stdout: b'stdout.write("hello\\n")\nstdout.flush()\n'
2016-07-27 00:48:38,440 timeout
2016-07-27 00:48:39,642 timeout
2016-07-27 00:48:40,242 new_data on stdout: b'stdout.write("world\\n")\n'
2016-07-27 00:48:40,242 new_data on stdout: b'stdout.flush()\n'
2016-07-27 00:48:40,242 new_data on stderr: b''
2016-07-27 00:48:40,242 new_data on stdout: b''
2016-07-27 00:48:40,242 new_data on stderr: b''

What is the cause of the Python interpreter not executing immediately the command which is sent to it? Is it possible to achieve this with the "subprocess" module somehow? (I assume "pexpect" will solve the problem, but I want to understand if or why not "subprocess" can solve it.)

Answer 1

I have found the following explanation for the observed behaviour in "man python3":

In non-interactive mode, the entire input is parsed before it is executed.