Python : communication with c++ command line program not working when using <cstdio>

Question

I have the following python code, which is supposed to provide the intial input to a C++ program, then take its output and feed it back into it, until the program finishes execution:

comm.py

p = subprocess.Popen('test__1.exe', bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False)
p.stdin.flush()
p.stdout.flush()
x = b'1\n'
while True:
    p.stdin.write(x)
    p.stdin.flush()
    p.stdout.flush()
    x = p.stdout.readline()
    print(x)
    if p.poll() != None:
        break

I am currently testing it with two simple C++ programs:

test__1.cpp:

#include <iostream>
using namespace std; 
int main()
{
    for( int i = 0; i < 3; ++i )
    {
        int n;
        cin >> n;
        cout << n+1 << endl; 
    }
    return 0;
}

test__2.cpp

#include <cstdio>
int main()
{
    for( int i = 0; i < 3; ++i )
    {
        int n;
        scanf("%d", &n);
        printf("%d\n", n+1);
    }
    return 0;
}

When comm.py opens test__1.exe everything works fine, but when it opens test__2.exe it hangs on the first call to readline(). Note that this problem does not occur when I feed test__2.exe the whole input before execution (i.e. initially set x = '1\n2\n3\n')

What could be causing this issue?

(Also, comm.py should be able to handle any valid C++ program, so only using iostream would not be an acceptable solution.)

EDIT: I also need the solution to work on Windows.

Answer 1

It is caused by the fact that std::endl flushes the ostream and printf does not flush stdout, as you can see by amending test__2.cpp as follows:

#include <cstdio>
int main()
{
    for( int i = 0; i < 3; ++i )
    {
        int n;
        scanf("%d", &n);
        printf("%d\n", n+1);
        fflush(stdout);  //<-- add this
    }
    return 0;
}

You say that you want to module to work correctly with any C++ program, so you can't rely upon it flushing the standard output (or standard error) after every write.

That means you must cause the program's standard streams to be unbuffered and do so externally to the program itself. You will need to do that in comm.py.

In Linux (or other host providing GNU Core Utils), you could so by executing the program via stdbuf, e.g.

import subprocess

cmd = ['/usr/bin/stdbuf', '-i0','-o0', '-e0', './test__2']
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False)
p.stdin.flush()
p.stdout.flush()
x = b'1\n'
while True:
    p.stdin.write(x)
    x = p.stdout.readline()
    print(x)
    if p.poll() != None:
        break

which unbuffers all the standard streams. For Windows, you will need to research how do the same thing. For the time being I don't know.