Getting live output from asyncio subprocess

Question

I'm trying to use Python asyncio subprocesses to start an interactive SSH session and automatically input the password. The actual use case doesn't matter but it helps illustrate my problem. This is my code:

    proc = await asyncio.create_subprocess_exec(
        'ssh', '[email protected]',
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.STDOUT,
        stdin=asyncio.subprocess.PIPE,
    )

    # This loop could be replaced by async for, I imagine
    while True:
        buf = await proc.stdout.read()
        if not buf:
            break
        print(f'stdout: { buf }')

I expected it to work something like asyncio streams, where I can create two tasks/subroutines/futures, one to listen to the StreamReader (in this case given by proc.stdout), the other to write to StreamWriter (proc.stdin).

However, it doesn't work as expected. The first few lines of output from the ssh command are printed directly to the terminal, until it gets to the password prompt (or host key prompt, as the case may be) and waits for manual input. I expected to be able to read the first few lines, check whether it was asking for password or the host prompt, and write to the StreamReader accordingly.

The only time it runs the line print(f'stdout: { buf }') is after I press enter, when it prints, obviously, that "stderr: b'Host key verification failed.\r\n'".

I also tried the recommended proc.communicate(), which isn't as neat as using StreamReader/Writer, but it has the same problem: Execution freezes while it waits for manual input.

How is this actually supposed to work? If it's not how I imagined, why not, and is there any way to achieve this without resorting to some sort of busy loop in a thread?

PS: I'm explaining using ssh just for clarity. I ended up using plink for what I wanted, but I want to understand how to do this with python to run arbitrary commands.

Answer 1

If anyone else landed here for a more generic answer to the question, see the following example:

import asyncio

async def _read_stream(stream, cb):  
    while True:
        line = await stream.readline()
        if line:
            cb(line)
        else:
            break

async def _stream_subprocess(cmd, stdout_cb, stderr_cb):  
    process = await asyncio.create_subprocess_exec(*cmd,
            stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)

    await asyncio.gather(
        _read_stream(process.stdout, stdout_cb),
        _read_stream(process.stderr, stderr_cb)
    )
    return await process.wait()


def execute(cmd, stdout_cb, stderr_cb):  
    loop = asyncio.get_event_loop()
    rc = loop.run_until_complete(
        _stream_subprocess(
            cmd,
            stdout_cb,
            stderr_cb,
    ))
    loop.close()
    return rc

if __name__ == '__main__':  
    print(execute(
        ["bash", "-c", "echo stdout && sleep 1 && echo stderr 1>&2 && sleep 1 && echo done"],
        lambda x: print("STDOUT: %s" % x),
        lambda x: print("STDERR: %s" % x),
    ))