Why does Python sometimes throw ValueErrors in this code?

Question

I have some Python 3.5 code that looks approximately like this:

try:
    my_process = Popen(someargs, stdin=None, stdout=PIPE, stderr=PIPE)
    stdout, stderr = my_process.communicate(timeout=10)
    my_process.wait()
except TimeoutExpired:
    my_process.kill()
    stdout, stderr = my_process.communicate()

I am trying to follow the principle described in the python subprocess documentation here, namely that in the case of a TimeoutError, I should kill the process manually, then complete the communication.

In principle that sounds fine, but periodically (maybe every 1 in 50 times, very approximately), I get an error like this:

Traceback (most recent call last):
  File "/Users/xyz/myprogram/myprogram", line 125, in <module>
    stdout, stderr = my_process.communicate()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/subprocess.py", line 1068, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/subprocess.py", line 1689, in _communicate
    selector.register(self.stdout, selectors.EVENT_READ)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/selectors.py", line 342, in register
    key = super().register(fileobj, events, data)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/selectors.py", line 228, in register
    key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/selectors.py", line 215, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/selectors.py", line 39, in _fileobj_to_fd
    "{!r}".format(fileobj)) from None
ValueError: Invalid file object: <_io.BufferedReader name=5>

Line 125 is the second communicate() line in my case.

It would appear that this is failing because some stream underlying the process object has closed or terminated - perhaps sometimes by chance this happens between the kill() and the communicate()?. But if so, is there a graceful way I should be handling this? The Python docs don't appear to cover this case.

Answer 1

To workaround my_process.communicate() raising ValueError in the exception handler, you could read from the streams directly (simpler code path -- don't use it in the general case):

from subprocess import Popen, PIPE, TimeoutExpired

with Popen(cmd, stdout=PIPE, stderr=PIPE) as process:
    try:
        stdout, stderr = process.communicate(timeout=10)
    except TimeoutExpired:
        process.kill()
        stdout = process.stdout.read() # the process is dead, no deadlock
        stderr = process.stderr.read()

On Python 3.5, you could use subprocess.run():

import subprocess
from subprocess import PIPE, TimeoutExpired

try:
    result = subprocess.run(cmd, timeout=10, stdout=PIPE, stderr=PIPE)
except TimeoutExpired as e:
    result = e
stdout, stderr = result.stdout, result.stderr

Though it handles TimeoutExpired the same way as your code and therefore you can get ValueError anyway. If you get ValueError with this code; report the issue at http://bugs.python.org