Python subprocess get stuck at communicate() call

Question

Context:

I am using python 2.7.5.

I need to run a subprocess from a python script, wait for its termination and get the output.

The subprocess is run around 1000 times.

In order to run my subprocess, I have defined a function:

def run(cmd):
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    (stdout, stderr) = p.communicate()
    return (p.returncode, stdout, stderr)

The subprocess to be executed is a bash script and is passed as the cmd parameter of the run() function.
The command and its arguments are given through a list (as expected by Popen()).

Issue:

In the past, it has always worked without any error.
But recently, the python script get always stuck on a subprocess call after having successfully executed a lot of calls. The subprocess in question is not executed at all (the bash script is not even started) and the python script blocks.

After stopping the execution with Ctrl+C, I get the point where it was stuck:

  [...]

  File "import_debug.py", line 20, in run
    (stdout, stderr) = p.communicate()
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt

I don't understand why I have this issue nor how to solve it.

I have found this SO thread that seems to tackle the same issue or something equivalent (since the output after the keyboard interruption is the same) but there is no answer.

Question: What is happening here ? What am I missing ? How to solve this issue ?

---

EDIT:

The call is under the form:

(code, out, err) = run(["/path/to/bash_script.sh", "arg1", "arg2", "arg3"])

print out
if code:
    print "Failed: " + str(err)

The bash script is doing some basic processing with the data (unzip archives and do something with the extracted data).
When the error occurs, none of the bash script instructions are executed.

_{I cannot provide the exact command, arguments and contents for company privacy concerns.}

Answer 1

The author of the original thread you're referring to says: "If I set stderr=None instead of stderr=subprocess.PIPE I never see this issue." -- I'd recommend to do exactly that and get your script working.

---

Added after reading the comment section:

There are a few useful options, you may want or not to use:

-f  freshen existing files, create none
-n  never overwrite existing files         
-o  overwrite files WITHOUT prompting