Why does killing this subprocess raise a ProcessLookupError?

Question

I can't understand when I need to kill subprocess.

for package in server.packages:
    n = subprocess.Popen(['which', package], stdout=subprocess.DEVNULL)
    n.wait()
    if n.returncode != 0:
        n.kill()
        <some other code>

I'm getting error (using Python3):

ProcessLookupError: [Errno 3] No such process

Can anybody explain me when subprocess kill himself and when I need to do it manually ?

Answer 1

In Python, Popen.wait is a blocking call that waits for the subprocess to terminate. So there is normally no need to kill the subprocess after the call to wait returns. See Python docs on Popen.wait().

Now, if you understand how that works, you can see that your code fails because at some point Popen.returncode returns a non-zero value and you then try to kill a process that does not exist any more.

This is why a ProcessLookupError is raised.

Now, as pointed out by another answer here, it is possible the returned value will be None, which indicates a possible (perhaps OS-specific) problem with the subprocess and could be checked for. The Python docs merely state that None indicates the process is still running (a negative value is also possible; see the docs for details).

Apart from that, if you need to kill a still-running subprocess for some reason, you either have to set and catch a timeout using:

   try:
      # time out in five seconds
      n.wait(timeout=5)
   except TimeOutExpired:
      # you can use kill, terminate or send_signal method; see the docs
      n.kill()

... or, alternatively, not use wait at all: Instead, let the subprocesses run while you do something else in your code, and then kill them later:

   import os

   processes = []
   for package in server.packages:
      n = subprocess.Popen(['which', package], stdout=subprocess.DEVNULL)
      processes.append(n)

   <some other code while the subprocesses run (and possibly terminate)>

   for p in processes:
      try:
          p.kill()
      except OSError:
          # silently fail if the subprocess has exited already
          pass

What if you want to just check if the process is alive? Unfortunately, Python stdlib does not have a good, convenient way to check if a process is running. But, it can be conveniently done using psutil, a third-party library. With it, checking if a process exists is as easy as:

   pid = n.pid # where n is a subprocess.Popen object from above examples
   import psutil
   psutil.pid_exists(pid) # returns True or False