Python multiprocessing.Process to use virtualenv

Question

I am running a program using virtualenv. But the multiprocessing.Process spawned here uses system python by default. How do I force it to use virtualenv python.

import os
from multiprocessing import Process

def function1():
    # do_something_here
    p = Process(func2(), args=(param,))
    p.start()
    return something

def func2(param):
    os.system("which python")

Here it prints "/usr/bin/python". But I need it to use virtualenv python instead.

Answer 1

With sudo venv/bin/python, you effectively activated virtualenv by using python executable in virtualenv directly.

multiprocessing.Process spawn child process with fork(), without exec(), it uses exactly the same python executable as the parent process.

You could confirm python executable in-use by:

>>> import sys
>>> print(sys.executable)
/Users/georgexsh/workspace/tmp/venv/bin/python
>>> print(sys.exec_prefix)
/Users/georgexsh/workspace/tmp/venv/bin/..

Do not use which to determine running python executable path. which, as a Bash command, searches each element of $PATH, for a directory containing an executable file named "python", as you use virtualenv's python directly, not by run its shell activate script first, $PATH not get patched with virtualenv's, as a result, shell command which python would output path of the system python executable.

In fact, at python layer, $PATH is irrelevant, patching $PATH is for the convenience at the Bash layer, to invoke python executable in the virtualenv path by just typing "python", rather than typing full path, What matters most is which python executable is invoked, but not how it is get invoked.

Answer 2

Your problem is here (copied your comment):

@georgexsh I was running it using sudo. By default if you use sudo then it will use system python. So, I have used "sudo venv/bin/python main.py" to run the program. Even though I am using venv's python here it returns "/usr/bin/python" for "os.system('which python')". I don't understand this behaviour

Basically, what you explain here is something where your virtualenv is not active.

When you activate a virtualenv (. venv/bin/activate), the activation script will change your environment so that your PYTHONPATH is correct and Python executable is searched (and found) first in the virtual env directory. This is what virtualenv does.

By just executing the Python binary from virtualenv directories, your environment is not set for the virtual environment, so any subsequent calls to Python use your default path - as virtualenv is not there to override it.

When you execute sudo, a new process/shell is created and it does not inherit your virtual environment. You might be able to use sudo -E to pass environment but it depends on your sudo. The bulletproof version that should work in every environment is to execute a shell that first activates virtualenv and then executes your script. Something like this:

sudo -- bash  -c ". /home/test/mytest/bin/activate; which python"

This executes a bash shell as root, then activates the virtual environment and finally tells you which python it uses. Just modify the above command with your virtual environment path and it might even work.

If your system is shared, just keep in mind that this is a horrible thing to allow your regular users do from security perspective. If you create a passwordless sudo for your regular users to do this, it would give them root access with little tweaking. If it is your own system and the requirement is the knowledge of root password anyway, it does not matter.