How can I Orchestrate Python Scripts with different virtual envs

Question

How can I use Rocketry (Python) with multiple virtual environments? I try to use Rocketry as Process Orchestrator (because it's easier than Airflow and I don't need to run on different machines).

But I don't know how to run different processes that use different virtualenvs.

(doc of rocketry: https://github.com/Miksus/rocketry)

For now, I saw that I could run some scripts before, but I think that it can be used with CLI only.

from rocketry import Rocketry
from rocketry.conds import daily, secondly

app = Rocketry()

@app.task(daily)
def do_daily(): # task from virtualenv A
    ...

@app.task(daily)
def do_secondly(): # task from virtualenv B
    ...


if __name__ == '__main__':
    app.run()

Answer 1

Author of Rocketry here. I have thought of at some point making a native virtual env support but at the moment doing that yourself is not difficult either. There are two options:

1. Use regular function task as a wrapper
2. Use a command task

Note that I have Windows now thus the location of your Python interpreter is somewhere else than I have if you use a different OS (I have it in env/Scripts/python.exe).

So basically both of these will:

1. Run the actual Rocketry app in one env/process
2. Launch a task in a separate thread (or async task)
3. Call your script using a shell command (this command calls your script with specified virtual env)

Using a regular task

import subprocess

app = Rocketry()

@app.task(daily, execution="thread")
def run_script():
    subprocess.check_call(r'"env/Scripts/python.exe" C:/myscript.py')

Read more from subprocess: https://docs.python.org/3/library/subprocess.html. Note that I used a threaded task as using process launches unnecessary extra child process. There is also an async-subprocess which you may also use and then set execution="async" if you prefer that.

Using a command task

app = Rocketry()

app.task(daily, command='"env/Scripts/python.exe" C:/myscript.py', execution="thread")

if __name__ == '__main__':
    app.run()

There is also an argument cwd (which seems to accept the path as a string at the moment) to change the current working directory.

It also seems the command task uses the default execution thus you may want to specify that as thread here as well (changing the default to be more reasonable for command tasks in the future).