With PyO3 how can I import a Python module once and then use it in multiple Rust threads?

Question

Using PyO3, from a long-running Rust application I want to be able to call some functions many times. Before that I need to import a few modules. How can I do the imports just once and then re-use the PyModules from other threads that call the functions? Here is a simplified example of what I'm trying to do:

let x: PyResult<&PyModule> = Python::with_gil(|py| {
    let pm = py.import("time")?;  // Only do this once
    Ok(pm)
});
let my_module = x.unwrap();
let mut handles = vec![];
for i in 0..2 {
    let t = thread::spawn(|| {
        Python::with_gil(|py| {
            // Use my_module here
        });
    });
    handles.push(t);
}
for h in handles {
    h.join();
}

The above code will not compile due to lifetime errors. How should I change it, so that I can use that shared my_module instance across my app, in different threads?

Answer 1

You need three things for this to work

• The return value of import is an &'py PyModule. The 'py means you cant take that out of the with_gil scope. (The for<'py> here enforces that. Roughly, it says that 'py might be　anything, for example as short as the call to with_gil.) You must convert it into a Py<PyModule>, which you're free to do with what you want.
• If you're planning on passing a reference to a thread (that is, a &Py<PyModule>), rust must know that the thread can't outlive the reference. Crucially, it must know that from the type/lifetime system, which isn't affected by h.join(). The solution is scoped threads. (You could also solve this by cloneing the Py in the loop before you spawn the thread.)
• Finally, you can remove that pesky Py wrapper with .as_ref(py) as long as you have another GIL.

use pyo3::prelude::*;

fn main() {
    let x: PyResult<Py<PyModule>> = Python::with_gil(|py| {
        let pm = py.import("time")?; // Only do this once
        Ok(pm.into())
    });
    let my_module = x.unwrap();
    std::thread::scope(|s| {
        for _ in 0..2 {
            s.spawn(|| {
                Python::with_gil(|py| {
                    my_module.as_ref(py);
                });
            });
        }
    });
}

I do hope you're doing something other than acquiring the GIL in your threads, otherwise spawning threads would be rather pointless.