Calling Python code from C++ shared library

Question

I'm using boost::python in a shared library plugin to run some python code.

The shared library is loaded as a plugin in run-time by my main program using boost::dll::shared_library API. i.e. my main program does not link with the shared library plugin.

My code runs on Ubuntu 20.04.

I'm linking my shared library with Python:

my_shared_lib/CMakeLists.txt:

find_package(Python3 COMPONENTS Development NumPy REQUIRED)
target_link_libraries(${PROJECT_NAME} PRIVATE ${Python3_LIBRARIES} Python3::NumPy)

When running the following code from my shared library I get an ImportError:

#include <boost/python.hpp>
Py_Initialize();
namespace np = boost::python::numpy;
np::initialize(); //ImportError here

I get the following error:

ImportError: /home/myuser/.local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter

I verified I have /usr/lib/x86_64-linux-gnu/libpython3.8.so in my LD_LIBRARY_PATH.

When linking my main program with Python, the program OK runs w/o the ImportError

main_program/CMakeLists.txt:

find_package(Python3 COMPONENTS Development NumPy REQUIRED)
target_link_libraries(${PROJECT_NAME} PRIVATE ${Python3_LIBRARIES} Python3::NumPy)

My conclusion here is that loading the shared library plugin in run-time does not seem to load libpython3.8.so, while loading libpython3.8.so in the main program makes it available for the shared library plugin.

If I manually load libpython3.8.so in my shared library like this:

#include <boost/python.hpp>
#include <dlfcn.h>
dlopen("/usr/lib/x86_64-linux-gnu/libpython3.8.so", RTLD_LAZY | RTLD_GLOBAL);
Py_Initialize();
namespace np = boost::python::numpy;
np::initialize();

I get a different error further along the way:

Traceback (most recent call last):
File "/home/myuser/.local/lib/python3.8/site-packages/numpy/core/init.py", line 23, in
from . import multiarray
File "/home/myuser/.local/lib/python3.8/site-packages/numpy/core/multiarray.py", line 10, in
from . import overrides
File "/home/myuser/.local/lib/python3.8/site-packages/numpy/core/overrides.py", line 6, in
from numpy.core._multiarray_umath import (

ImportError: PyCapsule_Import could not import module "datetime"

How can I make my shared library plugin load libpython3.8.so automatically when it's loaded?

EDIT

Using gdb I see my initial assumption was wrong. When the main app loads my shared library plugin, it also loads libpython3.8.so.1.0. Loaded Loaded '/home/myuser/.vs/mainapp/build/plugins/libap_python_detector.so'. Symbols loaded. Loaded '/usr/local/3rdparty/hpc/5.13.0-39_AMD_EPYC/boost176/build/lib/libboost_numpy38.so.1.76.0'. Symbols loaded. Loaded '/lib/x86_64-linux-gnu/libpython3.8.so.1.0'. Symbols loaded. Loaded '/usr/local/3rdparty/hpc/5.13.0-39_AMD_EPYC/boost176/build/lib/libboost_python38.so.1.76.0'. Symbols loaded.

So what's wrong loading numpy?

Answer 1

Currently, your design of loading NumPy from a Python interpreter embedded in a shared library is not supported by NumPy. (See Troubleshooting ImportError . It does not cover your scenario.)

The error occurs because NumPy shared libraries were not linked hardly to the Python runtime library. (You can inspect that with ldd.)

But at least on Linux, you can remedy this by loading the Python runtime library manually with RTLD_GLOBAL flag like the following example. (Load the runtime library inside your main program, not inside a plugin. Of course you will want to do that only if you are loading a Numpy-dependent plugin.)

#include <dlfcn.h>
int main()
{
    ...
    // When you load a NumPy-dependent plugin, do the following just before that.
    dlopen("libpython3.8.so", RTLD_LAZY | RTLD_GLOBAL);
    ...
}

But I am not sure it will not be broken in the future.