Python runtime: recompiling and reusing C library

Question

I am developing a tool for some numerical analysis of user-defined functions. The idea is to make a convenient UI in Python, where user can enter C function, then press a button - and receive some output data. Computations can take minutes or hours, so Numpy-only performance is not acceptable.

I have tried the following approach: the Python-based UI calls gcc, compiles dll from user functions that is than used by my core C-based algorithms in Cython wrappings. It works, but since there is no way to fully unload the python module, I can not recompile user-defined function until the whole UI program is closed and run again.

The only way I see now is to separate the computational core and UI processes and then make them interact via shared memory/messaging. As user wants to update his function, the program terminates the core, recompiles dll and starts the core again.

Can you suggest any common practice in such cases?

Thank you!

Answer 1

Python has really good multiprocessing support (and really not very good threading support), so you could spawn a new python process for each expression to be evaluated, compile and load the dll in the new process, then have it communicate the results back to the parent process. When the spawned process exits, everything should be unloaded.