Celery task executing code from another programming language

Question

I'm trying to call compiled/interpreted code from a Celery task. The code is written in something other than Python.

I want to know if there is a better solution to the problem than the ones I'm thinking of.

Solution 1. Start another process and execute/interpret the piece of code I'm interested in. This has the overhead of creating and killing a process. For a very small task, that overhead may be too high.

Solution 2. Use a Listener process that can execute code from a target language. It could listen on a local socket for function signatures (aka add(2,2), execute and return the result on the same socket. The listener could also implement something like a process/thread pool to handle multiple tasks efficiently.

Solution 3 (thanks to AndrewS). Building a worker process (connected to the broker). It implies rewriting the Celery worker into the target language. This is the most expensive version of the three in terms of development effort.

Answer 1

I've found the solution.

It's also a variation on Solution 2, use Thrift for RPC with the actual job code. The code is written in the target language and a Thrift IDL describes it to the Thrift compiler which can generate both client and server. The client is obviously Python code and the server is in the target language.

Any similar alternative to Thrift will do, like other RPC code generators.

Thanks for all the answers, I hope this ends up helping someone someday.