Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python callback for a multiprocess Queue or Pipe

Is there a way to create a callback that executes whenever something is sent to the main process from a child process initiated via multiprocessing? The best I can think of thus far is:

import multiprocessing as mp
import threading
import time    

class SomeProcess(mp.Process):
    def run(self):
        while True
            time.sleep(1)
            self.queue.put(time.time())

class ProcessListener(threading.Thread):
    def run(self):
        while True:
            value = self.queue.get()
            do_something(value)

if __name__ = '__main__':
    queue = mp.Queue()
    sp = SomeProcess()
    sp.queue = queue
    pl = ProcessListener()
    pl.queue = queue
    sp.start()
    pl.start()
like image 350
Jonathan Wheeler Avatar asked Oct 18 '25 01:10

Jonathan Wheeler


1 Answers

No there is no other clean way to do so than the one you already posted.

This is how concurrent.fututes.ProcessPoolExecutor and multiprocessing.Pool are actually implemented. They have a dedicated thread which drains the tasks/results queue and run any associated callback.

If you want to save some resource, you can use a SimpleQueue in this case.

like image 131
noxdafox Avatar answered Oct 20 '25 16:10

noxdafox



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!