Can a python Multiprocessing queue be passed to the child process?

Question

I have a big dataset in a data acquisition system I wrote in python that takes infinitely long to pass over a queue from the child process to the parent. I want to save the data acquired at the end of the acquisition and tried this using the queue function in Multiprocessing. Instead of doing it this way I would prefer it if I could instead pass a message over the queue from the parent to the child to save my data before I kill the child process. Is this possible? An example of what I thought it might look like is:

def acquireData(self, var1, queue):
    import h5py
    # Put my acquisition code here
    queue.get()
    if queue == True:
        f = h5py.File("FileName","w")
        f.create_dataset('Data',data=data)
        f.close()

if __name__ == '__main__': 
    from multiprocessing import Process, Queue
    queue = Queue()
    inter_thread = Process(target=acquireData, args=(var1,queue))
    queue.put(False)
    inter_thread.start()
    while True:
        if not args.automate:
        # Let c++ threads run for given amount of time
            # Wait for stop from OP GUI
        else:
            queue.put(True)
            break
    print("Acquisition finished, cleaning up...")
    sleep(2)
    inter_thread.terminate()

Is this allowed? If this type of interfacing between processes is allowed then do I have the right notation? For some reference I have on the order of 9e7 data points in the array I'm trying to save and I have 7 arrays which is simply not being passed to my parent process in a timely manner by putting these arrays into the queue. Thank you.

Answer 1

First, yes, passing a queue to a child is not only legal, but the main use case for queues. See the first example in the docs, which does exactly that.

However, you've got some problems with your code:

queue.get()
if queue == True:

First, your queue is never going to be the boolean value True, it's going to be a Queue. You almost never want to check if x == True: in Python; you want to check if x:. For example, if [1, 2]: will pass, while if [1, 2] == True: will not.

Second, your queue isn't even the thing you want to check in the first place. It isn't truthy or falsey (or it isn't relevant whether it is); it's the value the main process put on the queue and you pulled off that's either truthy or falsey. Which you discarded as soon as you retrieved it.

So, do this:

flag = queue.get()
if flag:

Or, more simply:

if queue.get():

---

I'm not sure whether this is exactly what you want or not. That queue.get() will block forever until the main process puts something there. Is that what you wanted? If so, great; you're done with this part of your code. If not, you need to think about what you wanted instead.

As designed, the parent will always wait 2 seconds, even if the child finished long before that. A better solution is to join the child with a timeout of 2 seconds. Then you can terminate it if times out.

Plus, are you sure the termination behavior you've designed is what you want? You're doing a "soft kill request" with the queue, then waiting 2 seconds, then doing a "medium-hard kill request" with terminate, and never doing a "hard kill" with kill. That could be a perfectly reasonable design—but if it's not your design, you've implemented the wrong thing.