How do I send large amounts of data from a forked process?

Question

I have a ctypes wrapper for a library. Unfortunately, this library is not 100% reliable (occasional segfaults, etc.). Because of how it's used, I want the wrapper to be reasonably resilient to the library crashing.

The best way to do this seems to be forking a process and sending the results back from the child. I'd like to do something along these lines:

r, w = os.pipe()
pid = os.fork()

if pid == 0:
    # child
    result = ctypes_fn()
    os.write(w, pickle.dumps(result))
    os.close(w)
else:
    # parent
    os.waitpid(pid, 0)
    result = os.read(r, 524288) # can be this big
    os.close(r)

    return pickle.loads(result)

This doesn't quite work, though. The forked process hangs on the write. Am I trying to send too much at once? Is there a simpler solution to this problem?

Answer 1

Probably you are trying to write more data than can fit into the pipe, so it is blocking until someone comes along and reads some of that info out of there. That will never happen, because the only reader is the parent process, which you appear to have written to wait until the child terminates before it reads anything. This is what we call a deadlock.

You might consider taking out that os.waitpid call and see what happens. Another option would be to see if os.pipe has any methods that give it a bigger buffer (I don't know your environment enough to say).