Python--Getting Queue.Empty exception from a nonempty multiprocessing.Queue

Question

I've having the opposite problem of many Python users--my program is using too little CPU. I already got help in switching to multiprocessing to utilize all four of my work computer's cores, and I have seen real performance improvement as a result. But the improvement is somewhat unreliable. The CPU usage of my program seems to deteriorate as it continues to run--even with six processes running. After adding some debug messages, I discovered this was because some of the processes I was spawning (which are all supposed to run until completion) were dying prematurely. The main body of the method the processes run is a while True loop, and the only way out is this block:

try:
    f = filequeue.get(False)
except Empty:
    print "Done"
    return

filequeue is populated before the creation of the subprocesses, so it definitely isn't actually empty. All the processes should exit at roughly the same time once it actually is empty. I tried adding a nonzero timeout (0.05) parameter to the Queue.get call, but this didn't fix the problem. Why could I be getting a Queue.empty exception from a nonempty Queue?

Answer 1

I suggest using filequeue.get(True) instead of filequeue.get(False). This will cause the queue to block until there are more elements.

It will, however, block forever after the final element has been processed.

To work around this, the main process could add a special "sentinel" object at the end of each queue. The workers would terminate upon seeing this special object (instead of relying on the emptiness of the queue).