architecture questions about a multithreaded approach in python

Question

I'm creating a socket server that will run forever and push files out to a remote service. The files will be big and many, so creating a threaded script that can help move things along. I have a few questions about how to approach this.

In this setup I will be using the Queue provided by python because it implements thread-safe transactions without any work on my part. I will arbitrarily set the thread count to 10 for now, but will play with it later to find the sweet number.

So which approach should I take:

Should I create 10 threads that are awake while the Queue is populated and sleep while it's empty? (Consumer producer type deal)

Or should I spawn threads until the max, 10, is reached, then sleep the main program until a thread finishes, then wake up the main thread to spawn one more thread and simply repeat until the Queue is empty.

What are the benefits/drawbacks of either method as well? Thanks.

Answer 1

Continually creating and terminating threads is expensive, error-prone and best avoided if at all possible on both performance and reliablilty grounds. Micro-managing threads from the 'main program' is just, well, just a really bad idea when you can just, well, not do it. Modern languages/environments/libraries are providing thread-pools to try and stop developers doing horrible things like continually creating and destroying threads, waiting for threads to finish with CPU/sleep loops, waiting for threads to finish with DoEvents loops in event-handlers, storing threads in lists and fiddling with them in an attempt to manage them, explicitly terminating threads when it is not necessary and many other nasty practices.

Go with plan 'A'. Shove you filePath, remote URI and all else needed, (eg. an error message string:), for the file-push into some object instance and push that onto a blocking producer-consumer queue that has your 10 threads hanging off it. Let the threads manage themselves.