Python: threads using join() in while loop

Question

I would like my while loop to block at most 5 seconds for all threads it creates in the for loop. However, the following code will block by the threads one by one. How can I approach my goal? Thanks.

threads = []
while True:
    for 3:
        newThread = threading.Thread(..)
        threads.append(newThread)
        newThread.start()
        newThread.join(5)

Answer 1

You need to use condition variable (threading.Condition in Python). It allows to wait for a predicate to become true. In your case the predicate is all threads have finished work or time out exceeded. Here is code which creates ten threads and waits until they are finished with 5sec time out. Verbose logs will help you:

import threading
import time
import logging


logging.basicConfig(
    format='%(threadName)s:%(message)s',
    level=logging.DEBUG,
)


NUM_OF_THREADS = 10
TIMEOUT = 5


def sleeping_thread(delay, cond):
    logging.debug("Hi, I'm going to delay by %d sec." % delay)
    time.sleep(delay)
    logging.debug("I was sleeping for %d sec." % delay)
    cond.acquire()
    logging.debug("Calling notify().")
    cond.notify()
    cond.release()


def create_sleeping_thread(delay, cond):
    return threading.Thread(target=sleeping_thread,
                            args=(delay, cond))


if __name__ == '__main__':
    cond = threading.Condition(threading.Lock())
    cond.acquire()

    working_counter = NUM_OF_THREADS
    for i in xrange(NUM_OF_THREADS):
        t = create_sleeping_thread(i, cond)
        t.start()

    start_time = time.time()
    while working_counter > 0 and (time.time() - start_time < TIMEOUT):
        cond.wait()
        working_counter -= 1
        logging.debug('%d workers still working', working_counter)
    cond.release()
    logging.debug('Finish waiting for threads (%d workers still working)',
                 working_counter)

Further information at comp.programming.threads FAQ.