Threaded scripts stall after ending without closing

Question

Hopefully this is just something small im doing wrong as these are some of my first threaded scripts using queues. Basically after running through it stops and sits there but wont exit.

import threading
import Queue
class Words(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        self.queue = Queue.Queue()     

    def word(self):
        read = open('words.txt')
        for word in read:
            word = word.replace("\n","")
            self.queue.put(word)       
        read.close() 
        for i in range(5):
            t = self.run()
            t.setDaemon(True)
            t.start()  
        self.queue.join()

    def run(self): 
        while True:
            word = self.queue.get()
            print word 
            self.queue.task_done()

    if __name__ == '__main__':
        Word =  Words()
        Word.word()

Answer 1

You are using threads incorrectly in a couple of ways in your code:

First, the code seems to be built on the incorrect assumption that the one Thread subclass object you have can spawn all of the threads you need to do the work. On the contrary, the Thread documentation says that start "must be called at most once per Thread object". In the case of the word method, this is the self reference.

However, it would not be useful to call self.start() because that would spawn a single thread to consume the queue, and you would gain nothing from threading. Since word would have to construct new instances of Words anyway to initiate multiple threads, and the queue object will need to be accessed by multiple Words instances, it would be useful to have both of those separate from the Words object. For example, word could be a function outside of the Words object that starts like:

def word():
    queue = Queue.Queue()
    read = open('words.txt')
    for word in read:
        word = word.replace("\n","")
        self.put(word)       
    read.close()
    #...

This would also mean that Words would have to take the queue object as a parameter so that multiple instances would share the same queue:

class Words(threading.Thread):
    def __init__(self, queue):
        threading.Thread.__init__(self)
        self.queue = queue

Second, your thread function (run) is an infinite loop, so the thread will never terminate. Since you are only running the queue consumer threads after you have added all items to the queue, you should not have a problem terminating the thread once the queue is empty, like so:

def run(self): 
    while True:
        try:
            word = self.queue.get(False)
        except Queue.Empty:
            break
        print word 
        self.queue.task_done()

It is useful to use exceptions here because otherwise the queue could empty out and then the thread could try to get from it and it would end up waiting forever for an item to be added.

Third, in your for loop you call self.run(), which passes control to the run method, which then processes the entire queue and returns None after the method is changed to terminate. The following lines would throw exceptions because t would be assigned the value None. Since you want to spawn other threads to do the work, you should do t = Word(queue) to get a new word thread and then t.start() to start. So, the code when put together should be

class Words(threading.Thread):
    def __init__(self, queue):
        threading.Thread.__init__(self)
        self.queue = queue

    def run(self): 
        while True:
            try:
                word = self.queue.get(False)
            except Queue.Empty:
                break
            print word 
            self.queue.task_done()

def word():
    queue = Queue.Queue()
    read = open('words.txt')
    for word in read:
        word = word.replace("\n","")
        self.put(word)       
    read.close()
    for i in range(5):
        t = Word()
        t.setDaemon(True)
        t.start()
    queue.join()

if __name__=='__main__':
    word()