Gracefully exit using ThreadPoolExecutor

Question

When using a ThreadPoolExecutor how do you gracefully exit on a signal interrupt? I would like to intercept a SIGINT and gracefully exit the process. I would like the currently running threads to finish, but no more to start, and to cancel all pending tasks.

Answer 1

In python 3.9 ThreadPoolExecutor#shutdown takes a cancel_futures argument that handles canceling the waiting tasks. In python 3.8 and below, though, this must be handled manually. Luckily we can use the code to do this in python 3.9 ourselves.

import sys
import queue
from concurrent.futures import ThreadPoolExecutor

def exit_threads( executor ):
    print( '\nWrapping up, please wait...' )

    py_version = sys.version_info
    if ( py_version.major == 3 ) and ( py_version.minor < 9 ):
        # py versions less than 3.9
        # Executor#shutdown does not accept
        # cancel_futures keyword
        # manually shutdown
        # code taken from https://github.com/python/cpython/blob/3.9/Lib/concurrent/futures/thread.py#L210
        
        # prevent new tasks from being submitted
        executor.shutdown( wait = False )
        while True:
            # cancel all waiting tasks
            try:
                work_item = executor._work_queue.get_nowait()
                                
            except queue.Empty:
                break
                                
            if work_item is not None:
                work_item.future.cancel()

     else:
           executor.shutdown( cancel_futures = True )
                    
     sys.exit( 0 )


executor = ThreadPoolExecutor( max_workers = 5 )            
signal.signal( 
    signal.SIGINT, 
    lambda sig, frame: exit_threads( executor ) 
)

# run desired code here...