IOCP loop termination may cause memory leaks? How to close IOCP loop gracefully

Question

I have the classic IOCP callback that dequeues i/o pending requests, process them, and deallocate them, in this way:

struct MyIoRequest { OVERLAPPED o; /* ... other params ... */ };
bool is_iocp_active = true;

DWORD WINAPI WorkerProc(LPVOID lpParam)
{
    ULONG_PTR dwKey;
    DWORD dwTrans;
    LPOVERLAPPED io_req;
    while(is_iocp_active)
    {
        GetQueuedCompletionStatus((HANDLE)lpParam, &dwTrans, &dwKey, (LPOVERLAPPED*)&io_req, WSA_INFINITE); 
       // NOTE, i could use GetQueuedCompletionStatusEx() here ^ and set it in the 
       // alertable state TRUE, so i can wake up the thread with an ACP request from another thread!

        printf("dequeued an i/o request\n");
        // [ process i/o request ]
        ...

        // [ destroy request ]
        destroy_request(io_req);
    }
    // [ clean up some stuff ] 
    return 0;
}

Then, in the code I will have somewhere:

MyIoRequest * io_req = allocate_request(...params...);
ReadFile(..., (OVERLAPPED*)io_req);

and this just works perfectly.

Now my question is: What about I want to immediately close the IOCP queue without causing leaks? (e.g. application must exit) I mean: if i set is_iocp_active to 'false', the next time GetQueuedCompletionStatus() will dequeue a new i/o request, that will be the last i/o request: it will return, causing thread to exit and when a thread exits all of its pending i/o requests are simply canceled by the system, according to MSDN.

But the structures of type 'MyIoRequest' that I have instanced when calling ReadFile() won't be destroyed at all: the system has canceled pending i/o request, but I have to manually destroy those structures I have created, or I will leak all pending i/o requests when I stop the loop!

So, how I could do this? Am I wrong to stop the IOCP loop with just setting that variable to false? Note that is would happen even if i use APC requests to stop an alertable thread.

The solution that come to my mind is to add every 'MyIoRequest' structures to a queue/list, and then dequeue them when GetQueuedCompletionStatusEx returns, but shouldn't that make some bottleneck, since the enqueue/dequeue process of such MyIoRequest structures must be interlocked? Maybe I've misunderstood how to use the IOCP loop. Can someone bring some light on this topic?

Answer 1

The way I normally shut down an IOCP thread is to post my own 'shut down now please' completion. That way you can cleanly shut down and process all of the pending completions and then shut the threads down.

The way to do this is to call PostQueuedCompletionStatus() with 0 for num bytes, completion key and pOverlapped. This will mean that the completion key is a unique value (you wont have a valid file or socket with a zero handle/completion key).

Step one is to close the sources of completions, so close or abort your socket connections, close files, etc. Once all of those are closed you can't be generating any more completion packets so you then post your special '0' completion; post one for each thread you have servicing your IOCP. Once the thread gets a '0' completion key it exits.