ThreadPool thread doing I/O operation - can thread be reused while waiting?

Question

I have been reading a bit about async/await vs ThreadPool vs Threads, and I have to admit, I'm not fully clear on the details. There is one specific question that I think I have the answer to, but I can't say for sure.

I am also aware of the many many questions on SO and elsewhere, where these questiosn are being discussed and explained. I have read a fair few here on SO, but I haven't found a clear answer, or at least as clear as I'd like.

What I gathered:

• Using async/await in an I/O operation, will make that thread available to use for other things, while the I/O operation continues elsewhere
• For a server application, this means higher throughput etc

However, a collegue said something like this:

• that using a ThreadPool to execute the same I/O operation would behave almost the same as async/await
• that when the ThreadPool thread "hands over" the I/O operation to the OS, it might wait for the answer, but that it won't matter much, since there is no work done on the CPU (the thread is waiting), and thus, the ThreadPool/OS/framework can use or spawn another thread to do some other work.
• Since the threadpool has a limit of some 32000+ threads, it wouldn't matter much, since it could just use more threads. The cost for a thread is very small, just a few bytes of memory

So, what I am asking is:

• Does the Thread from a ThreadPool block in I/O operations?
• Does it matter if it does, since the OS/framework/Threadpool can just use another thread from the pool if needed for some other work?
• Bottom line: I/O operations with async/await or ThreadPool; does it matter for efficiency and throughput?

Please note that I am not discussing the client-side of things, just from a server perspective.

And sorry in advance if I missed an exact answer to this, I have looked =)

Answer 1

The real issue is not "thread pool vs. async/await", it's synchronous I/O vs. asynchronous I/O. [1]

On Windows, threads are relatively expensive objects -- a thread has a lot of OS bookkeeping associated with it, not to mention 1 MB of preallocated stack space. This puts a fairly low cap on how many threads the system will even support, and even when you don't hit that cap, context switching between all those threads is not cheap either. That "32,000 threads" limit is a strictly theoretical one, and highly optimistic about how many threads you can have and still be responsive! [2]

Enter asynchronous I/O, which is optimized to use only as many threads as necessary (usually some conservative multiple of the number of physical processor cores in the system), ideally never even creating new threads beyond the initial batch. These threads are dedicated to handling completed I/O operations by removing them from a queue (known as a completion port). While an asynchronous operation is in progress, no thread is dedicated to it at all, not even as an item on a wait list (Stephen Cleary has a nice blog post about it that explains this in more detail). Not much imagination is needed to think about what's more efficient:

• A few thousand individual threads that each wait on a particular operation, which have to be woken up and switched to (and between) depending on what operation(s) completed; or
• A few dozen threads (if that), each of which can handle any completed operation, so that only as many ever need to run as necessary to be responsive.

As it turns out, the latter scales much better than the former; the "thread per request" model which is common in naive server code quickly shows its limits, even when you use a thread pool to reduce the creation of new threads. Note that this was an issue long before async/await was ever a thing, and so is the solution Windows went with; async/await is just a new way of writing code to use the existing mechanisms.

Are you likely to notice a difference with only a few requests in flight at a time? No. But since async/await essentially allows you to write code that looks synchronous but has the scalability of asynchronous I/O "for free", why would you not choose to use that in favor of synchronous I/O queued to the thread pool?

---

[1] Turns out Stephen Cleary already wrote most of what's in this answer a few years ago. I recommend you read that as well.

[2] Here's an older post by Mark Russinovich where he actually tries to squeeze as many threads out of the system as possible -- just for fun and profit. He "only" gets to 55K on a 64-bit machine before all resources are gone, and that's with adjusting the default stack size, and without doing any actually useful work. On a modern system you could probably get more, but the real question should not be "how many threads can I have" -- if it is, you're Doing it Wrong.