How does io_uring fulfill asynchronus read?

Question

Recently, I have been trying to incorporate async read into an existing feature and came around io_uring. I read the documentations present on internet and had the impression tht io_uring spawns a new thread io-wrk for doing the operations. But I am unable to see this thread being spawned. I am opening the file with following flags:

fd = open("file1.txt", O_DIRECT | O_RDONLY)

I am trying to see the thread as follows

top -H -p <pid_of_process>

Edit -

liburing snippet used for spawning io_uring and submitting read request

struct io_uring params;
memset(&params, 0 , sizeof(params));
io_uring_queue_init_params(128, &ring, &params);

char* ptr = (char*)aligned_alloc(512, 4096);
auto sqe = io_uring_get_sqe(&ring)
io_uring_prep_read(sqe, fd, ptr, 4096, -1);

Any help on what is happening is much appreciated.

Answer 1

As far as I can tell, io_uring does not spawn any user threads. That being said, kernel threads can be spawned depending on how io_uring has been setup.

The setup function is io_uring_setup and many configuration are possible. Some of them should not create kernel threads while others should. For example, with an interrupt-driven configuration, the kernel can avoid creating a background kernel thread since the progression can be done during calls to io_uring_enter. Alternatively, with a kernel-polling strategy, the only possible implementation is to create a kernel thread since the progression is guaranteed without doing any system call as long as the pooling kernel thread is not sleeping (but it can be awaken by the application with special io_uring_enter calls as explained in the above manual link). See this post for more information about the possible high-level modes).

Using liburing, io_uring_queue_init-based functions call io_uring_setup based on the provided parameter structure or flags. Since your flags are manually zero-initialised before calling io_uring_queue_init_params and liburing pass the parameters indicated by params straight through io_uring_setup, I expect all the flags to be disabled. The (long) manual specifically states what happens in this case (in the middle though):

If no flags are specified, the io_uring instance is setup for interrupt driven I/O . I/O may be submitted using io_uring_enter(2) and can be reaped by polling the completion queue.

Besides the default behaviour which applies is the following:

io_uring will process all outstanding work at the end of any system call or thread interrupt.

io_uring will interrupt a task running in userspace when a completion event comes in. This is to ensure that completions run in a timely manner. [...]. Most applications don't need the forceful interruption, as the events are processed at any kernel/user transition. [...]

This does not specify whether the implementation creates a kernel thread or not in the interrupt-driven mode (which is the one used in your case). This is I think dependent of the implementation and may change. By default, if it does not create one, then progression is ensured at the end of any system calls, and if it does, then the kernel thread will cause the user process to be interrupted.

Note a kernel thread can be created for a very short period of time so it can hardly be seen. I advise you to use perf sched (with elevated privileges) so to profile precisely that instead of just using top.

If you want a kernel thread to ensure the progression of asynchronous IO operations, then I think you should use another mode enforcing that like IORING_SETUP_SQPOLL:

When this flag is specified, a kernel thread is created to perform submission queue polling. An io_uring instance configured in this way enables an application to issue I/O without ever context switching into the kernel. By using the submission queue to fill in new submission queue entries and watching for completions on the completion queue, the application can submit and reap I/Os without doing a single system call.

I advise you to carefully read the details of this mode since its as non-negligible implications (e.g. on io_uring_enter syscalls and liburing's io_uring_submit).

Answer 2

Being unable to see a io-wrk thread may be expected - by default io_uring will try and complete an operation "inline" and only if said operation would need to block before finishing quickly does it push the work to a helper thread. If you want io_uring to always punt to a thread you can use the IOSQE_ASYNC flag (5.6 and above kernels). From the io_uring_sqe_set_flags man-page:

IOSQE_ASYNC

Normal operation for io_uring is to try and issue an sqe as non-blocking first, and if that fails, execute it in an async manner. To support more efficient overlapped operation of requests that the application knows/assumes will always (or most of the time) block, the application can ask for an sqe to be issued async from the start. Note that this flag immediately causes the SQE to be offloaded to an async helper thread with no initial non-blocking attempt. This may be less efficient and should not be used liberally or without understanding the performance and efficiency tradeoffs.