multithreaded epoll

I think option 1 is more popular since the primary purpose of non-blocking IO is to avoid the overhead of create & destroy threads.

take the popular web server nginx as an example, it create multiple processes (not threads) to handle incoming events on a handle, and the process the events in the subprocess. all of them share the same listening socket. it's quite similar to option 1.

I'm also writing a server using epoll, and I have considered the same model as you attached.

It is possible to use option 1, but it may cause "thundering herd" effect, you can read the source of nginx to find the solution. As for option 2, I deem that it is better to use thread pool instead of spawning a new thread each time.

And you can also the following model：

Main thread/process: accept incoming connection with blocking IO, and send the fd to the other threads using BlockingList or to the other processes using PIPE.

Sub threads/process: create an instance of epoll respectively, and add the incoming fd to the epoll, then processing them with non-blocking IO.

epoll is thread safe, a good solution is that your main process stay in accept(2), once you get the file descriptor register that one in the epoll fd for the target thread, that means that you have a epoll queue for each thread, once you create the thread you share the epoll file descriptor as parameter in the call pthread_create(3), so when a new connection arrives, you do a epoll_ctl(...EPOLL_CTL_ADD..) using the epoll fd for the target thread and the new socket created after accept(2), make sense ?