20 Receives per second with SocketAsyncEventArgs

The reason it's slowing down is because each one of those threads needs to context switch, and that's a relatively expensive operation. The more threads you add, the larger percentage of your CPU is spent simply on context switching and not in your actual code.

You've hit this one-thread-per-client bottleneck in a rather odd way. The whole point of server-side async is to reduce the number of threads -- to not have one thread per client, but ideally only one or two for each of the logical processors in your system.

The async code you posted looks fine, so I can only guess your Process method has some easy to overlook non-async, blocking I/O in it, i.e. a database or file access. When I/O blocks, the .NET thread pool detects this and automatically spins up a new thread -- it's basically spiraling out of control here with the I/O in Process as a bottleneck.

An async pipeline really needs to be 100% async to get any significant benefit from it. Half-in will have you writing complex code that performs just as poorly as simple sync code.

If you absolutely can't make the Process method purely async, you might have some luck faking it. Have things wait in a queue for processing by a small, limited-size thread pool.