Behavior of C++ std::thread

This application is certainly going to be I/O-bound and not CPU-bound, meaning that the vast majority of the time spent processing any single request is spent waiting in blocking I/O operations, not actually doing computation.

So having more threads (up to a point, but likely past 256 of them) is going to be faster, because it allows more concurrent I/O on different sockets as they all swap off the CPUs.

In other words, it's not the 8 cores that's the bottleneck, it's the socket communication. So you want to parallelize that as much as possible (or use non-blocking I/O).

What matters is not the cost of creating a thread vs. the cost of the work. What matters is the cost of your thread queue and work distribution relative to the total cost of thread creation and command execution.

In particular, I find myself disconcerted by this in your thread queue dispatch logic:

    std::unique_lock<std::mutex>     lock(safe);    cond.wait(lock, [this](){return !(this->futures.empty() && !this->finished);});

Every job's execution will be preceded by this lock. This makes getting something out of your thread queue expensive. Furthermore, a similar lock is used when adding a job. And there's only one thread which adds jobs; if it blocks, then some queue may not get a next job.

Indeed, if any of these block, somebody's probably going to be left in the cold. That's probably why adding more threads to the queue helped; it decreased the chance of someone having to block.

I don't know the deep details of asynchronous programming, but I think lock-free queues would be better for performance.

If you want to scale linearly across more than about 8 cores you need to look at lock-free algorithms. This means implement compare-and-swap methods for the data you need to share between the threads.

There is a great book by Anthony Williams, C++ Concurrency in Action: Practical Multithreading that will show you some of the ideas.