unicorn request queuing

I've just diagnosed a similar looking New relic graph as being entirely the fault of SSL. Try turning it off. We are seeing 400ms request queuing time, which drops to 20ms without SSL.

Some interesting points on why some SSL providers might be slow: http://blog.cloudflare.com/how-cloudflare-is-making-ssl-fast

What version of ruby, unicorn, nginx (shouldn't matter much but worth mentioning) and newrelic_rpm are you using?

Also, I would try running a baseline perf test without newrelic. NewRelic parses the response and there are cases where this can be slow due to the issue with 'rindex' in ruby pre-1.9.3. This is usually only noticeable when you're response is very large and doesn't contain 'body' tags (e.g. AJAX, JSON, etc). I saw an example of this where a 1MB AJAX response was taking 30 seconds for NewRelic to parse.

Are you sure that you are buffering the requests from the clients in nginx and then buffering the responses from the unicorns before sending them back to the clients. From your setup it seems that you do (because this is by default), but I will suggest you double check that.

The config to look at is:

http://wiki.nginx.org/HttpProxyModule#proxy_buffering

This is for the buffering of the response from the unicorns. You definitely need it because you don't want to keep unicorns busy sending data to a slow client.

For the buffering of the request from the client I think that you need to look at:

http://wiki.nginx.org/HttpCoreModule#client_body_buffer_size

I think all this can't explain a delay of 100ms, but I am not familiar with all of your system setup, so it is worth it to have a look at this direction. It seems that your queuing is not caused by a CPU contention, but by some kind of IO blocking.