Rails 3 + Heroku + Delayed jobs - Help me understand!

If you don't know why you need Resque, then you don't need it ;)

Resque is for high-scalability. delayed_job is fine for smaller-scale stuff, but once you get to the size of, say, Github, you will need something like Resque. If delayed_job works for you, then stay with it. You don't need to worry about replacing it until your background jobs queue gets around 30,000 or so.

To autoscale heroku workers using delayed job, you can hook into the enqueue and after hooks and use the heroku api to query/update the number of workers.

For the most basic implementation on enqueue, check to see if there are workers and if not add a worker. On after, check to see if there are other delayed jobs and if not reduce the workers to 0.

You can obviously make this more sophisticated in the way that you scale.

Here is a basic implementation: https://github.com/phaza/Heroku-Delayed-Job-Autoscale

hirefireapp is a new-ish simple drop-in solution to auto-scaling workers.

It spawns workers for you based on queue size (configurable) and then "fires" them when they are no longer necessary. You pay for the dyno time (to the nearest second) and for the hirefireapp service. In theory you could roll your own using the open source hirefire gem too.

It also handles scaling the web side if you choose, so you can spawn more web dynos based on current latency.