How to manage/balance semi persistent jobs over service instances
You should have a look at Gearman.
It composes of a client which assigns the jobs, one or more workers which will pick up and execute the jobs and a server which will maintain the list of functions (services) and jobs pending. It will re-assign the jobs if a worker dies.
Your workers sound like (api-less) services itself. So, your requirements can be reformulated as:
- Knows about deployed services
- Knows about nodes that can host there services
- Can deploy services to nodes
- Can [send job updates to services] = redeploy services/invoke some API on deployed services
- Can redeploy service if service or node dies
Look at Docker to deploy, run and manage isolated processes on host.
RabbitMq is simple message queue that is fairly easy to get going with.