Distributed Job scheduling, management, and reporting
I have since found Spring Batch and Spring Batch Integration which appear to address many of my requirements. I will let you know how I get on.
Take a look at Quartz. I think it supports stuff like managing jobs remotely and clustering several machines to run jobs.
I guess you are looking for a workflow engine for CPU intensive tasks (also know "scientific workflow", e.g. http://www.extreme.indiana.edu/swf-survey). But I'm not sure how distributed do you want it to be. Usually all workflow engines have a "single point of failure".