How to gracefully handle thousands of Quartz misfires?

In both of these cases, around 10,000 quartz jobs get spawned

No need to spawn new quartz jobs. Quartz is a scheduler - not a task manager.

In the nightly reprocess - you need only that one quartz cron job to invoke some service responsible for managing and running the 10,000 tasks. In the "on demand" scenario, quartz shouldn't be involved at all. Just invoke that service directly.

How does the service manage 10,000 tasks?

Typically, when only one JVM is available, you'd just use some ExecutorService. Here, since you have 6 nodes under your fingers, you can easily use Hazelcast. Hazelcast is a java library that enables you to cluster your nodes, sharing resources efficiently with each other. Hazelcast has a straightforward solution distributing your ExecutorService, that's called Distributed Executor Service. It's as easy as creating a Hazelcast ExecutorService and submitting the task on all members. Here's an example from the documentation for invoking on a single member:

Callable<String> task = new Echo(input); // Echo is just some CallableHazelcastInstance hz = Hazelcast.newHazelcastInstance();IExecutorService executorService = hz.getExecutorService("default");Future<String> future = executorService.submitToMember(task, member);String echoResult = future.get();

I would do this by making use of a queue (RabbitMQ/ActiveMQ). The cron job (or whatever your on-demand trigger is) populates the queue with messages representing the 10,000 work instructions (i.e. the instruction to reprocess the data for a given domain object).

On each of your nodes you have a pool of executors which pull from the queue and carry out the work instruction. This solution means that each executor is kept as busy as possible while there are still work items on the queue, meaning that the overall processing is accomplished as quickly as possible.

The best way is to use a cluster of Quartz Instances. This will share the jobs between many cluster nodes :http://quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigJDBCJobStoreClustering