YARN not preempting resources based on fair shares when running a Spark job

You need to set one of the preemption timeouts in your allocation xml. One for minimum share and one for fair share, both are in seconds. By default, the timeouts are not set.

From Hadoop: The Definitive Guide 4th Edition

If a queue waits for as long as its minimum share preemption timeout without receiving its minimum guaranteed share, then the scheduler may preempt other containers. The default timeout is set for all queues via the defaultMinSharePreemptionTimeout top-level element in the allocation file, and on a per-queue basis by setting the minSharePreemptionTimeout element for a queue.

Likewise, if a queue remains below half of its fair share for as long as the fair share preemption timeout, then the scheduler may preempt other containers. The default timeout is set for all queues via the defaultFairSharePreemptionTimeout top-level element in the allocation file, and on a per-queue basis by setting fairSharePreemptionTimeout on a queue. The threshold may also be changed from its default of 0.5 by setting defaultFairSharePreemptionThreshold and fairSharePreemptionThreshold (per-queue).

Fair Scheduler doesn't kill containers for the first job, It only wait until some resources are free up and reserve them to be used by the second job. If not resources are free up from the first job, the scheduler can not assign those resources to the second job.

In MapReduce jobs, each map or reduce task require to instantiate a new container and the scheduler can block the job to instantiate new containers if it has exceeded its quote (based on the queue capacity).

In Spark the things are different, the executors are being initiated at the beginning of the job and the different tasks (stages) are sent to them. Then the resources are not being free up and they can not be reallocated.

May be dynamic allocation could help: http://spark.apache.org/docs/1.6.1/configuration.html#dynamic-allocation