What are the Advantages and Disadvantages of Two Level Scheduler (like in Apache Mesos) vs Single Level Scheduler (like in Nomad or Kubernetes)?
Full disclosure: I work for Mesosphere :)
One advantage of two-level schedulers is that they can manage different workloads in different ways.
Some workloads require access to special resources only available on certain nodes (external storage for example). Some workloads might run best on special resources but could use other types(TensorFlow should run on GPUs, but can run on CPUs in a pinch). Some workloads can be rescheduled anywhere if they node they are running on dies (Spark). In addition to variation in the resources they need, workloads also have different installation, scaling, backup, and upgrade processes. This is especially complicated for multi-node applications that need different configurations depending on their lifecycle stage.
Two-level scheduling lets you define different rules for each workload while running them all on a common pool of resources, which can keep your resource utilization high.
I think that this diagram from this blog post illustrates two-level scheduling pretty well.