Optimizing Apache Spark on Kubernetes using custom plugins and the scheduling framework
It sounds like what you would like to have is Gang Scheduling 📆. If you'd like to have that capability, I suggest you use Volcano to schedule/run 🏃 your jobs in Kubernetes with Gang Scheduling.
Another approach is to create your own scheduler using the scheduler extender as described here or use the Palantir gang scheduler extender.
✌️