yarn is not honouring yarn.nodemanager.resource.cpu-vcores yarn is not honouring yarn.nodemanager.resource.cpu-vcores hadoop hadoop

yarn is not honouring yarn.nodemanager.resource.cpu-vcores


YARN is running more containers than allocated cores because by default DefaultResourceCalculator is used. It considers only memory.

public int computeAvailableContainers(Resource available, Resource required) {// Only consider memoryreturn available.getMemory() / required.getMemory();  }

Use DominantResourceCalculator, It uses both cpu and memory.

Set below config in capacity-scheduler.xml

yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

More about DominantResourceCalculator