how to update config parameter of hadoop mapred-site.xml without restarting the cluster how to update config parameter of hadoop mapred-site.xml without restarting the cluster hadoop hadoop

how to update config parameter of hadoop mapred-site.xml without restarting the cluster


You can set this per-job in your job's configuration. If you want to change this behaviour globally you have to restart your cluster. There is no way to change it without restarting.


According to the Hadoop - The Definitive Guide

Be aware that some properties have no effect when set in the client configuration. For example, if you set mapred.tasktracker.map.tasks.maximum in your job submission with the expectation that it would change the number of task slots for the tasktrackers running your job, you would be disappointed, because this property is honored only if set in the tasktracker’s mapred-site.xml file. In general, you can tell the component where a property should be set by its name, so the fact that mapred.tasktracker.map.tasks.maximum starts with mapred.tasktracker gives you a clue that it can be set only for the tasktracker daemon. This is not a hard and fast rule, however, so in some cases you may need to resort to trial and error, or even to reading the source.

The properties which are honored in the client side and are job specific don't a need a cluster restart, but on the other hand cluster specific configurations like mapred.tasktracker.map.tasks.maximum would require the cluster to be restarted.


If you are using MRv2 (YARN for Resource scheduling), the changes made in mapred-site.xml file will be read every time when you submit the job and all the configs will be saved as job config xml file (when the job is submitted to cluster). There is no need to restart any services.