How do I increase Tez's container physical memory? How do I increase Tez's container physical memory? hadoop hadoop

How do I increase Tez's container physical memory?


I have had this problem a lot. The changing

Set hive.tez.container.size=6656;Set hive.tez.java.opts=-Xmx4g;

does not fix the problem for me but this does:

set tez.am.resource.memory.mb=4096;


Set the Tez container size to be a larger multiple of the YARN container size (4GB):

SET hive.tez.container.size=4096MB

"hive.tez.container.size" and "hive.tez.java.opts" are the parameters that alter Tez memory settings in Hive. If "hive.tez.container.size" is set to "-1" (default value), it picks the value of "mapreduce.map.memory.mb". If "hive.tez.java.opts" is not specified, it relies on the "mapreduce.map.java.opts" setting. Thus, if Tez specific memory settings are left as default values, memory sizes are picked from mapreduce mapper memory settings "mapreduce.map.memory.mb".

https://documentation.altiscale.com/memory-settings-for-tez

For more info Tez configuration and Tez memory tuning

Note: Set in MB with Ambari


Incase anyone else stumbles upon this thread trying to solve this above, here is a link to a real solution that worked for me where all the other solutions did not.

http://moi.vonos.net/bigdata/hive-cli-memory/

TL;DR add these to your hive call --hiveconf tez.am.resource.memory.mb=<size as int>--hiveconf tez.am.launch.cmd-opts=""