Hadoop: Heap space and gc problems Hadoop: Heap space and gc problems hadoop hadoop

Hadoop: Heap space and gc problems


If you maximum memory size hasn't changed, it will be 1/4 of main memory i.e. about 3 GB plus some overhead for non-heap usage could be 3.5 GB.

I suggest you try

export HADOOP_OPTS="-XX:+UseParallelGC -Xmx8g"

to set the maximum memory to 8 GB.


By default the maximum heap size is 1/4 of the memory (unless you are running a 32-bit JVM on Windows). So if the maximum heap size is being ignored it will still be 3 GB.

Whether you use one GC or another, it won't make much difference to when you run out of memory.

I suggest you take a heap dump with -XX:+HeapDumpOnOutOfMemoryError and read this in a profiler, e.g. VisualVM to see why it's using so much memory.