Can SPARK use multicore properly? Can SPARK use multicore properly? multithreading multithreading

Can SPARK use multicore properly?


Yes, it can, as this is its stated purpose — to split and parallelize what is parallelizeable. You can even specify amount of memory to be used by each executor.

However, some tasks cannot be parallelized, which is why sometimes Spark only occupies one core.

If you use the Spark shell, make sure you set the number of cores to use, as it is said in the answer to this question Why is Spark not using all cores on local machine

Source: official Spark docs https://spark.apache.org/docs/latest/configuration.html