parallel execution of random forest in R

Setting .multicombine to TRUE can make a significant difference:

rf <- foreach(ntree=rep(25000, 6), .combine=randomForest::combine,              .multicombine=TRUE, .packages='randomForest') %dopar% {    randomForest(x, y, ntree=ntree)}

This causes combine to be called once rather than five times. On my desktop machine, this runs in 8 seconds rather than 19 seconds.

Are you aware that the caret package can do a lot of the hand-holding for parallel runs (as well as data prep, summaries, ...) for you?

Ultimately, of course, if there are some costly operations left in the random forest computation itself, there is little you can do as Andy spent quite a few years on improving it. I would expect few to no low-hanging fruits to be around for the picking...

H20 package can be used to solve your problem.

According to H20 documentation page H2O is "the open sourcemath engine for big data that computes parallel distributedmachine learning algorithms such as generalized linear models,gradient boosting machines, random forests, and neural networks(deep learning) within various cluster environments."

Random Forest implementation using H2O:

https://www.analyticsvidhya.com/blog/2016/05/h2o-data-table-build-models-large-data-sets/