How to clusterExport a function without its evaluation environment
It's hard to guess the problem without a complete example, but I'm wondering if the error message isn't coming from clusterExport
, rather than parLapply
. That would happen if functionList
was defined in a function rather than the global environment, since the clusterExport
envir
argument specifies the environment from which to export the variables.
To export variables defined in a function, from that same function, you would use:
clusterExport(cl, varlist=c("functionList", "y"), envir=environment())
I'm just guessing this might be a problem for you since I don't know how or where you defined functionList
. Note that clusterExport
always assigns the variables to the global environment of the cluster workers.
I'm also suspicious of the way that you are apparently setting the environment of a list: that seems to be legal, but I don't think it will change the environment of functions in that list. In fact, I suspect that exporting functions to the workers in a list may have other problems that you haven't encountered yet. I would use something like this:
mainFunction <- function(cl) { fa <- function(x) fb(x) fb <- function(x) fc(x) fc <- function(x) x y <- 7 workerFunction <- function(i) { do.call(functionNames[[i]], list(y)) } environment(workerFunction) <- .GlobalEnv environment(fa) <- .GlobalEnv environment(fb) <- .GlobalEnv environment(fc) <- .GlobalEnv functionNames <- c("fa", "fb", "fc") clusterExport(cl, varlist=c("functionNames", functionNames, "y"), envir=environment()) parLapply(cl, seq_along(functionNames), workerFunction)}library(parallel)cl <- makeCluster(detectCores())mainFunction(cl)stopCluster(cl)
Note that I've taken liberties with your example, so I'm not sure how well this corresponds with your problem.