R devel - Dec 2013 - Parallel computing: how to transmit multiple parameters to a function in parLapply?

Hi R-developers

In the package Parallel, the function parLapply(cl, x, f) seems to allow
transmission of only one parameter (x) to the function f. Hence in order to
compute f(x, y) parallelly, I had to define f(x, y) as f(x) and tried to
access y within the function, whereas y was defined outside of f(x).

Script:

library(parallel)

f <- function(x) {
  z <- 2 * x + .GlobalEnv$y  # Try to access y in the global scope.
  return(z)
}

np <- detectCores(logical = FALSE)  # Two cores of my laptop
x <- seq(1, 10, by = 1)
y <- 0.5  # Y may be an array in reality.
cl <- makeCluster(np)  # initiate the cluster
  r <- parLapply(cl, x, f)  # apply f to x for parallel computing
stopCluster(cl)

The r was a list with 10 empty elements which means f failed to access y.

Then I tested f without parallel computing:
z <- f(x)
print(z)
[1]  2.5  4.5  6.5  8.5 10.5 12.5 14.5 16.5 18.5 20.5

The results indicates that we can access y using .GlobalEnv$y in a function
without parLapply.

The question is, is there any method for me to transmit y to f, or access y
within f during parallel computing?

The version of my R is 3.0.1 and I am running it on a Win8-64x system.

Thanks,

Yu



--
View this message in context:
http://r.789695.n4.nabble.com/Parallel-computing-how-to-transmit-multiple-parameters-to-a-function-in-parLapply-tp4682667.html
Sent from the R devel mailing list archive at Nabble.com.

This works:

clusterExport(cl, c("f","y"), envir=environment())
r <- parLapply(cl, x, function(x) f(x,y))

You need to export your function (?f?) and additional variables (?y?), and then 
define that function inside parLapply ("f(x,y)?). If you were to also make
use of
additional libraries (or source some scripts) then you should also consult 
?clusterEvalQ?.
The makeCluster command (at least in windows via socket) just initializes new R 
processes which do not know about your functions or variables unless you
export those to them.

Perhaps a question best suited for R-help.

Alexios



On 24 Dec 2013, at 06:15, Yu Wan <walterwan at 126.com> wrote:
> Hi R-developers
> 
> In the package Parallel, the function parLapply(cl, x, f) seems to allow
> transmission of only one parameter (x) to the function f. Hence in order to
> compute f(x, y) parallelly, I had to define f(x, y) as f(x) and tried to
> access y within the function, whereas y was defined outside of f(x).
> 
> Script:
> 
> library(parallel)
> 
> f <- function(x) {
>  z <- 2 * x + .GlobalEnv$y  # Try to access y in the global scope.
>  return(z)
> }
> 
> np <- detectCores(logical = FALSE)  # Two cores of my laptop
> x <- seq(1, 10, by = 1)
> y <- 0.5  # Y may be an array in reality.
> cl <- makeCluster(np)  # initiate the cluster
>  r <- parLapply(cl, x, f)  # apply f to x for parallel computing
> stopCluster(cl)
> 
> The r was a list with 10 empty elements which means f failed to access y.
> 
> Then I tested f without parallel computing:
> z <- f(x)
> print(z)
> [1]  2.5  4.5  6.5  8.5 10.5 12.5 14.5 16.5 18.5 20.5
> 
> The results indicates that we can access y using .GlobalEnv$y in a function
> without parLapply.
> 
> The question is, is there any method for me to transmit y to f, or access y
> within f during parallel computing?
> 
> The version of my R is 3.0.1 and I am running it on a Win8-64x system.
> 
> Thanks,
> 
> Yu
> 
> 
> 
> --
> View this message in context:
http://r.789695.n4.nabble.com/Parallel-computing-how-to-transmit-multiple-parameters-to-a-function-in-parLapply-tp4682667.html
> Sent from the R devel mailing list archive at Nabble.com.
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

You can put the function of interest and any global
variables it needs into a private environment, which gets sent
along with the function to the child processes.  E.g.

library(parallel)
cl3 <- makeCluster(3)
y <- c(1,100,10000)
addY <- function(x) x + y
withGlobals <- function(FUN, ...){
    environment(FUN) <- list2env(list(...))
    FUN
}
parLapply(cl3, 1:4, withGlobals(addY, y=y))
# [[1]]
# [1]     2   101 10001
# 
# [[2]]
# [1]     3   102 10002
# ...

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com

> -----Original Message-----
> From: r-devel-bounces at r-project.org [mailto:r-devel-bounces at
r-project.org] On Behalf
> Of Yu Wan
> Sent: Monday, December 23, 2013 10:16 PM
> To: r-devel at r-project.org
> Subject: [Rd] Parallel computing: how to transmit multiple parameters to a
function in
> parLapply?
> 
> Hi R-developers
> 
> In the package Parallel, the function parLapply(cl, x, f) seems to allow
> transmission of only one parameter (x) to the function f. Hence in order to
> compute f(x, y) parallelly, I had to define f(x, y) as f(x) and tried to
> access y within the function, whereas y was defined outside of f(x).
> 
> Script:
> 
> library(parallel)
> 
> f <- function(x) {
>   z <- 2 * x + .GlobalEnv$y  # Try to access y in the global scope.
>   return(z)
> }
> 
> np <- detectCores(logical = FALSE)  # Two cores of my laptop
> x <- seq(1, 10, by = 1)
> y <- 0.5  # Y may be an array in reality.
> cl <- makeCluster(np)  # initiate the cluster
>   r <- parLapply(cl, x, f)  # apply f to x for parallel computing
> stopCluster(cl)
> 
> The r was a list with 10 empty elements which means f failed to access y.
> 
> Then I tested f without parallel computing:
> z <- f(x)
> print(z)
> [1]  2.5  4.5  6.5  8.5 10.5 12.5 14.5 16.5 18.5 20.5
> 
> The results indicates that we can access y using .GlobalEnv$y in a function
> without parLapply.
> 
> The question is, is there any method for me to transmit y to f, or access y
> within f during parallel computing?
> 
> The version of my R is 3.0.1 and I am running it on a Win8-64x system.
> 
> Thanks,
> 
> Yu
> 
> 
> 
> --
> View this message in context:
http://r.789695.n4.nabble.com/Parallel-computing-how-
> to-transmit-multiple-parameters-to-a-function-in-parLapply-tp4682667.html
> Sent from the R devel mailing list archive at Nabble.com.
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel