R devel - Dec 2020 - [External] setting .libPaths() with parallel::clusterCall

Dear all,

It is not possible to set library paths on worker nodes with
parallel::clusterCall (or snow::clusterCall) and I wonder if this is
intended behavior.

Example.

library(parallel)
libdir <- "./tmplib"
if (!dir.exists(libdir)) dir.create("./tmplib")

cl <- makeCluster(2)
clusterCall(cl, .libPaths, c(libdir, .libPaths()) )

The output is as expected with the extra libdir returned for each worker
node. However, running

clusterEvalQ(cl, .libPaths())

Shows that the library paths have not been set.

If this is indeed a bug, I'm happy to file it at bugzilla. Tested on R
4.0.3 and r-devel.

Best,
Mark
ps: a workaround is documented here:
https://www.markvanderloo.eu/yaRb/2020/12/17/how-to-set-library-path-on-a-parallel-r-cluster/

> sessionInfo()
R Under development (unstable) (2020-12-21 r79668)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS

Matrix products: default
BLAS:   /home/mark/projects/Rdev/R-devel/lib/libRblas.so
LAPACK: /home/mark/projects/Rdev/R-devel/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=nl_NL.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=nl_NL.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=nl_NL.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=nl_NL.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods
[8] base

loaded via a namespace (and not attached):
[1] compiler_4.1.0

	[[alternative HTML version deleted]]

On Tue, 22 Dec 2020, Mark van der Loo wrote:
> Dear all,
>
> It is not possible to set library paths on worker nodes with
> parallel::clusterCall (or snow::clusterCall) and I wonder if this is
> intended behavior.
>
> Example.
>
> library(parallel)
> libdir <- "./tmplib"
> if (!dir.exists(libdir)) dir.create("./tmplib")
>
> cl <- makeCluster(2)
> clusterCall(cl, .libPaths, c(libdir, .libPaths()) )
>
> The output is as expected with the extra libdir returned for each worker
> node. However, running
>
> clusterEvalQ(cl, .libPaths())
>
> Shows that the library paths have not been set.
Use this:

     clusterCall(cl, ".libPaths", c(libdir, .libPaths()) )

This will find the function .libPaths on the workers.

Your clusterCall sends across a serialized copy of your process'
.libPaths and calls that. Usually that is equivalent to calling the
function found by the name you used on the workers, but not when the
function has an enclosing environment that the function modifies by
assignment.

Alternate implementations of .libPaths that are more
serialization-friendly are possible in principle but probably not
practical given limitations of the base package.

The distinction between providing a function value or a character
string as the function argument to clusterCall and others could
probably use a paragraph in the help file; happy to consider a patch
if anyone wants to take a crack at it.

Best,

luke
>
> If this is indeed a bug, I'm happy to file it at bugzilla. Tested on R
> 4.0.3 and r-devel.
>
> Best,
> Mark
> ps: a workaround is documented here:
>
https://www.markvanderloo.eu/yaRb/2020/12/17/how-to-set-library-path-on-a-parallel-r-cluster/
>
>
>> sessionInfo()
> R Under development (unstable) (2020-12-21 r79668)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Ubuntu 20.04.1 LTS
>
> Matrix products: default
> BLAS:   /home/mark/projects/Rdev/R-devel/lib/libRblas.so
> LAPACK: /home/mark/projects/Rdev/R-devel/lib/libRlapack.so
>
> locale:
> [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
> [3] LC_TIME=nl_NL.UTF-8        LC_COLLATE=en_US.UTF-8
> [5] LC_MONETARY=nl_NL.UTF-8    LC_MESSAGES=en_US.UTF-8
> [7] LC_PAPER=nl_NL.UTF-8       LC_NAME=C
> [9] LC_ADDRESS=C               LC_TELEPHONE=C
> [11] LC_MEASUREMENT=nl_NL.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] parallel  stats     graphics  grDevices utils     datasets  methods
> [8] base
>
> loaded via a namespace (and not attached):
> [1] compiler_4.1.0
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu