R help - Jun 2012 - undefined S4 class in parallel computing at snowfall

Dear All,

I have a question of how to export S4 class specification to
clusters/workers in parallel computing. The package I used is snowfall. The
problem is reproducible as follows. Any hint is greatly appreciated.

Edwin Sun

=== begin ==========library(snowfall)
sfInit(parallel = TRUE, cpus = 2)
setClass("catt", representation(aa = "numeric"))
setClass("dogg", representation(bb = 'character', cc =
"catt"))
f1 <- function(y1) { new('catt', aa = y1 + 1) }
f3 <- function(y2, y3) { new('dogg', bb=c(y2, 'GA'), cc =
f1(y3)) }
dat <- 1:5
f1(dat)
f3('NY', dat)     
sapply(c("state1", 'state2', 'state3'), f3, dat) 
sfExportAll()
sfClusterEval(ls())
sfSapply(c("s1", 's2', 's3'), f3, dat) 
sfStop() 

# sfSapply generates the following error
#    Error in checkForRemoteErrors(val) : 
#      2 nodes produced errors; first error: "dogg" is not a defined
class
=== end ============
--
View this message in context:
http://r.789695.n4.nabble.com/undefined-S4-class-in-parallel-computing-at-snowfall-tp4634757.html
Sent from the R help mailing list archive at Nabble.com.

Hi --

On 06/28/2012 04:51 PM, Edwin Sun wrote:
> Dear All,
>
> I have a question of how to export S4 class specification to
> clusters/workers in parallel computing. The package I used is snowfall. The
> problem is reproducible as follows. Any hint is greatly appreciated.
>
> Edwin Sun
>
> === begin ==========> library(snowfall)
> sfInit(parallel = TRUE, cpus = 2)
> setClass("catt", representation(aa = "numeric"))
> setClass("dogg", representation(bb = 'character', cc =
"catt"))
> f1<- function(y1) { new('catt', aa = y1 + 1) }
> f3<- function(y2, y3) { new('dogg', bb=c(y2, 'GA'), cc =
f1(y3)) }
> dat<- 1:5
> f1(dat)
> f3('NY', dat)
> sapply(c("state1", 'state2', 'state3'), f3, dat)
> sfExportAll()
I think it makes sense to actually define the classes (and other 
cluster-specific objects) on the cluster;

   sfInit(parallel = TRUE, cpus = 2)
   sfClusterEval({
       setClass("catt", representation(aa = "numeric"))
       setClass("dogg", representation(bb = 'character', cc =
"catt"))
       f1 <- function(y1) new('catt', aa = y1 + 1)
   })

   f3 <- function(y2, y3) { new('dogg', bb=c(y2, 'GA'), cc =
f1(y3)) }
   dat <- 1:5
   sfSapply(c("s1", 's2', 's3'), f3, dat)

It's true that setClass creates variables in the global environment (see 
the .__C__* objects in the output of ls(all=TRUE)) and that sfExportAll 
seems not to export these, so that you could export them explicitly, 
e.g., sfExport(list=ls(all=TRUE)), but this assuming that these 
variables are the only side effect of calling setClass. Maybe the 
'setup' actions are defined in a separate script setup.R that 
sfClusterEval (and the main program, if that's relevant) source, 
sfClusterEval(source("setup.R")).

Martin
> sfClusterEval(ls())
> sfSapply(c("s1", 's2', 's3'), f3, dat)
> sfStop()
>
> # sfSapply generates the following error
> #    Error in checkForRemoteErrors(val) :
> #      2 nodes produced errors; first error: "dogg" is not a
defined class
> === end ============>
> --
> View this message in context:
http://r.789695.n4.nabble.com/undefined-S4-class-in-parallel-computing-at-snowfall-tp4634757.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

-- 
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793