R help - Apr 2007 - Error in clusterApply(): recursive default argument reference

Hi,

I want to compute a distribution of the intersection of a graph and
'randomized' graphs induced by the permutations of node labels (to
preserve the graph topology).
Since I ll have many permutations to perform, I was thinking of using
the snow package and in particular "parSapply" to divide the work
between my 4 CPUs.
But I get the following error message :

Error in clusterApply(cl, splitList(x, length(cl)), lapply, fun, ...) : 
        recursive default argument reference

What am i doing wrong ?

Some details about my platform and R version :
-----------------------------------------------
$platform
[1] "x86_64-redhat-linux-gnu"
$version.string
[1] "R version 2.4.1 (2006-12-18)"


Below is the snippet of code generating the error message :
-----------------------------------------------------------
### libraries ###
library(graph)
library(snow)

### functions ###
nullDistribIntersection <- function(g1, g2, perm=1000)
{
  n1 <- nodes(g1)
  sapply(1:perm,
	  function(i)
	  {
            nodes(g1) <- sample(n1)
	    numEdges(intersection(g1,g2))
	  }
	 )
}


parNullDistribIntersection <- function(g1, g2, perm=1000, cl=cl)
{
  n1 <- nodes(g1)
  parSapply(cl, 1:perm,
	  function(i)
	  {
            nodes(g1) <- sample(n1)
	    numEdges(intersection(g1,g2))
	  }
	 )
}


## initialize the graph and the rand seed
set.seed(123)
g <- randomEGraph(LETTERS[1:15], edges = 100)

## compute a distribution of the intersection of the graph 
## and a 'randomized' graph induced by the permutations of
## node labels (to preserve the graph topology)
cat("1CPU sys time:",
    system.time(
                 dist <- nullDistribIntersection(g,g)
               ),
    "\n"
    )

cl <- makeCluster(c("localhost", "localhost"), type =
"SOCK")
cat("Cluster sys time:",
    system.time(
                 dist <- parNullDistribIntersection(g,g)
               ),
    "\n"
    )
----------------------------------------------------------------

Thanks in advance.

Nicolas Bertin 
GSC / RIKEN Yokohama

Hi Nicolas --

I think your code is assuming that all nodes have access to the same
set of variables. One solution is to write in a more completely
'functional' style

parNullDistribIntersection <- function(g1, g2, perm=1000, cl=cl) {
    n1 = nodes(g1)                      
    parSapply(cl, 1:perm,
              function(i, g1, g2, n1) {
                  nodes(g1) <- sample(n1)
                  numEdges(intersection(g1,g2))
              },
              g1, g2, n1)
}              

Another possibility is to ensure that the relevant variables are
exported to the cluster before parSapply. With both of these you'll
want to pay some attention to the costs of communicating the data to
the nodes, which could easily be overwhelming (Rmpi's version of
parSapply is smarter at doing the data transfer -- log n time rather
than linear time, where n is the number of nodes -- and more flexible
in distributing the work).

Hope that helps,

Martin


nicolas bertin <nbertin at gsc.riken.jp> writes:
> Hi,
>
> I want to compute a distribution of the intersection of a graph and
> 'randomized' graphs induced by the permutations of node labels (to
> preserve the graph topology).
> Since I ll have many permutations to perform, I was thinking of using
> the snow package and in particular "parSapply" to divide the work
> between my 4 CPUs.
> But I get the following error message :
>
> Error in clusterApply(cl, splitList(x, length(cl)), lapply, fun, ...) : 
>         recursive default argument reference
>
> What am i doing wrong ?
>
> Some details about my platform and R version :
> -----------------------------------------------
> $platform
> [1] "x86_64-redhat-linux-gnu"
> $version.string
> [1] "R version 2.4.1 (2006-12-18)"
>
>
> Below is the snippet of code generating the error message :
> -----------------------------------------------------------
> ### libraries ###
> library(graph)
> library(snow)
>
> ### functions ###
> nullDistribIntersection <- function(g1, g2, perm=1000)
> {
>   n1 <- nodes(g1)
>   sapply(1:perm,
> 	  function(i)
> 	  {
>             nodes(g1) <- sample(n1)
> 	    numEdges(intersection(g1,g2))
> 	  }
> 	 )
> }
>
>
> parNullDistribIntersection <- function(g1, g2, perm=1000, cl=cl)
> {
>   n1 <- nodes(g1)
>   parSapply(cl, 1:perm,
> 	  function(i)
> 	  {
>             nodes(g1) <- sample(n1)
> 	    numEdges(intersection(g1,g2))
> 	  }
> 	 )
> }
>
>
> ## initialize the graph and the rand seed
> set.seed(123)
> g <- randomEGraph(LETTERS[1:15], edges = 100)
>
> ## compute a distribution of the intersection of the graph 
> ## and a 'randomized' graph induced by the permutations of
> ## node labels (to preserve the graph topology)
> cat("1CPU sys time:",
>     system.time(
>                  dist <- nullDistribIntersection(g,g)
>                ),
>     "\n"
>     )
>
> cl <- makeCluster(c("localhost", "localhost"), type
= "SOCK")
> cat("Cluster sys time:",
>     system.time(
>                  dist <- parNullDistribIntersection(g,g)
>                ),
>     "\n"
>     )
> ----------------------------------------------------------------
>
> Thanks in advance.
>
> Nicolas Bertin 
> GSC / RIKEN Yokohama
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
-- 
Martin Morgan
Bioconductor / Computational Biology
http://bioconductor.org

I cannot manage to install Rmpi_0.5-3
... using the regular "Rshell" install.packages("Rpmi")

I followed the advice of Martin to run the shell install command and
specify the location of mpi (located in "/usr/lib64/lam/" on my
machine)
That is :
R CMD  INSTALL --configure-args="--with-mpi=/usr/lib64/lam"
R/Rmpi_0.5-3.tar.gz 

But still get the following error
	* Installing *source* package 'Rmpi' ...	
	Try to find mpi.h ...
	Found in /usr/include/lam
	Try to find libmpi or libmpich ...
	checking for main in -lmpi... no
	libmpi not found. exiting...
	ERROR: configuration failed for package 'Rmpi'
	** Removing '/usr/lib64/R/library/Rmpi'

Is there anything I do wrong, or any advice on what to check to figure
out the source of my problem?

Thank you for your help

Nicolas Bertin 
GSC / RIKEN Yokohama

> Hi Nicolas --
> 
> Please consider posting your problem and eventual solution to the R
> list serv, so others can benefit.
> 
> Is this Rmpi_0.5-3 ? The messages do not quite look consistent. How
> are you installing Rmpi? It might be necessary to do so from the
> command line, e.g. with
> 
> R CMD  INSTALL --configure-args="--with-mpi=/path/to/mpi" Rmpi
> 
> where /path/to/mpi is such that /path/to/mpi/include /path/to/mpi/lib,
> etc exists
> >   I would like to try to use Rmpi as you advised me but I have some
> > problem to install this package.
> > The MPI installed on my machine was openmpi and upon installing Rmpi I
> > got the following error
> > 	checking mpi.h usability... no
> > 	checking mpi.h presence... no
> > 	checking for mpi.h... no
> > 	Try to find libmpi or libmpich ...
> > 	checking for main in -lmpi... no
> > 	libmpi not found. exiting...
> >
> >    I could read that there are some issue with openmpi and proceed to
> > get remove openmpi and install lam/mpi instead.
> > LAM/mpi seems to work fine but I now get the following error message
> > upon installing Rmpi
> > 	checking mpi.h usability... yes
> > 	checking mpi.h presence... yes
> > 	checking for mpi.h... yes
> > 	Try to find libmpi or libmpich ...
> > 	checking for main in -lmpi... no
> > 	libmpi not found. exiting...
> >
> >   Do you have any advice on what went possibly be wrong ?