R help - Feb 2007 - Snow Package and R: Exported Variable Problem

Hello and thanks in advance for your time.

I've created a simulation on my cluster which uses a custom package  
developed by me for different functions and also the snow package.   
Right now I'm using LAM to communicate between nodes and am currently  
only testing my code on 3 nodes for simplicity, though I plan on  
expanding to 16 later.  My problem is this error:

"Error in fn(par, ..) : object \"x1\" not found \n"
attr(,"class")
"try-error"

In my simulation I need to run a function several times with an  
different variable each time.  All the invocations on the functions  
are independent of the others.  I start the simulation on one node,  
create a cluster of several nodes, load my custom package and snow on  
all of them, use  clusterExport(cl, "x1") to export the variable  
x1(among other variable I need), then I call my simulation on the  
cluster using clusterApplyLB(cl, 2:S, simClust)  where cl is the  
cluster and S is a constant defined above as 500.  Using print  
statements (since snow, or R for that matter, has next to no ability  
to debug) I found that the error cropped up in this statement:

theta6 = optim(c(0,0,0,0,0,0,.2), loglikelihood, scrore6, method =  
"CG", control=list(fnscale=-1,reltol=1e-8,maxit=2000))$par

Both the functions loglikelelihood and score6 use x1, but I know that  
it is getting exported to the node correctly since it gets assigned  
earlier in the simulation:

x1 = rep(0,n1)

The error I stated above happens fo every itteration of the simulation  
(499 times) and I'm really at a loss as to why its happening and what  
I can do to determine what it is.  I'm wondering at this point if  
exporting the variable makes it unavailable to certain other packages,  
though that doesn't really make any sense.

If anyone can help me with this problem, or let me know how I can  
debug this, or even a clue as to why it might be happening I would  
greatly appreciate it.  I've been wrestling with this for some time  
and no online documentation can help.  Thank you for your time and help.

Just so you know I'm a Computer Scientist not a Statistician, though I  
will be able to give any information about the statistics involved in  
this program.  I am reluctant to give away all source code since it is  
not my work but rather code I'm converting from standard code to  
parallelized code for a professor of mine.

Dear Robert,


On 2/2/07, robert.robinson at maine.edu <robert.robinson at maine.edu>
wrote:
> Hello and thanks in advance for your time.
>
> I've created a simulation on my cluster which uses a custom package
> developed by me for different functions and also the snow package.
> Right now I'm using LAM to communicate between nodes and am currently
> only testing my code on 3 nodes for simplicity, though I plan on
> expanding to 16 later.  My problem is this error:
>
> "Error in fn(par, ..) : object \"x1\" not found \n"
> attr(,"class")
> "try-error"
>
> In my simulation I need to run a function several times with an
> different variable each time.  All the invocations on the functions
> are independent of the others.  I start the simulation on one node,
> create a cluster of several nodes, load my custom package and snow on
> all of them, use  clusterExport(cl, "x1") to export the variable
> x1(among other variable I need), then I call my simulation on the
> cluster using clusterApplyLB(cl, 2:S, simClust)  where cl is the
> cluster and S is a constant defined above as 500.  Using print
> statements (since snow, or R for that matter, has next to no ability
> to debug) I found that the error cropped up in this statement:
>
> theta6 = optim(c(0,0,0,0,0,0,.2), loglikelihood, scrore6, method >
"CG", control=list(fnscale=-1,reltol=1e-8,maxit=2000))$par
>
> Both the functions loglikelelihood and score6 use x1, but I know that
> it is getting exported to the node correctly since it gets assigned
> earlier in the simulation:
>
> x1 = rep(0,n1)
>
> The error I stated above happens fo every itteration of the simulation
> (499 times) and I'm really at a loss as to why its happening and what
> I can do to determine what it is.  I'm wondering at this point if
> exporting the variable makes it unavailable to certain other packages,
> though that doesn't really make any sense.
>
>From reading quickly through your description, I do not see anything
obviously wrong.

> If anyone can help me with this problem, or let me know how I can
> debug this, or even a clue as to why it might be happening I would
> greatly appreciate it.  I've been wrestling with this for some time
> and no online documentation can help.  Thank you for your time and help.
>
When I was feeling really lost, I've resorted to assigning
intermediate output from commands such as ls, search, etc, to
variables (i.e., something like "this.ls <<- ls()" from inside
you
function call, e.g., simClust) and then, e.g., from mpi.remote.exec,
looking at the value of those variables.

And, for over a year now, I've been doing most of my MPI stuff with
papply; the one nice thing of papply is that, if you have no LAM/MPI
universe, it will use a serial (not a parallel) version, so it is
much, much, much easier to debug, because you see the warnings and the
errors. So most of the frustration of things like launching something
and seeing it never return, etc, is gone.



Best,

R.
> Just so you know I'm a Computer Scientist not a Statistician, though I
> will be able to give any information about the statistics involved in
> this program.  I am reluctant to give away all source code since it is
> not my work but rather code I'm converting from standard code to
> parallelized code for a professor of mine.
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Ramon Diaz-Uriarte
Statistical Computing Team
Structural Biology and Biocomputing Programme
Spanish National Cancer Centre (CNIO)
http://ligarto.org/rdiaz