Dear R-help, Has anyone tried running R on a Beowulf-type cluster? I can get R to run in batch (using R CMD BATCH) on a cluster, but am wondering if it is possible to get an interactive R session on a compute node. Right now, if I run: beorun --nolocal R I just get the R start-up message and back to the shell prompt. If I try bpsh 0 R I can get R started (but the R prompt does not appear) and do computations, but no access to an x11() device. I'd very much appreciate any pointer. Best, Andy Andy Liaw, PhD Biometrics Research PO Box 2000, RY33-300 Merck Research Labs Rahway, NJ 07065 andy_liaw <at> merck.com 732-594-0820
look into the following knot of CRAN packages: SNOW Rmpi Rpvm rlecuyer rsprng SNOW-FT RScaLAPACK et cetera. these provide some pretty interesting bits - like a suite of parallelized apply() functions. --elijah> Has anyone tried running R on a Beowulf-type cluster? I can get R to run in > batch (using R CMD BATCH) on a cluster, but am wondering if it is possible > to get an interactive R session on a compute node. Right now, if I run: > > beorun --nolocal R > > I just get the R start-up message and back to the shell prompt. If I try > > bpsh 0 R > > I can get R started (but the R prompt does not appear) and do computations, > but no access to an x11() device. I'd very much appreciate any pointer. > > Best, > Andy > > Andy Liaw, PhD > Biometrics Research PO Box 2000, RY33-300 > Merck Research Labs Rahway, NJ 07065 > andy_liaw <at> merck.com 732-594-0820 > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html >
Hi Elijah, Thanks for the reply. I'm aware of almost all of those things you mentioned, and have played with some of them. I'm not looking to do distributed computing within R at this point. What I'm after, though, is not really addressed by any of them. I want a user to be able to start an interactive R session on a compute node that the system determines (dynamically) to be most available at the time. That's what `beorun' does, but I couldn't get it to run R interactively. Cheers, Andy> From: elijah wright > > look into the following knot of CRAN packages: > > SNOW > Rmpi > Rpvm > rlecuyer > rsprng > SNOW-FT > RScaLAPACK > > et cetera. > > these provide some pretty interesting bits - like a suite of > parallelized > apply() functions. > > > --elijah > > > > Has anyone tried running R on a Beowulf-type cluster? I > can get R to run in > > batch (using R CMD BATCH) on a cluster, but am wondering if > it is possible > > to get an interactive R session on a compute node. Right > now, if I run: > > > > beorun --nolocal R > > > > I just get the R start-up message and back to the shell > prompt. If I try > > > > bpsh 0 R > > > > I can get R started (but the R prompt does not appear) and > do computations, > > but no access to an x11() device. I'd very much appreciate > any pointer. > > > > Best, > > Andy > > > > Andy Liaw, PhD > > Biometrics Research PO Box 2000, RY33-300 > > Merck Research Labs Rahway, NJ 07065 > > andy_liaw <at> merck.com 732-594-0820 > > > > ______________________________________________ > > R-help at stat.math.ethz.ch mailing list > > https://stat.ethz.ch/mailman/listinfo/r-help > > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html > > > >
That's a shell/R script issue. I've come across it before, but I've no longer got access to any clusters of that form (yet :-). Depending on the jobs, it's not clear to me that you want a truly interactive process on a subnode, but the idea is the same as the chicanery we used to have a remote R process. best, -tony On Mon, 7 Feb 2005 14:59:33 -0500, Liaw, Andy <andy_liaw at merck.com> wrote:> Dear R-help, > > Has anyone tried running R on a Beowulf-type cluster? I can get R to run in > batch (using R CMD BATCH) on a cluster, but am wondering if it is possible > to get an interactive R session on a compute node. Right now, if I run: > > beorun --nolocal R > > I just get the R start-up message and back to the shell prompt. If I try > > bpsh 0 R > > I can get R started (but the R prompt does not appear) and do computations, > but no access to an x11() device. I'd very much appreciate any pointer. > > Best, > Andy > > Andy Liaw, PhD > Biometrics Research PO Box 2000, RY33-300 > Merck Research Labs Rahway, NJ 07065 > andy_liaw <at> merck.com 732-594-0820 > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html >-- best, -tony "Commit early,commit often, and commit in a repository from which we can easily roll-back your mistakes" (AJR, 4Jan05). A.J. Rossini blindglobe at gmail.com
> From: Barry Rowlingson > > Liaw, Andy wrote: > > > Thanks for the reply. I'm aware of almost all of those things you > > mentioned, and have played with some of them. I'm not looking to do > > distributed computing within R at this point. What I'm > after, though, is > > not really addressed by any of them. I want a user to be > able to start an > > interactive R session on a compute node that the system determines > > (dynamically) to be most available at the time. That's > what `beorun' does, > > but I couldn't get it to run R interactively. > > > > As an alternative suggestion, have you thought of using OpenMOSIX > instead of your Beowulf tools? With an OpenMOSIX cluster your process > runs on whichever processor is best for it, and jumps around as CPU > availability within the cluster changes. If you've got a cluster with > slow and fast processors, jobs will run on the fast > processors first, no > matter where the user is logged in. > > You can also add machines to the cluster without having to install > software on them - system calls are made back on the initial > machine so > the process always sees the files it had there. Its like > plugging in a > new CPU. > > We've been running an OpenMOSIX cluster for a few years > now, and apart > from the odd problem, it works really well. Older versions of R had > trouble migrating, but anything above 1.7 quickly finds itself a nice > zippy host and uses that. > > Note this is x86 processor only, since it's a set of patches to the > linux kernel. > > BazBaz, Thanks for the suggestion. oM would have been my first choice, too (especially with Quantian). The problem is that these are dual Opterons... The Scyld Beowulf has many of the advantages that you mentioned: it's trivial to bring up diskless compute nodes, and the head node can dynamically determine which compute node has the most resource to run a job. One thing that oM does that, AFAICT Scyld Beowulf doesn't do (at least not automatically) is migrating from node to node, but one can argue whether that's necessarily a good thing (and depend on how homogeneous the nodes are). I googled a bit more on the subject, and it seems like it's just about impossible to get an interactive shell on a compute node on the Beowulf. I wonder if that also means no interactive R session... BTW, a couple of people mentioned setting DISPLAY. Yes, I did do that (both using IP and host name), but alas, no luck. Best, Andy
Hello! Sorry, may be I will just say something trivial, but:>I can get R started (but the R prompt does not appear) and do computations, >but no access to an x11() device. > >looks like you have started R with --slave switch and remote console has no DISPLAY environment variable set (or your X-server discards remote connections). Hope this help.