I am about to write a "cluster-lite" R solution for myself. I wanted
to
know whether it already exists. If not, I will probably write up how I do
this, and I will make the code available.
Background: we have various linux and OSX systems, which are networked, but
not set up as a cluster. I have no one here to set up a cluster, so I need
a "hack" that facilitates parallel programming on standard networked
machines. I have accounts on all the machines, ssh access (of course
password-less), and networked file directory access.
what I am ultimately trying to accomplish is built around a "simple"
function, that my master program would invoke:
master.R:
multisystem( c("R slv.R 1 20 file1.out", "R slv.R 21 40
file2.out", "ssh
anotherhost R slv.R 41 80 file3.out"), announce=300)
multisystem() should submit all jobs simultaneously and continue only after
all are completed. it should also tell me every 300 seconds what jobs it is
still waiting for, and which have completed.
with basically no logic in the cluster, my master and slv programs have to
make up for it. master.R must have the smarts to know where it can spawn
jobs and how big each job should be. slv.R must have the smarts to place
its outputs into the marked files on the networked file directory. master.R
needs the smarts to combine the outputs of all jobs, and to resubmit jobs
that did not complete successfully. again, the main reason for doing all of
this is to avoid setting up a cluster across OSX and linux system, and still
to make parallel processing across linux/osx as easy as possible. I don't
think it gets much simpler than this.
now, I know how to write the multisystem() in perl, but not in R. so, if I
roll it myself, I will probably rely on a mixed R/perl system here. This is
not desirable, but it is the only way I know how to do this. if something
like multisystem() already exists in R native, please let me know and save
me from reinventing the wheel. if it does not, some perl/R combo for this
soon will.
regards,
/iaw
--
Ivo Welch (ivo.welch@brown.edu, ivo.welch@gmail.com)
[[alternative HTML version deleted]]
perhaps a long shot but maybe the package 'nws' could handle some of it. I've recently begun looking into something similar lately as well. -c ivo welch wrote:> I am about to write a "cluster-lite" R solution for myself. I wanted to > know whether it already exists. If not, I will probably write up how I do > this, and I will make the code available. > > Background: we have various linux and OSX systems, which are networked, but > not set up as a cluster. I have no one here to set up a cluster, so I need > a "hack" that facilitates parallel programming on standard networked > machines. I have accounts on all the machines, ssh access (of course > password-less), and networked file directory access. > > what I am ultimately trying to accomplish is built around a "simple" > function, that my master program would invoke: > > master.R: > multisystem( c("R slv.R 1 20 file1.out", "R slv.R 21 40 file2.out", "ssh > anotherhost R slv.R 41 80 file3.out"), announce=300) > > multisystem() should submit all jobs simultaneously and continue only after > all are completed. it should also tell me every 300 seconds what jobs it is > still waiting for, and which have completed. > > with basically no logic in the cluster, my master and slv programs have to > make up for it. master.R must have the smarts to know where it can spawn > jobs and how big each job should be. slv.R must have the smarts to place > its outputs into the marked files on the networked file directory. master.R > needs the smarts to combine the outputs of all jobs, and to resubmit jobs > that did not complete successfully. again, the main reason for doing all of > this is to avoid setting up a cluster across OSX and linux system, and still > to make parallel processing across linux/osx as easy as possible. I don't > think it gets much simpler than this. > > now, I know how to write the multisystem() in perl, but not in R. so, if I > roll it myself, I will probably rely on a mixed R/perl system here. This is > not desirable, but it is the only way I know how to do this. if something > like multisystem() already exists in R native, please let me know and save > me from reinventing the wheel. if it does not, some perl/R combo for this > soon will. > > regards, > > /iaw > > >
The R.batch package does most of this. It's been several years, but I
used it run batch jobs on multiple (30-40) machines with different
OS:es but sharing the same file system. No communication needed
between hosts. It worked like a charm and I could process 3-4 CPU
years in a few weeks for some bootstrap simulations. It got exception
handling and much more. See also r-help thread 'R.batch (Was: Re: [R]
Calling R from R and specifying "wait until script is finished")'
on
May 22, 2005:
https://www.stat.math.ethz.ch/pipermail/r-help/2005-May/071981.html
The installation is no longer as documented there, but instead you can
grab it from R-forge:
http://r-forge.r-project.org/R/?group_id=428
This very moment, the r-forge server seems to be down, so you can also
install a copy via:
source("http://www.braju.com/R/hbLite.R");
hbLite("R.batch");
Then try this:
library("R.batch");
example("JobBatch");
See the FileProgressBar class in the R.utils package for reporting
progress via the file size (0-100 bytes), which allows you to use ls
-l (or ftp remotely) to check the progress of your jobs.
Feel free to do whatever you want with it.
/Henrik
On Tue, Sep 15, 2009 at 5:01 PM, ivo welch <ivo_welch at brown.edu>
wrote:> I am about to write a "cluster-lite" R solution for myself. ?I
wanted to
> know whether it already exists. ?If not, I will probably write up how I do
> this, and I will make the code available.
>
> Background: we have various linux and OSX systems, which are networked, but
> not set up as a cluster. ?I have no one here to set up a cluster, so I need
> a "hack" that facilitates parallel programming on standard
networked
> machines. ? I have accounts on all the machines, ssh access (of course
> password-less), and networked file directory access.
>
> what I am ultimately trying to accomplish is built around a
"simple"
> function, that my master program would invoke:
>
> master.R:
> ? multisystem( c("R slv.R 1 20 file1.out", "R slv.R 21 40
file2.out", "ssh
> anotherhost R slv.R 41 80 file3.out"), announce=300)
>
> multisystem() should submit all jobs simultaneously and continue only after
> all are completed. ?it should also tell me every 300 seconds what jobs it
is
> still waiting for, and which have completed.
>
> with basically no logic in the cluster, my master and slv programs have to
> make up for it. ?master.R must have the smarts to know where it can spawn
> jobs and how big each job should be. ?slv.R must have the smarts to place
> its outputs into the marked files on the networked file directory.
?master.R
> needs the smarts to combine the outputs of all jobs, and to resubmit jobs
> that did not complete successfully. ?again, the main reason for doing all
of
> this is to avoid setting up a cluster across OSX and linux system, and
still
> to make parallel processing across linux/osx as easy as possible. ?I
don't
> think it gets much simpler than this.
>
> now, I know how to write the multisystem() in perl, but not in R. ?so, if I
> roll it myself, I will probably rely on a mixed R/perl system here. ?This
is
> not desirable, but it is the only way I know how to do this. ?if something
> like multisystem() already exists in R native, please let me know and save
> me from reinventing the wheel. ?if it does not, some perl/R combo for this
> soon will.
>
> regards,
>
> /iaw
>
>
> --
> Ivo Welch (ivo.welch at brown.edu, ivo.welch at gmail.com)
>
> ? ? ? ?[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>