R help - Dec 2006 - simple parallel computing on single multicore machine

Dear List,

the advent of multicore machines in the consumer segment makes me wonder
whether it would, at least in principle, be possible to divide a
computational task into more slave R processes running on the different
cores of the same processor, more or less in the way package SNOW would
do on a cluster. I am thinking of simple 'embarassingly parallel'
problems, just like inverting 1000 matrices, estimating 1000 models or
the like.

I have seen some talk here on making R multi-threaded and the like, but
this is much simpler. I am just a curious useR, so don't bother if you
don't have time, but maybe you can point me at some resource, or just
say "this is nonsense"...

Cheers 
Giovanni

Giovanni Millo
Research Dept.,
Assicurazioni Generali SpA
Via Machiavelli 4, 
34131 Trieste (Italy)
tel. +39 040 671184 
fax  +39 040 671160
 
Ai sensi del D.Lgs. 196/2003 si precisa che le informazioni ...{{dropped}}

Millo Giovanni wrote:
> Dear List,
> 
> the advent of multicore machines in the consumer segment makes me wonder
> whether it would, at least in principle, be possible to divide a
> computational task into more slave R processes running on the different
> cores of the same processor, more or less in the way package SNOW would
> do on a cluster. I am thinking of simple 'embarassingly parallel'
> problems, just like inverting 1000 matrices, estimating 1000 models or
> the like.
> 
> I have seen some talk here on making R multi-threaded and the like, but
> this is much simpler. I am just a curious useR, so don't bother if you
> don't have time, but maybe you can point me at some resource, or just
> say "this is nonsense"...

Just use snow itself, for example.
Or on a completely other level a tuned BLAS for perallel computations 
such as ATLAS.

Uwe Ligges

> Cheers 
> Giovanni
> 
> Giovanni Millo
> Research Dept.,
> Assicurazioni Generali SpA
> Via Machiavelli 4, 
> 34131 Trieste (Italy)
> tel. +39 040 671184 
> fax  +39 040 671160
>  
> Ai sensi del D.Lgs. 196/2003 si precisa che le informazioni ...{{dropped}}
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

Millo Giovanni wrote:
> Dear List,
>
> the advent of multicore machines in the consumer segment makes me wonder
> whether it would, at least in principle, be possible to divide a
> computational task into more slave R processes running on the different
> cores of the same processor, more or less in the way package SNOW would
> do on a cluster. I am thinking of simple 'embarassingly parallel'
> problems, just like inverting 1000 matrices, estimating 1000 models or
> the like.
>
> I have seen some talk here on making R multi-threaded and the like, but
> this is much simpler. I am just a curious useR, so don't bother if you
> don't have time, but maybe you can point me at some resource, or just
> say "this is nonsense"...
>
>
>   
I don't think snow (or rather its underlying message-passing interface)
cares whether its processes are on different physical machines. So this
is easily doable. Of course you need to be aware that the processes are
competing for resources like RAM and disc.

-- 
   O__  ---- Peter Dalgaard             ?ster Farimagsgade 5, Entr.B
  c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
 (*) \(*) -- University of Copenhagen   Denmark          Ph:  (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)                  FAX: (+45) 35327907

On Friday 01 December 2006 13:23, Millo Giovanni wrote:
> Dear List,
>
> the advent of multicore machines in the consumer segment makes me wonder
> whether it would, at least in principle, be possible to divide a
> computational task into more slave R processes running on the different
> cores of the same processor, more or less in the way package SNOW would
> do on a cluster. I am thinking of simple 'embarassingly parallel'
> problems, just like inverting 1000 matrices, estimating 1000 models or
> the like.
>
> I have seen some talk here on making R multi-threaded and the like, but
> this is much simpler. I am just a curious useR, so don't bother if you
> don't have time, but maybe you can point me at some resource, or just
> say "this is nonsense"...


Dear Millo,

I find the usage of papply (from the library with the same name), which itself 
uses Rmpi to be easy and ideal for those cases. The papply documentation 
shows clearly what you need to do to pass the required arguments to papply. 
And once you have your MPI universe up and running (with whichever number of 
slaves you specify) it just works. As well, I find debugging very simple: 
just start an MPI universe with only one node, which forces papply to run 
serially (non-parallel) so wrong arguments, missing libraries, etc, are easy 
to spot.

Best,

R.




>
> Cheers
> Giovanni
>
> Giovanni Millo
> Research Dept.,
> Assicurazioni Generali SpA
> Via Machiavelli 4,
> 34131 Trieste (Italy)
> tel. +39 040 671184
> fax  +39 040 671160
>
> Ai sensi del D.Lgs. 196/2003 si precisa che le informazioni ...{{dropped}}
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html and provide commented, minimal,
> self-contained, reproducible code.
-- 
Ram?n D?az-Uriarte
Centro Nacional de Investigaciones Oncol?gicas (CNIO)
(Spanish National Cancer Center)
Melchor Fern?ndez Almagro, 3
28029 Madrid (Spain)
Fax: +-34-91-224-6972
Phone: +-34-91-224-6900

http://ligarto.org/rdiaz
PGP KeyID: 0xE89B3462
(http://ligarto.org/rdiaz/0xE89B3462.asc)



**NOTA DE CONFIDENCIALIDAD** Este correo electr?nico, y en s...{{dropped}}

Look at the nws package, I have had success using it to parallelize
simulations using a couple of computers that were not being used at the
time.  I don't have a multicore machine, but the examples in the package
make it look like using it for multicore would be even easier.

This is on windows 2000 machines with cygwin installed.

Hope this helps, 


-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at intermountainmail.org
(801) 408-8111
 

-----Original Message-----
From: r-help-bounces at stat.math.ethz.ch
[mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Millo Giovanni
Sent: Friday, December 01, 2006 5:24 AM
To: r-help at stat.math.ethz.ch
Subject: [R] simple parallel computing on single multicore machine

Dear List,

the advent of multicore machines in the consumer segment makes me wonder
whether it would, at least in principle, be possible to divide a
computational task into more slave R processes running on the different
cores of the same processor, more or less in the way package SNOW would
do on a cluster. I am thinking of simple 'embarassingly parallel'
problems, just like inverting 1000 matrices, estimating 1000 models or
the like.

I have seen some talk here on making R multi-threaded and the like, but
this is much simpler. I am just a curious useR, so don't bother if you
don't have time, but maybe you can point me at some resource, or just
say "this is nonsense"...

Cheers
Giovanni

Giovanni Millo
Research Dept.,
Assicurazioni Generali SpA
Via Machiavelli 4,
34131 Trieste (Italy)
tel. +39 040 671184
fax  +39 040 671160
 
Ai sensi del D.Lgs. 196/2003 si precisa che le informazioni
...{{dropped}}

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.