Hi all, I am quite new in the world of parallelization and I wonder if there is a way to increase the speed of creation of a parallel socket cluster. The time spend to include threads increase exponentially with the number of thread considered and I use of computer with two 8 cores CPU and thus showing a total of 32 threads in windows 7. Currently, I use the default parameters (type = "PSOCK"), but is there any fine tuning parameters that I can use to take advantage of this system ? Thanks in advance for your help ! Arnaud R version 3.0.1 (2013-05-16) Platform: x86_64-w64-mingw32/x64 (64-bit) [[alternative HTML version deleted]]
See library(help = "parallel?) On 28 Oct 2013, at 17:19, Arnaud Mosnier <a.mosnier at gmail.com> wrote:> Hi all, > > I am quite new in the world of parallelization and I wonder if there is a > way to increase the speed of creation of a parallel socket cluster. The > time spend to include threads increase exponentially with the number of > thread considered and I use of computer with two 8 cores CPU and thus > showing a total of 32 threads in windows 7. > Currently, I use the default parameters (type = "PSOCK"), but is there any > fine tuning parameters that I can use to take advantage of this system ? > > Thanks in advance for your help ! > > Arnaud > > R version 3.0.1 (2013-05-16) > Platform: x86_64-w64-mingw32/x64 (64-bit) > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.
On 28/10/2013 16:19, Arnaud Mosnier wrote:> Hi all, > > I am quite new in the world of parallelization and I wonder if there is a > way to increase the speed of creation of a parallel socket cluster. The > time spend to include threads increase exponentially with the number ofIt increases linearly in my tests (or a decent OS). But really if parallel computing is worthwhile you will be doing minutes of work on each worker process and the startup time will not be signifcant.> thread considered and I use of computer with two 8 cores CPU and thus > showing a total of 32 threads in windows 7.The first way to speed things up: use a decent OS: forking clusters is much faster.> Currently, I use the default parameters (type = "PSOCK"), but is there any > fine tuning parameters that I can use to take advantage of this system ? > > Thanks in advance for your help ! > > Arnaud > > R version 3.0.1 (2013-05-16) > Platform: x86_64-w64-mingw32/x64 (64-bit) > > [[alternative HTML version deleted]]-- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595
Thanks Brian, I thought that forking clusters was better ... but as you mentioned, it is not available on windows. Unfortunately, you do not always choose the OS used by your company ! Arnaud Date: Mon, 28 Oct 2013 17:59:10 +0000 From: Prof Brian Ripley <ripley@stats.ox.ac.uk> To: r-help@r-project.org Subject: Re: [R] speed of makeCluster (package parallel) Message-ID: <526EA5EE.9060109@stats.ox.ac.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 28/10/2013 16:19, Arnaud Mosnier wrote:> Hi all, > > I am quite new in the world of parallelization and I wonder if there is a > way to increase the speed of creation of a parallel socket cluster. The > time spend to include threads increase exponentially with the number ofIt increases linearly in my tests (or a decent OS). But really if parallel computing is worthwhile you will be doing minutes of work on each worker process and the startup time will not be signifcant.> thread considered and I use of computer with two 8 cores CPU and thus > showing a total of 32 threads in windows 7.The first way to speed things up: use a decent OS: forking clusters is much faster.> Currently, I use the default parameters (type = "PSOCK"), but is there any > fine tuning parameters that I can use to take advantage of this system ? > > Thanks in advance for your help ! > > Arnaud > > R version 3.0.1 (2013-05-16) > Platform: x86_64-w64-mingw32/x64 (64-bit) > > [[alternative HTML version deleted]]-- Brian D. Ripley, ripley@stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595 [[alternative HTML version deleted]]
Apparently Analagous Threads
- Modify objects in function
- SUGGESTION: Proposal to mitigate problem with stray processes left behind by parallel::makeCluster()
- Still have problems with tcltk in R 64 bit
- Armadillo error in R extension
- Define lower-upper bound for parameters in Optim using Nelder-Mead method