similar to: does parLapplyLB do load-balancing?

Dear R-Devel List, **TL;DR:** The function **parLapplyLB** of the parallel package has [reportedly][1] (see also attached RRD output) not been doing its job, i.e. not actually balancing the load. My colleague Dirk Sarpe and I found the cause of the problem and we also have a patch to fix it (attached). A similar fix has also been provided [here][2]. [1]:

Dear Christian and Henrik, thank you for spotting the problem and suggestions for a fix. We'll probably add a chunk.size argument to parLapplyLB and parLapply to follow OpenMP terminology, which has already been an inspiration for the present code (parLapply already implements static scheduling via internal function staticClusterApply, yet with a fixed chunk size; parLapplyLB already

Hi, I'm trying to understand the rationale for your proposed amount of splitting and more precisely why that one is THE one. If I put labels on your example numbers in one of your previous post: nbrOfElements <- 97 nbrOfWorkers <- 5 With these, there are two extremes in how you can split up the processing in chunks such that all workers are utilized: (A) Each worker, called

Dear R-Devel List, I have installed R 3.4.3 with the patch applied on our cluster and ran a *real-world* job of one of our users to confirm that the patch works to my satisfaction. Here are the results. The original was a series of jobs, all essentially doing the same stuff using bootstrapped data, so for the original there is more data and I show the arithmetic mean with standard deviation. The

Dear Tomas, Thanks for your commitment to fix this issue and also to add the chunk size as an argument. If you want our input, let us know ;) Best Regards On 02/26/2018 04:01 PM, Tomas Kalibera wrote: > Dear Christian and Henrik, > > thank you for spotting the problem and suggestions for a fix. We'll probably add a chunk.size argument to parLapplyLB and parLapply to follow OpenMP

Dear Henrik, The rationale is just that it is within these extremes and that it is really simple to calculate, without making any assumptions and knowing that it won't be perfect. The extremes A and B you are mentioning are special cases based on assumptions. Case A is based on the assumption that the function has a long runtime or varying runtime, then you are likely to get the best load

Hi, I'm trying to follow the R-admin instructions for using MKL10 as the external BLAS compiling R-2.6.2 under Linux on a RH EL head node of a cluster. The configure process seems to have problems when it checks for dgemm in the BLAS. I'm using configure as: ./configure CC=icc F77=ifort --with-lapack="$MKL" --with-blas="$MKL" where $MKL is defined as in R-admin

[cross-posted on R-devel and Bioc-devel, since the functions from the parallel package discussed here are mirrored in the BiocGenerics package] Hi, I am currently running a lengthy simulation study (no details necessary) on a large multi-core system. The simulated data sets are stored in a long list and they are unevenly sized (hence, the computation times vary greatly between data sets), so

When I run the gam function as part of an optimization and do the optimization many times using a loop, I'm finding that memory use increases over time (based on simply monitoring top). Below is some example code that involves varying the penalty parameter in gam, trying to find the value that gives exactly 50 edf for a simple smoothing problem. I thought I would post to the list to see if

Greetings, I am having an interesting problem and I wonder if anyone else has seen this behavior. I am running R 2.11.1 with SNOW 0.3-3 on a Dell cluster running CentOS 5.5. I create my cluster using: cluster<- makeCluster(nodes,type="SOCK",port=10191) # nodes is a vector of compute nodes I then wrap a loop around clusterApplyLB to evaluate my function multiple times, with

Hi, Here are the issues I found: Typos ----- (a) Found: It a parallel version of ?evalq?, "is" missing. (b) Found: 'parLapplyLB', 'parSapplyLB' are load-balancing versions, intended for use when applying ?FUN? to 'parLapplyLB' has no 'FUN' arg (more on this below). (c) Found: 'clusterApply' calls 'fun' on the first

Hello R-devel, A function to be run inside lapply() or one of its friends is trivial to augment with side effects to show a progress bar. When the code is intended to be run on a 'parallel' cluster, it generally cannot rely on its own side effects to report progress. I've found three approaches to progress bars for parallel processes on CRAN: - Importing 'snow' (not

Hello, I?d like to compare two strings internally the way R would, so I need Scollate which is not part of the authorized R api. So: - Can Scollate (and perhaps Seql) be promoted to api ? - If not what are the alternatives ? Using strcmp or stroll does not seem as general as Scollate. Romain PS: Here is some context: https://github.com/hadley/dplyr/issues/325

NIMBLE version 0.6-2 has been released on CRAN and at r-nimble.org. NIMBLE is a system that allows you to: - Write general hierarchical statistical models in BUGS code and create a corresponding model object to use in R. - Build Markov chain Monte Carlo (MCMC), particle filters, Monte Carlo Expectation Maximization (MCEM), or write generic algorithms that can be applied to any model. -

NIMBLE version 0.6-2 has been released on CRAN and at r-nimble.org. NIMBLE is a system that allows you to: - Write general hierarchical statistical models in BUGS code and create a corresponding model object to use in R. - Build Markov chain Monte Carlo (MCMC), particle filters, Monte Carlo Expectation Maximization (MCEM), or write generic algorithms that can be applied to any model. -

Hello, thanks for bringing this topic up, and it would be excellent if we could come of with a generic solution for this in base R. It is one of the top frequently asked questions and requested features in parallel processing, but also in sequential processing. We have also seen lots of variants on how to attack the problem of reporting on progress when running in parallel. As the author

The build system rolled up R-2.15.0.tar.gz (codename "Easter Beagle") at 9:00 this morning. This is the first release of the 2.15 series and contains several new features and changes; see the list below for details. You can get the source code from http://cran.r-project.org/src/base/R-2/R-2.15.0.tar.gz or wait for it to be mirrored at a CRAN site nearer to you. Binaries for various

The build system rolled up R-2.15.0.tar.gz (codename "Easter Beagle") at 9:00 this morning. This is the first release of the 2.15 series and contains several new features and changes; see the list below for details. You can get the source code from http://cran.r-project.org/src/base/R-2/R-2.15.0.tar.gz or wait for it to be mirrored at a CRAN site nearer to you. Binaries for various

The build system rolled up R-3.5.0.tar.gz (codename "Joy in Playing") this morning. The list below details the changes in this release. You can get the source code from http://cran.r-project.org/src/base/R-3/R-3.5.0.tar.gz or wait for it to be mirrored at a CRAN site nearer to you. Binaries for various platforms will appear in due course. For the R Core Team, Peter Dalgaard

The build system rolled up R-3.5.0.tar.gz (codename "Joy in Playing") this morning. The list below details the changes in this release. You can get the source code from http://cran.r-project.org/src/base/R-3/R-3.5.0.tar.gz or wait for it to be mirrored at a CRAN site nearer to you. Binaries for various platforms will appear in due course. For the R Core Team, Peter Dalgaard