search for: parlapplylb

Looking at parLapplyLB, one sees that it takes in X and then passes splitList(X, length(cl)) to clusterApplyLB, which then calls dynamicClusterApply. Thus while dynamicClusterApply does handle tasks in a load-balancing fashion, sending out individual tasks as previous tasks complete, parLapplyLB preempts that by splitti...

...iocGenerics package] Hi, I am currently running a lengthy simulation study (no details necessary) on a large multi-core system. The simulated data sets are stored in a long list and they are unevenly sized (hence, the computation times vary greatly between data sets), so I naively thought that parLapplyLB() would be the right function to parallelize my computations. Presently, only two thirds of my computations are finished, however, only 16 of my 72 workers seem to be active, whereas the remaining 56 seem to run idle. I tried to find out in more detail what parLapplyLB() actually does and I wa...

Dear R-Devel List, **TL;DR:** The function **parLapplyLB** of the parallel package has [reportedly][1] (see also attached RRD output) not been doing its job, i.e. not actually balancing the load. My colleague Dirk Sarpe and I found the cause of the problem and we also have a patch to fix it (attached). A similar fix has also been provided [here][2]. [1]...

...ch works: ``` $ qacct -j 4562202 | qacct-efficiency 97.36 ``` The graph from our monitoring system is attached. As you can see, the load balancing works to a satisfying degree and the efficiency is well above 90% which was what I had hoped for :-) ## Additional Notes The list used in this jobs `parLapplyLB` is 5812 elements long. With the `splitList`-chunking from the patch, you'll get 208 lists of about 28 elements (208 chunks of size 28). The job ran on 28 CPU cores and had a wallclock time of 120351.590 seconds, i.e. 33.43 hours. Thus, the function we apply to our list takes about 580 seconds...

...acct-efficiency > 97.36 > ``` > > The graph from our monitoring system is attached. As you can see, the load balancing works to a satisfying degree and the efficiency is well above 90% which was what I had hoped for :-) > > ## Additional Notes > > The list used in this jobs `parLapplyLB` is 5812 elements long. With the `splitList`-chunking from the patch, you'll get 208 lists of about 28 elements (208 chunks of size 28). The job ran on 28 CPU cores and had a wallclock time of 120351.590 seconds, i.e. 33.43 hours. Thus, the function we apply to our list takes about 580 seconds...

Dear Christian and Henrik, thank you for spotting the problem and suggestions for a fix. We'll probably add a chunk.size argument to parLapplyLB and parLapply to follow OpenMP terminology, which has already been an inspiration for the present code (parLapply already implements static scheduling via internal function staticClusterApply, yet with a fixed chunk size; parLapplyLB already implements dynamic scheduling via internal function...

...``` >> >> The graph from our monitoring system is attached. As you can see, the >load balancing works to a satisfying degree and the efficiency is well >above 90% which was what I had hoped for :-) >> >> ## Additional Notes >> >> The list used in this jobs `parLapplyLB` is 5812 elements long. With >the `splitList`-chunking from the patch, you'll get 208 lists of about >28 elements (208 chunks of size 28). The job ran on 28 CPU cores and >had a wallclock time of 120351.590 seconds, i.e. 33.43 hours. Thus, the >function we apply to our list takes ab...

...and also to add the chunk size as an argument. If you want our input, let us know ;) Best Regards On 02/26/2018 04:01 PM, Tomas Kalibera wrote: > Dear Christian and Henrik, > > thank you for spotting the problem and suggestions for a fix. We'll probably add a chunk.size argument to parLapplyLB and parLapply to follow OpenMP terminology, which has already been an inspiration for the present code (parLapply already implements static scheduling via internal function staticClusterApply, yet with a fixed chunk size; parLapplyLB already implements dynamic scheduling via internal function dynam...

Hi, Here are the issues I found: Typos ----- (a) Found: It a parallel version of ?evalq?, "is" missing. (b) Found: 'parLapplyLB', 'parSapplyLB' are load-balancing versions, intended for use when applying ?FUN? to 'parLapplyLB' has no 'FUN' arg (more on this below). (c) Found: 'clusterApply' calls 'fun' on the first node with arguments 'seq[[1]]' and ...

.... o A default cluster can be registered by the new function setDefaultCluster(): this will be used by default in functions such as parLapply(). o clusterMap() has a new argument .scheduling to allow the use of load-balancing. o There are new load-balancing functions parLapplyLB() and parSapplyLB(). o makePSOCKCluster() has a new option useXDR = FALSE which can be used to avoid byte-shuffling for serialization when all the nodes are known to be little-endian (or all big-endian). PACKAGE INSTALLATION: o Non-ASCII vignettes without a declared e...

.... o A default cluster can be registered by the new function setDefaultCluster(): this will be used by default in functions such as parLapply(). o clusterMap() has a new argument .scheduling to allow the use of load-balancing. o There are new load-balancing functions parLapplyLB() and parSapplyLB(). o makePSOCKCluster() has a new option useXDR = FALSE which can be used to avoid byte-shuffling for serialization when all the nodes are known to be little-endian (or all big-endian). PACKAGE INSTALLATION: o Non-ASCII vignettes without a declared e...

...mes() on an S4 object derived from "environment" behaves (by default) like calling names() on an ordinary environment. * read.table() with a non-default separator now supports quotes following a non-whitespace character, matching the behavior of scan(). * parLapplyLB and parSapplyLB have been fixed to do load balancing (dynamic scheduling). This also means that results of computations depending on random number generators will now really be non-reproducible, as documented. * Indexing a list using dollar and empty string (l$"")...

...mes() on an S4 object derived from "environment" behaves (by default) like calling names() on an ordinary environment. * read.table() with a non-default separator now supports quotes following a non-whitespace character, matching the behavior of scan(). * parLapplyLB and parSapplyLB have been fixed to do load balancing (dynamic scheduling). This also means that results of computations depending on random number generators will now really be non-reproducible, as documented. * Indexing a list using dollar and empty string (l$"")...