search for: mcmapply

Hello, I have a minimal example that shows a problem I'm having with parallel processing. library(parallel) mcmapply(function(x, y) { print("Running") Sys.sleep(10) }, as.list(1:10), as.list(10:1), mc.cores = 16, SIMPLIFY = FALSE) I see "Running" printed once every ten seconds. I read the documentation for mcmapply, but I don't understand why it wouldn't print all ten output state...

...to a parallelized 'mapply', which basically sets up a series of calls and then uses mclapply. It's clearly sub-optimal, as setting up the series of calls is done with a 'for' which can have a non-negligible cost. Could this be useful for the multicore package? Any suggestions? mcmapply <- function(FUN, ..., mc.preschedule = TRUE, mc.set.seed = TRUE, mc.silent = FALSE, mc.cores=1) { FUN <- match.fun(FUN) dots <- list(...) if (length(dots)==1) { ans <- mclapply(dots[[1]], FUN, mc.preschedule=mc.preschedule, mc.set.seed=mc.set.seed, mc.silent=mc.silent, mc.core...

...triz sparse en un objeto python con esta info: <138493x26744 sparse matrix of type '<type 'numpy.int64'> Quiero hacer lo mismo con R pero me he topado con la lentitud de los bucles para rellenar la sparse (también por mi supina ignorancia de los apply y de su versión paralela mcmapply; o la que sea) y tengo prisa. Me pregunto si es posible pasar más o menos directamente de python a R (nada de pasar a texto el objeto, porque eso me lleva al mismo punto de llenar la sparse en R). He probado rPython (gracias Carlos Gil por esta librería!) pero suelta este error raise TypeError...

...d my proposal of being able to disable forked processing via an option or an environment variable make sense? I've prototyped a working patch that works like: > options(fork.allowed = FALSE) > unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) [1] 14058 14058 > parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) [1] 14058 14058 > parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) [1] 14058.1 14058.2 > f <- parallel::mcparallel(Sys.getpid()) Error in allowFork(assert = TRUE) : Forked processing is not allowed per option ?fork.allowed? or environment va...

...s just some race condition somewhere? > > Tomas > >> I've prototyped a working patch that >> works like: >>> options(fork.allowed = FALSE) >>> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) >> [1] 14058 14058 >>> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) >> [1] 14058 14058 >>> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) >> [1] 14058.1 14058.2 >>> f <- parallel::mcparallel(Sys.getpid()) >> Error in allowFork(assert = TRUE) : >> Forked processing is not...

...it could be fixed? Perhaps it is just some race condition somewhere? Tomas > I've prototyped a working patch that > works like: > >> options(fork.allowed = FALSE) >> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) > [1] 14058 14058 >> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) > [1] 14058 14058 >> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) > [1] 14058.1 14058.2 >> f <- parallel::mcparallel(Sys.getpid()) > Error in allowFork(assert = TRUE) : > Forked processing is not allowed per option ?fo...

...gt;>> >>>> I've prototyped a working patch that >>>> works like: >>>>> options(fork.allowed = FALSE) >>>>> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) >>>> [1] 14058 14058 >>>>> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) >>>> [1] 14058 14058 >>>>> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) >>>> [1] 14058.1 14058.2 >>>>> f <- parallel::mcparallel(Sys.getpid()) >>>> Error in allowFork(assert = TRU...

On Mon, 15 Apr 2019 at 08:44, Tomas Kalibera <tomas.kalibera at gmail.com> wrote: > > On 4/13/19 12:05 PM, I?aki Ucar wrote: > > On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at gmail.com> wrote: > >> I think it's worth saying that mclapply() works as documented > > Mostly, yes. But it says nothing about fork's copy-on-write and memory >

...rototyped a working patch that >>>>>> works like: >>>>>>> options(fork.allowed = FALSE) >>>>>>> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) >>>>>> [1] 14058 14058 >>>>>>> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) >>>>>> [1] 14058 14058 >>>>>>> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) >>>>>> [1] 14058.1 14058.2 >>>>>>> f <- parallel::mcparallel(Sys.getpid()) >>>>...

...> > > Tomas > > > >> I've prototyped a working patch that > >> works like: > >>> options(fork.allowed = FALSE) > >>> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) > >> [1] 14058 14058 > >>> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) > >> [1] 14058 14058 > >>> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) > >> [1] 14058.1 14058.2 > >>> f <- parallel::mcparallel(Sys.getpid()) > >> Error in allowFork(assert = TRUE) : > >&...

...> I've prototyped a working patch that > >>>> works like: > >>>>> options(fork.allowed = FALSE) > >>>>> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) > >>>> [1] 14058 14058 > >>>>> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) > >>>> [1] 14058 14058 > >>>>> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) > >>>> [1] 14058.1 14058.2 > >>>>> f <- parallel::mcparallel(Sys.getpid()) > >>>> Error...

...rototyped a working patch that >>>>>> works like: >>>>>>> options(fork.allowed = FALSE) >>>>>>> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) >>>>>> [1] 14058 14058 >>>>>>> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) >>>>>> [1] 14058 14058 >>>>>>> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) >>>>>> [1] 14058.1 14058.2 >>>>>>> f <- parallel::mcparallel(Sys.getpid()) >>>>...

...> I've prototyped a working patch that > >>>> works like: > >>>>> options(fork.allowed = FALSE) > >>>>> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) > >>>> [1] 14058 14058 > >>>>> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) > >>>> [1] 14058 14058 > >>>>> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) > >>>> [1] 14058.1 14058.2 > >>>>> f <- parallel::mcparallel(Sys.getpid()) > >>>> Error...

...lt;lawrence.michael at gene.com> wrote: > >> I agree that it would fruitful to have parLapply in BiocGenerics. It looks >> to be a flexible abstraction and its presence in the parallel package >> makes >> it ubiquitous. If it hasn't been done already, mclapply (and mcmapply) >> would be good candidates, as well. The fork-based parallelism is >> substantively different in terms of the API from the more general >> parallelism of parLapply. >> >> Someone was working on some more robust and convenient wrappers around >> mclapply. Did th...

...rototyped a working patch that >>>>>> works like: >>>>>>> options(fork.allowed = FALSE) >>>>>>> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) >>>>>> [1] 14058 14058 >>>>>>> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) >>>>>> [1] 14058 14058 >>>>>>> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) >>>>>> [1] 14058.1 14058.2 >>>>>>> f <- parallel::mcparallel(Sys.getpid()) >>>>...

...that > >>>>>> works like: > >>>>>>> options(fork.allowed = FALSE) > >>>>>>> unlist(parallel::mclapply(1:2, FUN = function(x) Sys.getpid())) > >>>>>> [1] 14058 14058 > >>>>>>> parallel::mcmapply(1:2, FUN = function(x) Sys.getpid()) > >>>>>> [1] 14058 14058 > >>>>>>> parallel::pvec(1:2, FUN = function(x) Sys.getpid() + x/10) > >>>>>> [1] 14058.1 14058.2 > >>>>>>> f <- parallel::mcparallel(Sys.getpid...

...e "difftime" replacement method of units tries harder to preserve other attributes of the argument. (Wish of PR#14839.) o poly(raw = TRUE) no longer requires more unique points than the degree. (Requested by John Fox.) PACKAGE parallel: o There is a new function mcmapply(), a parallel version of mapply(), and a wrapper mcMap(), a parallel version of Map(). o A default cluster can be registered by the new function setDefaultCluster(): this will be used by default in functions such as parLapply(). o clusterMap() has a new argument .schedul...

...e "difftime" replacement method of units tries harder to preserve other attributes of the argument. (Wish of PR#14839.) o poly(raw = TRUE) no longer requires more unique points than the degree. (Requested by John Fox.) PACKAGE parallel: o There is a new function mcmapply(), a parallel version of mapply(), and a wrapper mcMap(), a parallel version of Map(). o A default cluster can be registered by the new function setDefaultCluster(): this will be used by default in functions such as parLapply(). o clusterMap() has a new argument .schedul...

...Replaced with usual weighted version. (PR#16056) * Time-series ts(*, start, end, frequency) with fractional frequency are supported more consistently; thanks to a report from Johann Kleinbub and analysis and patch by Duncan Murdoch in PR#17669. * In case of errors mcmapply() now preserves attributes of returned "try-error" objects and avoids simplification, overriding SIMPLIFY to FALSE. (PR#17653) * as.difftime() gets new optional tz = "UTC" argument which should fix behaviour during daylight-savings-changeover days, fixing...

...Replaced with usual weighted version. (PR#16056) * Time-series ts(*, start, end, frequency) with fractional frequency are supported more consistently; thanks to a report from Johann Kleinbub and analysis and patch by Duncan Murdoch in PR#17669. * In case of errors mcmapply() now preserves attributes of returned "try-error" objects and avoids simplification, overriding SIMPLIFY to FALSE. (PR#17653) * as.difftime() gets new optional tz = "UTC" argument which should fix behaviour during daylight-savings-changeover days, fixing...