similar to: mccollect does not return named, ordered results when wait=FALSE

Hi there, I've tried to implement an asynchronous job scheduler using parallel::mcparallel() and parallel::mccollect(..., wait=FALSE). My goal was to send processes to the background, leaving the R session open for interactive use while all jobs store their results on the file system. To keep track of the running jobs I've stored the process ids and written a little helper to not spawn

On 5/3/19 3:04 PM, Gergely Dar?czi wrote: > On Thu, May 2, 2019 at 7:24 PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote: >> On 5/1/19 12:25 AM, Gergely Dar?czi wrote: >>> Dear All, >>> >>> I'm running into issues with calling mccollect on a list containing NULL >>> using R 3.6 (this used to work in 3.5.3): >>> >>> jobs

I'd like to importFrom(parallel, mccollect, mcparallel) but on Windows these are not exported because this if(tools:::.OStype() == "unix") { export(mccollect, mcparallel, mc.reset.stream, mcaffinity) } appears at src/library/parallel/NAMESPACE:6 of svn r61857. So should I be doing if (tools:::.OStype() == "unix") { importFrom(parallel, mccollect, mcparallel) }

On Thu, May 2, 2019 at 7:24 PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote: > > On 5/1/19 12:25 AM, Gergely Dar?czi wrote: > > Dear All, > > > > I'm running into issues with calling mccollect on a list containing NULL > > using R 3.6 (this used to work in 3.5.3): > > > > jobs <- lapply( > > list(NULL, 'foobar'), >

Dear All, I'm running into issues with calling mccollect on a list containing NULL using R 3.6 (this used to work in 3.5.3): jobs <- lapply( list(NULL, 'foobar'), function(x) mcparallel(identity(x))) mccollect(jobs, wait = FALSE, timeout = 0) #> Error in names(res) <- pnames[match(s, pids)] : #> 'names' attribute [2] must be the same length as the vector

On 5/1/19 12:25 AM, Gergely Dar?czi wrote: > Dear All, > > I'm running into issues with calling mccollect on a list containing NULL > using R 3.6 (this used to work in 3.5.3): > > jobs <- lapply( > list(NULL, 'foobar'), > function(x) mcparallel(identity(x))) > mccollect(jobs, wait = FALSE, timeout = 0) > #> Error in names(res) <-

I stumbled upon the following: f <- parallel::mcparallel(raw(0L)) parallel::mccollect(f) # $`77083` # NULL but f <- parallel::mcparallel(raw(1L)) parallel::mccollect(f) # Error in unserialize(r) : read error traceback() # 2: unserialize(r) # 1: parallel::mccollect(f) (restarting because the above appears to corrupt the R session) f <- parallel::mcparallel(raw(2L))

On 08/31/2018 03:13 PM, G?bor Cs?rdi wrote: > On Fri, Aug 31, 2018 at 2:51 PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote: > [...] >> kill(sig=0) is specified by POSIX but indeed as you say there is a race >> condition due to PID-reuse. In principle, detecting that a worker >> process is still alive cannot be done correctly outside base R. > I am not sure

On 08/31/2018 01:18 AM, Henrik Bengtsson wrote: > Hi, I'd like to test whether a (localhost) PSOCK cluster node is still > running or not by its PID, e.g. it may have crashed / core dumped. > I'm ok with getting false-positive results due to *another* process > with the same PID has since started. kill(sig=0) is specified by POSIX but indeed as you say there is a race

On Fri, Aug 31, 2018 at 2:51 PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote: [...] > kill(sig=0) is specified by POSIX but indeed as you say there is a race > condition due to PID-reuse. In principle, detecting that a worker > process is still alive cannot be done correctly outside base R. I am not sure why you think so. > At user-level I would probably consider some

Dear All, Since upgrading to 3.6.0, I've been getting a strange error messages from the child process when using mcparallel/mccollect. Before filing a report in the Bugzilla, I want to figure out whether I had been doing something wrong all this time and R 3.6.0 has exposed it, or whether something else is going on. # Background # Ultimately, what I want to do is to be able to set a time

On Fri, Aug 31, 2018 at 3:35 PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote: > > On 08/31/2018 03:13 PM, G?bor Cs?rdi wrote: > > On Fri, Aug 31, 2018 at 2:51 PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote: > > [...] > >> kill(sig=0) is specified by POSIX but indeed as you say there is a race > >> condition due to PID-reuse. In

Dear maintainers of the parallel package, I ran into an issue with the parallel package in R-3.1.0. The following code prints the message "NULL!" quite a lot. library(parallel) for (n in 1:1000) { p <- mcparallel(sqrt(n)) res <- mccollect(p, wait=FALSE, timeout=1000) mccollect(p) if (is.null(res)) cat(n," NULL!\n") } It does not happen in

QUESTION: Is it possible to query number of active jobs running after launching them with parallel::mcparallel()? For example, if I launch 3 jobs using: > library(parallel) > f <- lapply(1:3, FUN=mcparallel) then I can inspect them as: > str(f) List of 3 $ :List of 2 ..$ pid: int 142225 ..$ fd : int [1:2] 8 13 ..- attr(*, "class")= chr [1:3] "parallelJob"

Hi, there seem to be some (small) bugs in the mclapply function in parallel. I discovered this in the current R release version, and I checked that it is still present in R-devel. I think it only occurs in the part of the code corresponding to argument option mc.preschedule = FALSE. Here are two examples: a) library(parallel) mclapply(list(), identity, mc.preschedule=FALSE) Error in

On 20/06/2015 22:21, Henrik Bengtsson wrote: > QUESTION: > Is it possible to query number of active jobs running after launching > them with parallel::mcparallel()? > > For example, if I launch 3 jobs using: > >> library(parallel) >> f <- lapply(1:3, FUN=mcparallel) > > then I can inspect them as: > >> str(f) > List of 3 > $ :List of 2 >

On Mar 30, 2015, at 4:40 PM, Valerie Obenchain <vobencha at fredhutch.org> wrote: > On 03/25/2015 07:48 PM, Simon Urbanek wrote: >> On Mar 25, 2015, at 3:46 PM, Valerie Obenchain <vobencha at fredhutch.org> wrote: >> >>> Hi Simon, >>> >>> I'm having trouble with nested parallel workers, specifically, forking inside socket connections.

In pvec(list(1, 2), FUN, mc.cores=2) FUN sees integer() arguments whereas pvec(list(1, 2, 3), FUN, mc.cores=2) FUN sees list() arguments; the latter seems consistent with pvec's description. This came up in a complicated Bioconductor thread about generics and parallel evaluation https://stat.ethz.ch/pipermail/bioc-devel/2012-October/003745.html One relevant point is that a

Hi Simon, I'm having trouble with nested parallel workers, specifically, forking inside socket connections. When mclapply is called inside a SOCK, PSOCK or FORK worker I get an error in unserialize(). cl <- makeCluster(1, "SOCK") fun = function(i) { library(parallel) mclapply(1:2, sqrt) } Failure occurs after multiple calls to clusterApply: > clusterApply(cl, 1,

On Mar 25, 2015, at 3:46 PM, Valerie Obenchain <vobencha at fredhutch.org> wrote: > Hi Simon, > > I'm having trouble with nested parallel workers, specifically, forking inside socket connections. > You simply can't by definition - when you fork *all* the workers share the same connection inherited from the parent, so you cannot use any I/O operations that you