search for: mccollect

On Thu, May 2, 2019 at 7:24 PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote: > > On 5/1/19 12:25 AM, Gergely Dar?czi wrote: > > Dear All, > > > > I'm running into issues with calling mccollect on a list containing NULL > > using R 3.6 (this used to work in 3.5.3): > > > > jobs <- lapply( > > list(NULL, 'foobar'), > > function(x) mcparallel(identity(x))) > > mccollect(jobs, wait = FALSE, timeout = 0) > > #> Error in names(r...

Dear All, I'm running into issues with calling mccollect on a list containing NULL using R 3.6 (this used to work in 3.5.3): jobs <- lapply( list(NULL, 'foobar'), function(x) mcparallel(identity(x))) mccollect(jobs, wait = FALSE, timeout = 0) #> Error in names(res) <- pnames[match(s, pids)] : #> 'names' attribute [2...

On 5/3/19 3:04 PM, Gergely Dar?czi wrote: > On Thu, May 2, 2019 at 7:24 PM Tomas Kalibera <tomas.kalibera at gmail.com> wrote: >> On 5/1/19 12:25 AM, Gergely Dar?czi wrote: >>> Dear All, >>> >>> I'm running into issues with calling mccollect on a list containing NULL >>> using R 3.6 (this used to work in 3.5.3): >>> >>> jobs <- lapply( >>> list(NULL, 'foobar'), >>> function(x) mcparallel(identity(x))) >>> mccollect(jobs, wait = FALSE, timeout = 0) >>>...

I stumbled upon the following: f <- parallel::mcparallel(raw(0L)) parallel::mccollect(f) # $`77083` # NULL but f <- parallel::mcparallel(raw(1L)) parallel::mccollect(f) # Error in unserialize(r) : read error traceback() # 2: unserialize(r) # 1: parallel::mccollect(f) (restarting because the above appears to corrupt the R session) f <- parallel::mcparallel(raw(2L)) parallel...

?mccollect say 'mccollect' returns any results that are available in a list. The results will have the same order as the specified jobs. If there are multiple jobs and a job has a name it will be used to name the result, otherwise its process ID will be used. If none of the...

Dear maintainers of the parallel package, I ran into an issue with the parallel package in R-3.1.0. The following code prints the message "NULL!" quite a lot. library(parallel) for (n in 1:1000) { p <- mcparallel(sqrt(n)) res <- mccollect(p, wait=FALSE, timeout=1000) mccollect(p) if (is.null(res)) cat(n," NULL!\n") } It does not happen in R-3.0.3. (To be exact, I have checked out R-devel revision r65130 for my current tests.) I found that this behavior is introduced in revision r65143. When I insert...

On 5/1/19 12:25 AM, Gergely Dar?czi wrote: > Dear All, > > I'm running into issues with calling mccollect on a list containing NULL > using R 3.6 (this used to work in 3.5.3): > > jobs <- lapply( > list(NULL, 'foobar'), > function(x) mcparallel(identity(x))) > mccollect(jobs, wait = FALSE, timeout = 0) > #> Error in names(res) <- pnames[match(s, pids)] :...

...que id. With this you can check if the process is still running, by checking that the pid exists, and that its startup time matches. This is all very simple with the ps package, on Linux, macOS and Windows. Gabor > In base R, one can do this correctly for forked processes via > mcparallel/mccollect, not for PSOCK cluster workers which are based on > system() (and I understand it would be a useful feature) > > > j <- mcparallel(Sys.sleep(1000)) > > mccollect(j, wait=FALSE) > NULL > > # kill the child process > > > mccollect(j, wait=FALSE) > $`1542`...

...unction returns non-null, it works g <- function(x) 0 mclapply(1, g, mc.preschedule = FALSE, mc.cores = 1) # OK mclapply(1:2, g, mc.preschedule = FALSE, mc.cores = 1) # OK mclapply(1:2, g, mc.preschedule = FALSE, mc.cores = 2) # OK Digging around in mclapply(), I think it happens because mccollect(jobs) is returning an empty list. But when I use options(error=recover) and debug the function, I find that when I call mccollect(jobs) again, it returns a list with values -- it's as though mccollect() is returning too early. This will illustrate: > mclapply(1, f, mc.preschedule = FALSE, m...

Hi there, I've tried to implement an asynchronous job scheduler using parallel::mcparallel() and parallel::mccollect(..., wait=FALSE). My goal was to send processes to the background, leaving the R session open for interactive use while all jobs store their results on the file system. To keep track of the running jobs I've stored the process ids and written a little helper to not spawn new threads before alre...

Dear All, Since upgrading to 3.6.0, I've been getting a strange error messages from the child process when using mcparallel/mccollect. Before filing a report in the Bugzilla, I want to figure out whether I had been doing something wrong all this time and R 3.6.0 has exposed it, or whether something else is going on. # Background # Ultimately, what I want to do is to be able to set a time limit for an expression to be evaluated...

...euse. In principle, detecting that a worker >> process is still alive cannot be done correctly outside base R. > I am not sure why you think so. To avoid the race with PID re-use one needs access to signal handling, to blocking signals, to handling sigchld. system/system2 and mcparallel/mccollect in base R use these features and the interaction is still safe given the specific use in system/system2 and mcparallel/mccollect, yet would have to be re-visited if either of the two uses change. These features cannot be safely used outside of base R in contributed packages. Tomas > >&g...

Hi, I'd like to test whether a (localhost) PSOCK cluster node is still running or not by its PID, e.g. it may have crashed / core dumped. I'm ok with getting false-positive results due to *another* process with the same PID has since started. I can the PID of each cluster nodes by querying them for their Sys.getpid(), e.g. pids <- parallel::clusterEvalQ(cl, Sys.getpid()) Is there

...portion of the code is this: if (!mc.preschedule) { FUN <- match.fun(FUN) if (length(X) <= cores) { jobs <- lapply(seq_along(X), function(i) mcparallel(FUN(X[[i]], ...), name = names(X)[i], mc.set.seed = mc.set.seed, silent = mc.silent)) res <- mccollect(jobs) if (length(res) == length(X)) names(res) <- names(X) has.errors <- sum(sapply(res, inherits, "try-error")) } If there are 0 jobs, mccollect returns NULL, sapply on this returns the empty list, leads to an error in sum. Possible fix might be to use...

I'd like to importFrom(parallel, mccollect, mcparallel) but on Windows these are not exported because this if(tools:::.OStype() == "unix") { export(mccollect, mcparallel, mc.reset.stream, mcaffinity) } appears at src/library/parallel/NAMESPACE:6 of svn r61857. So should I be doing if (tools:::.OStype() == "unix"...

...e.? In principle, detecting that a worker process is still alive cannot be done correctly outside base R. At user-level I would probably consider some watchdog, e.g. the parallel tasks would be repeatedly touching a file. In base R, one can do this correctly for forked processes via mcparallel/mccollect, not for PSOCK cluster workers which are based on system() (and I understand it would be a useful feature) > j <- mcparallel(Sys.sleep(1000)) > mccollect(j, wait=FALSE) NULL # kill the child process > mccollect(j, wait=FALSE) $`1542` NULL More details indeed in ?mcparallel. The...

I hope that someone can help me here. in mcollective server.cfg I have following: identity = fqdn When I do mc-inventory (or any other function) the host name says: "fqdn". If i put hostname itself in it than that works, however that would mean I have to fill in that field on every server. Of course I can just create an erb template in puppet and specify hostname, but that just

.... ~/tmp >cat test1.R ## like mclapply ## should run 'forever' but terminates semi-randomly library(parallel) children <- parallel:::children while (TRUE) { n <- 8 ## n == dectectCores() jobs <- lapply(seq_len(n), function(i) mcparallel(Sys.sleep(20))) mccollect(children(jobs), FALSE) parallel:::mckill(children(jobs), tools::SIGTERM) leni <- length(mccollect(children(jobs))) message("leni: ", leni) } ~/tmp >R-dev --vanilla --slave -f test1.R leni: 6 leni: 7 leni: 7 leni: 7 leni: 7 leni: 7 leni: 7 leni: 7 leni: 8 leni: 7 leni...

...tecting that a worker > >> process is still alive cannot be done correctly outside base R. > > I am not sure why you think so. > To avoid the race with PID re-use one needs access to signal handling, > to blocking signals, to handling sigchld. system/system2 and > mcparallel/mccollect in base R use these features and the interaction is > still safe given the specific use in system/system2 and > mcparallel/mccollect, yet would have to be re-visited if either of the > two uses change. These features cannot be safely used outside of base R > in contributed packages. Ye...

...9;forever' but terminates semi-randomly >> library(parallel) >> children <- parallel:::children >> >> while (TRUE) { >> n <- 8 ## n == dectectCores() >> jobs <- lapply(seq_len(n), function(i) mcparallel(Sys.sleep(20))) >> mccollect(children(jobs), FALSE) >> parallel:::mckill(children(jobs), tools::SIGTERM) >> leni <- length(mccollect(children(jobs))) >> message("leni: ", leni) >> } >> >> ~/tmp >R-dev --vanilla --slave -f test1.R >> leni: 6 >> leni: 7...