similar to: Problem with mclapply -- losing output/data

Hi all, I am trying to download, decompress and read a csv file. My code: myurl <- "ftp://ftp.ncbi.nih.gov/pub/geo/DATA/supplementary/series/GSE24729/GSE24729_MitoNuclear_suppl_male_stats.csv.gz" # myfile <- "GSE24729_MitoNuclear_suppl_male_stats.csv.gz" # download.file(myurl, destfile=myfile, mode="w") # mycon <- gzcon(gzfile(myfile,

I've been using mclapply and have encountered situations where it gives errors or returns incorrect results. Here's a minimal example, which gives the error on R 2.15.2 on Mac and Linux: library(parallel) f <- function(x) NULL mclapply(1, f, mc.preschedule = FALSE, mc.cores = 1) # Error in sum(sapply(res, inherits, "try-error")) : # invalid 'type' (list) of argument

Dear R-devel friends: I'm back to bother you again about the conflict between mclapply and tcltk. I've been monitoring several packages that want to use mclapply to parallelize computations and need to figure out what should be done. It appears tcltk cannot be safely unloaded, so the best we can do is check for the presence of tcltk and stop if it is found before mclapply() is used. I

Dear R-devel, I am running mclapply with many iterations over a function that modifies nothing and makes no copies of anything. It is taking up a lot of memory, so it seems to me like this is a bug. Should I post this to bugs.r-project.org? A minimal reproducible example can be obtained by first starting a memory monitoring program such as htop, and then executing the following code while

Hi Inaki, > "Performant"... in terms of what. If the cost of copying the data > predominates over the computation time, maybe you didn't need > parallelization in the first place. Performant in terms of speed. There's no copying in that example using `mclapply` and so it is significantly faster than other alternatives. It is a very simple and contrived example, but

> On Feb 24, 2015, at 10:50 AM, <luke-tierney at uiowa.edu> wrote: > > The documentation is not specific enough on the indented semantics in > this situation to consider this a bug. The original R-level > implementation of lapply was > > lapply <- function(X, FUN, ...) { > FUN <- match.fun(FUN) > if (!is.list(X)) > X <-

Yes I am running on Rstudio 1.2.5033. I was also running this code without error on Ubuntu in Rstudio. Checking again on the terminal and it does indeed work fine even with large data.frames. Any idea as to what interaction between Rstudio and mclapply causes this? Thanks, Shian On 28 Apr 2020, at 7:29 pm, Simon Urbanek <simon.urbanek at R-project.org<mailto:simon.urbanek at

The following program is whittled down from a much larger program that always works on Intel, and always works on AMD's threadripper with lapply but not mclappy. With mclapply on AMD, all processes go into "suspend" mode and the program then hangs. This bug is replicable on an AMD Ryzen Threadripper 2950X 16-Core Processor (128GB RAM), running latest ubuntu 18.04. The R version

Just throwing my two cents in: I think removing/deprecating fork would be a bad idea for two reasons: 1) There are no performant alternatives 2) Removing fork would break existing workflows Even if replaced with something using the same interface (e.g., a function that automatically detects variables to export as in the amazing `future` package), the lack of copy-on-write functionality would

Below is my full implementation (tried to make it simple as for demonstration) Lapply_me = function(X = X, FUN = FUN, Apply_MC = FALSE, ...) { if (Apply_MC) { return(mclapply(X, FUN, ...)) } else { if (any(names(list(...)) == 'mc.cores')) { myList = list(...)[!names(list(...)) %in% 'mc.cores'] } return(lapply(X, FUN, myList)) } } Lapply_me(as.list(1:4), function(xx) { if (xx ==

Hello, I have been having issues using parallel::mclapply in a memory-efficient way and would like some guidance. I am using a 40 core machine with 96 GB of RAM. I've tried to run mclapply with 20, 30, and 40 mc.cores and it has practically brought the machine to a standstill each time to the point where I do a hard reset. When running mclapply with 10 mc.cores, I can see that each process

> On Feb 25, 2015, at 5:35 PM, Benjamin Tyner <btyner at gmail.com> wrote: > > Actually, it depends on the number of cores: Under current semantics, yes. Each 'stream' of function calls is lazily capturing the last value of `i` on that core. Under Luke's proposed semantics (IIUC), the result would be the same (2,4,6,8) for both parallel and serial execution. This is

Dear members, I am using arfima in an mclapply construction (from the parallel package): Browse[2]> LYG <- mclapply(LYGH, FUN = arfima, mc.cores = detectCores()) ^C Browse[2]> LYG <- mclapply(LYGH[1:10], FUN = arfima, mc.cores = detectCores()) ^C Browse[2]> LYG <- mclapply(LYGH[1:2], FUN = arfima, mc.cores = detectCores()) ^C You can see that I am

Thanks Henrik, That clears things up significantly. I did see the warning but failed to include it my initial email. It sounds like an RStudio issue, and it seems like that it?s quite intrinsic to how forks interact with RStudio. Given this code is eventually going to be a part of a package, should I expect it to fail mysteriously in RStudio for my users? Is the best solution here to migrate all

dear r experts---Is there a multicore equivalent of by(), just like mclapply() is the multicore equivalent of lapply()? if not, is there a fast way to convert a data.table into a list based on a column that lapply and mclapply can consume? advice appreciated...as always. regards, /iaw ---- Ivo Welch (ivo.welch at gmail.com)

Thanks Simon, I will take note of the sensible default for core usage. I?m trying to achieve small scale parallelism, where tasks take 1-5 seconds and make fuller use of consumer hardware. Its not a HPC-worthy computation but even laptops these days come with 4 cores and I don?t see a reason to not make use of it. The goal for the current piece of code I?m working on is to bootstrap many

On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at gmail.com> wrote: > > I think it's worth saying that mclapply() works as documented Mostly, yes. But it says nothing about fork's copy-on-write and memory overcommitment, and that this means that it may work nicely or fail spectacularly depending on whether, e.g., you operate on a long vector. -- I?aki ?car

Hi, I wondered about the behavior described in the following stackoverflow question: https://stackoverflow.com/questions/20674538/mclapply-returns-null-randomly More specifically, I would like to know if you ever considered the suggestion made in the comments of the first answer, namely to somehow warn the user if one of the processes has been killed by the out-of-memory killer ? I am always

I have this function and want to run it parallel with different sets of data. Using SNOW and clusterApplyLB. system.time(out <- mclapply(cData, plotGraph)) #each cData contains 100X6000 doubles system.time(out <- mclapply(cData2, plotGraph)) system.time(out <- mclapply(cData3, plotGraph)) system.time(out <- mclapply(cData4, plotGraph)) system.time(out <- mclapply(cData5,

The reason that it works for Apply_MC=TRUE is that in that case you call mclapply(X,FUN,...) and the mclapply() function strips off the mc.cores argument from the "..." list before calling FUN, so FUN is being called with zero arguments, exactly as it is declared. A quick workaround is to change the line Lapply_me(as.list(1:4), function(xx) { to Lapply_me(as.list(1:4),