similar to: Parallelizing Other Apply Functions, e.g. by, the Easy (Wrong?) Way

dear R experts---I am looking at a fairly uninformative error in my program: Error in mclapply(1:nrow(opts), solveme) : (converted from warning) all scheduled cores encountered errors in user code the doc on ?mclapply tells me that In addition, each process is running the job inside try(..., silent=TRUE) so if error occur they will be stored as try-error objects in the list. I looked up

On 4 April 2019 at 17:28, ivo welch wrote: | The following program is whittled down from a much larger program that | always works on Intel, and always works on AMD's threadripper with | lapply but not mclappy. With mclapply on AMD, all processes go into | "suspend" mode and the program then hangs. This bug is replicable on an | AMD Ryzen Threadripper 2950X 16-Core Processor (128GB

Dear R developers: ... 7: lapply(seq_len(cores), inner.do) 8: FUN(1:3[[3]], ...) 9: sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE)) Selection: .....................Error in sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE)) : long vectors not supported yet: memory.c:3100 admittedly, my outcome will be a very big list, with 30,000 elements, each containing data frames

for the googleable r-help archives, I thought I would post what I wrote into my .Rprofile to automatically set some system information. the most relevant aspect is the determination of mc.cores. this is useful when users want to use the parallel package options(uname= system("uname", intern=TRUE)) options(os= if (getOption("uname")=="Darwin") "osx"

The following program is whittled down from a much larger program that always works on Intel, and always works on AMD's threadripper with lapply but not mclappy. With mclapply on AMD, all processes go into "suspend" mode and the program then hangs. This bug is replicable on an AMD Ryzen Threadripper 2950X 16-Core Processor (128GB RAM), running latest ubuntu 18.04. The R version

dear r experts---Is there a multicore equivalent of by(), just like mclapply() is the multicore equivalent of lapply()? if not, is there a fast way to convert a data.table into a list based on a column that lapply and mclapply can consume? advice appreciated...as always. regards, /iaw ---- Ivo Welch (ivo.welch at gmail.com)

Dear R experts--- Has anyone written parallel versions of "by" (i.e., mcby) and "ave" (i.e. mcave) ? I did ask a question like this a year ago, and then the answer was no. for those who are googling the group for the answer to this question, in the meantime, the poor man's version of "by" is mclapply( split( ds, factor ), FUN ) I don't know the poor

ISSUE: Using *forks* for parallel processing in R is not always safe. The `parallel::mclapply()` function uses forked processes to parallelize. One example where it has been confirmed that forked processing causes problems is when running R via RStudio. It is recommended to use PSOCK clusters (`parallel::makeCluster()`) rather than *forked* processes when running R from RStudio (

Dear R wizards--- I have a wrapper on mclapply() that makes it a little easier for me to do multiprocessing. (Posting this may make life easier for other googlers.) I pass a data frame, a vector that tells me what rows should be recomputed, and the function; and I get back a vector or matrix of answers. d <- data.frame( id=1:6, val=11:16 ) loc <- c(TRUE,TRUE,FALSE,TRUE,FALSE,TRUE)

On 1/10/20 7:33 AM, Henrik Bengtsson wrote: > I'd like to pick up this thread started on 2019-04-11 > (https://hypatia.math.ethz.ch/pipermail/r-devel/2019-April/077632.html). > Modulo all the other suggestions in this thread, would my proposal of > being able to disable forked processing via an option or an > environment variable make sense? I don't think R should be doing

The RStudio GUI was just one example. AFAIK, and please correct me if I'm wrong, another example is where multi-threaded code is used in forked processing and that's sometimes unstable. Yes another, which might be multi-thread related or not, is https://stat.ethz.ch/pipermail/r-devel/2018-September/076845.html: res <- parallel::mclapply(urls, function(url) { download.file(url,

On Fri, Jan 10, 2020 at 7:23 PM Simon Urbanek <simon.urbanek at r-project.org> wrote: > > Henrik, > > the example from the post works just fine in CRAN R for me - the post was about homebrew build so it's conceivably a bug in their libraries. I think it works now, because Apple switched to a different SSL library for libcurl. It usually crashes or fails on older macOS

On Fri, Jan 10, 2020 at 11:23 AM Simon Urbanek <simon.urbanek at r-project.org> wrote: > > Henrik, > > the example from the post works just fine in CRAN R for me - the post was about homebrew build so it's conceivably a bug in their libraries. Thanks for ruling that example out. > That's exactly why I was proposing a more general solution where you can simply define

dear R experts: ?apologies for all my speed and memory questions. ?I have a bet with my coauthors that I can make R reasonably efficient through R-appropriate programming techniques. this is not just for kicks, but for work. for benchmarking, my [3 year old] Mac Pro has 2.8GHz Xeons, 16GB of RAM, and R 2.13.1. right now, it seems that 'split()' is why I am losing my bet. ?(split is an

Not a direct answer but you may find lm.fit worth experimenting with. Also try the high-performance computing task view on CRAN Cheers, Andrew -- Andrew Robinson Chief Executive Officer, CEBRA and Professor of Biosecurity, School/s of BioSciences and Mathematics & Statistics University of Melbourne, VIC 3010 Australia Tel: (+61) 0403 138 955 Email: apro at unimelb.edu.au Website:

> On Jan 10, 2020, at 3:10 PM, G?bor Cs?rdi <csardi.gabor at gmail.com> wrote: > > On Fri, Jan 10, 2020 at 7:23 PM Simon Urbanek > <simon.urbanek at r-project.org> wrote: >> >> Henrik, >> >> the example from the post works just fine in CRAN R for me - the post was about homebrew build so it's conceivably a bug in their libraries. > > I

On Fri, Jan 10, 2020 at 7:23 PM Simon Urbanek <simon.urbanek at r-project.org> wrote: > > Henrik, > > the whole point and only purpose of mc* functions is to fork. That's what the multicore package was about, so if you don't want to fork, don't use mc* functions - they don't have any other purpose. But, with that same argument I'm surprised we have fake

>>>>> Andrew Robinson via R-help >>>>> on Thu, 14 Nov 2024 12:45:44 +0000 writes: > Not a direct answer but you may find lm.fit worth > experimenting with. Yes, lm.fit() is already faster, and .lm.fit() {added to base R by me, when a similar question was asked years ago ...} is even an order of magnitude faster in some cases. See

I think it's worth saying that mclapply() works as documented: it relies on forking, and so doesn't work well in environments where it's unsafe to fork. This is spelled out explicitly in the documentation of ?mclapply: It is strongly discouraged to use these functions in GUI or embedded environments, because it leads to several processes sharing the same GUI which will likely cause

Thanks, and all well taken. But are my beautiful GPUs (with integrated memory architecture) really nothing more than a cooling area for the chip? On Fri, Nov 15, 2024 at 6:06?AM Martin Maechler <maechler at stat.math.ethz.ch> wrote: > >>>>> Andrew Robinson via R-help > >>>>> on Thu, 14 Nov 2024 12:45:44 +0000 writes: > > > Not a direct