similar to: R-devel Digest, Vol 149, Issue 22

Hello, I have been having issues using parallel::mclapply in a memory-efficient way and would like some guidance. I am using a 40 core machine with 96 GB of RAM. I've tried to run mclapply with 20, 30, and 40 mc.cores and it has practically brought the machine to a standstill each time to the point where I do a hard reset. When running mclapply with 10 mc.cores, I can see that each process

On 14/07/2015 6:08 PM, Radford Neal wrote: > In testing pqR on Solaris SPARC systems, I have found two bugs that > are also present in recent R Core versions. You can see the bugs and > fixes at the following URLs: > > https://github.com/radfordneal/pqR/commit/739a4960a4d8f3a3b20cfc311518369576689f37 Thanks for the report. Just one followup on this one: There are two sections

On Tue, Jul 14, 2015 at 07:52:56PM -0400, Duncan Murdoch wrote: > On 14/07/2015 6:08 PM, Radford Neal wrote: > > In testing pqR on Solaris SPARC systems, I have found two bugs that > > are also present in recent R Core versions. You can see the bugs and > > fixes at the following URLs: > > > >

On Sun, 1 Mar 2015, Radford Neal wrote: > I think the discussion of this issue has gotten more complicated than > necessary. The discussion has gotten no more complicated than it needs to be. There are other instances, such as Reduce where there is a bug report pending that amounts to the same issue. Performing surgery on expressions and calling eval is not good practice at the R level

> From: Tomas Kalibera <tomas.kalibera at gmail.com> > > Thanks for the report, I am working on a patch that will address this. > > I confirm there is a lot of potential for speedup. On my system, > > 'N=200000; x <- substring(paste(rep("A", N), collapse=""), 1:N, 1:N)' > > spends 96% time in checking if the string is ascii and 3%

Dmitriy Selivanov (selivanov.dmitriy at gmail.com) wrote: > Consider following example: > > a = matrix(1:2, nrow = 2, dimnames = list(c("row1", "row2"), c("col1"))) > a[1, ] > # 1 > > It returns *unnamed* vector `1` where I would expect named vector. In fact > it returns named vector when number of columns is > 1. > Same issue applicable

> From: "Pavel S. Ruzankin" <ruzankin at math.nsc.ru> > Let us consider the current uniform sampling without replacement > algorithm. It resides in function do_sample in > https://svn.r-project.org/R/trunk/src/main/random.c > Its complexity is obviously O(n), where the sample is selected from > 1...n, since the algorithm has to create a vector of length n. So

I have rewritten the R parser in the new version of pqR that I recently released (pqR-2015-09-14, at pqR-project.org). The new version of the parser is much cleaner, is faster (sometimes quite substantially faster), has a better interface to the read-eval-print loop, and provides a better basis for future extensions. The deparser has also been substantially revised in pqR, and is better

Well it's only a leak if you don't get the memory back after it returns, right? Anyway, one (untested by me) possibility is the copying of memory pages when the garbage collector touches objects, as pointed out by Radford Neal here: http://r.789695.n4.nabble.com/Re-R-devel-Digest-Vol-149-Issue-22-td4710367.html If so, I don't think this would be easily avoidable, but there may be

In testing pqR on Solaris SPARC systems, I have found two bugs that are also present in recent R Core versions. You can see the bugs and fixes at the following URLs: https://github.com/radfordneal/pqR/commit/739a4960a4d8f3a3b20cfc311518369576689f37 https://github.com/radfordneal/pqR/commit/339b7286c7b43dcc6b00e51515772f1d7dce7858 The first bug, in nls, is most likely to occur on a 64-bit

> On Fri, 2017-10-20 at 14:01 +0000, brodie gaslam via R-devel wrote: > > I'm thinking of this passage: > > > > > Logical values are sent as 0 (FALSE), 1 (TRUE) or INT_MIN = > > > -2147483648 (NA, but only if NAOK is true), and the compiled code > > > should return one of these three values. (Non-zero values other > > > than INT_MIN are

Along with getting pqR to work on Windows, I've also been testing it in the context of embedded R, and in the process have found some problems with the examples given of embedded R use. One problem can be seen in R_ReplDLLinit, in src/main/main.c: void R_ReplDLLinit(void) { SETJMP(R_Toplevel.cjmpbuf); R_GlobalContext = R_ToplevelContext = R_SessionContext = &R_Toplevel;

> Henrik Bengtsson: > > I'm looking for a way to get the length of an object 'x' as given by > base data type without dispatching on class. The performance improvement you're looking for is implemented in the latest version of pqR (pqR-2016-10-24, see pqR-project.org), along with corresponding improvements in several other circumstances where unclass(x) does not

I will shortly be releasing a new version of pqR (you can get a test version from pqR-project.org now - scroll to the bottom of the page). One new feature in this version requires adding a bit to the flags written out when data is serialized. I thought I'd let you know about this so as to avoid any possible conflicts. The new feature is that a few R objects are defined as constants, which

Regarding the discussion of getting length(unclass(x)) without an unclassed version of x being created... There are already no copies done for length(unclass(x)) in pqR (current version of 2017-06-09 at pqR-project.org, as well as the soon-to-be-release new version). This is part of a more general facility for avoiding copies from unclass in other circumstances as well - eg,

> From: Philippe GROSJEAN <Philippe.GROSJEAN at umons.ac.be> > > ... for latest CRAN version, we have successfully installed 4999 > packages among the 5321 CRAN package on our platform. ... It is > strange that a large portion of R CMD check errors on CRAN occur and > disappear *without any version update* of a package or any of its > direct or indirect dependencies!

Suharto, If you're interested in performance with subscripting, you might want to look at pqR (pqR-project.org). It has some substantial performance improvements for subscripting over R Core versions. This is especially true for the current development version of pqR (probably leading to a new release in about a month). You can look at a somewhat-stable snapshot of recent pqR development

Hi Radford On Fri, Aug 21, 2015 at 8:38 PM, Radford Neal <radford at cs.toronto.edu> wrote: > > I've been getting pqR to work on windows systems, and in the process > have discovered various problems with R core versions of R and with > Rtools. We happen to be working on a new version of the windows tool chain, perhaps you are interested to test if problems still exist in

> From: "Cohn, Robert S" <robert.s.cohn at intel.com> > > I am using R to multiply some large (30k x 30k double) matrices on a > 64 core machine (xeon phi). I added some timers to > src/main/array.c to see where the time is going. All of the time is > being spent in the matprod function, most of that time is spent in > dgemm. 15 seconds is in matprod in

While doing some performance testing with the new version of pqR (see pqR-project.org), I've encountered an extreme, and quite unnecessary, inefficiency in the current R Core implementation of R, which I think you might want to correct. The inefficiency is in access to columns of a data frame, as in expressions such as df$col[i], which I think are very common (the alternatives of