similar to: v3 serialization of compact_intseq altrep should write modified data

Hi Gabriel, thanks for your detailed explanation, that definitely clarifies the design choices that were made in setting up the ALTREP framework and I can see how those choices make sure existing code won't break. My specific use-case for wanting to check whether a vector is an ALTREP is the following: the fst package wraps an external C++ library (fstlib, independent from R) that was made

For now you can use R_altrep_inherits(x, R_compact_intseq_class) The variable R_compact_intseq_class should currently be visible to packages on all platforms, though that may change if we eventually provide a string-based lookup mechanism, e.g. somehting like R_find_altrep_class("compact_intseq", "base") Best, luke On Tue, 4 Jun 2019, Mark Klik wrote: > Hi Gabriel,

Hi Gabriel, Thanks for your explanation, I totally understand that it is almost impossible to change the data structure of STRSXP. However, what I'm proposing is not about changing the internal representation, but rather about how we design and use the ALTREP API. I might do not state the workarounds clearly as English is not my first language. Please let me explain them again in detail.

Hello from Bioconductor, I'm developing a package to share R objects across clusters using boost library. The concept is similar to mmap package: https://cran.r-project.org/web/packages/mmap/index.html . However, I have a problem when I was trying to write Dataptr_method for the alternative string. Based on my understanding, the return value of the Dataptr_method function should be a vector

thanks for clearing that up, so these methods are actually not meant to be exported on Windows and OSX? Some of the ALTREP methods that now use 'attribute_hidden' would be very useful to packages that aim to be ALTREP aware, should the currently (exported) API be considered final? thanks for your time & best, Mark On Tue, Jun 4, 2019 at 6:52 PM Tierney, Luke <luke-tierney at

I can work on this. Thanks Luke. ~G On Wed, Jul 24, 2019 at 8:25 AM Tierney, Luke <luke-tierney at uiowa.edu> wrote: > If one of you wanted to try to create a patch to support ALTREP > generic vectors here are some notes: > > The main challenge I am aware of (there might be others): Allowing > DATAPTR to return a writable pointer would be too dangerous because > the GC

I am not able to #include "R_ext/Altrep.h" from a C++ file. I think it needs two changes: 1. add the same __cplusplus check as most of the other header files: #ifdef __cplusplus extern "C" { #endif ... #ifdef __cplusplus } #endif 2. change the line R_new_altrep(R_altrep_class_t class, SEXP data1, SEXP data2); to

While implementing R's new 'altrep' functionality in the TERR engine, I discovered a bug in R's 'deferred_string' altrep object: it is not using the correct value of the 'OutDec' option when it expands a deferred_string. See the following example: R 3.5.1: (same results in R 3.6.0 devel engine built 10/5) > options(scipen=0, OutDec=".") >

On Fri, 10 May 2019, ??? wrote: > Hi Gabriel, > > Thanks for your explanation, I totally understand that it is almost > impossible to change the data structure of STRSXP. However, what I'm > proposing is not about changing the internal representation, but rather > about how we design and use the ALTREP API. > > I might do not state the workarounds clearly as English is

Hi Jiefei, The issue here is that while the memory consequences of what you're describing may be true, this is simply how R handles character vector (what you're calling string) values internally. It doesn't actually have anything to do with ALTREP. Standard character vector SEXPs have an array of CHARSXP pointers in their payload (what is returned by DATAPTR) as well. As far as I

If one of you wanted to try to create a patch to support ALTREP generic vectors here are some notes: The main challenge I am aware of (there might be others): Allowing DATAPTR to return a writable pointer would be too dangerous because the GC write barrier needs to see all mutations. So it would be best if Dataptr and Dataptr_or_null methods were not allowed to be defined. The default methods in

Hi Mark, So depending pretty strongly on what you mean by "ALTREP aware", packages aren't necessarily supposed to be ALTREP aware. What I mean by this is that as of right now, ALTREP objects are designed to be interacted with by non-ALTREP-implementing package code, *more-or-less *exactly as standard (non-AR) SEXPs are: via the published C API. The more or less comes from the fact

Jiefei, Beyond the general response that Luke gave, to be a bit more specific to what you said, DATAPTR and INTEGER_GET_REGION involve ALTREP method execution (for ALTREP objects, obviously) so even they are not as simple and straightforward as they were a couple years ago. They should not (any longer) be thought of as being guaranteed to be essentially bare metal data retrieval from memory.

Hi Kylie, Is it a list with only numerics in it? (I only see REALSXPs there, but obviously inspect isn't showing all of them). If so, you could load it up into one big vector and then also keep partitioning information around. Bioconductor does this (see ?IRanges::CompressedList ). The potential benefit here being that the underlying large vector could then be a big out-of-memory altrep. How

Hi, This problem started as of r73472 ("Merged in the rest of the basic ALTREP framework."); I tested r73471 and it did not exist. I'm building R devel on Ubuntu 16.04.3 LTS (64-bit), with the following flags: CC="gcc -std=gnu99 -fsanitize=address -fno-omit-frame-pointer" CFLAGS="-fno-omit-frame-pointer -g -O2 -Wall -pedantic -mtune=native" and using

Hi, I am curious about whether there exist thread-safe functions in `Rinternals.h`. I know that R is single-threaded designed, but for the simple and straightforward functions like `DATAPTR` and `INTEGER_GET_REGION`, are these functions safe to call in a multi-thread environment? Best, Jiefei [[alternative HTML version deleted]]

Hi Bob, Thanks for sending around the link to that. It looks mostly right and looks like a useful onramp. There are a few things to watch out for though (I've cc'ed Romain so he's aware of these comments). @romain I hope you taake the following comments as they are intended, as help rather than attacks. The largest issue I see is that the contract for Get_region is that it *populates

Hello, It has been a while since my last R Devel build, and now I'm running into an issue I'm not sure how to isolate. After updating to the latest SVN sources, and using: ---<--------------------cut here---------------start------------------->--- tools/rsync-recommended R_PAPERSIZE=letter \ R_BATCHSAVE="--no-save --no-restore" \ R_BROWSER=xdg-open \ LIBnn=lib \

>From copyVector in duplicate.c : void copyVector(SEXP s, SEXP t) { int i, ns, nt; nt = LENGTH(t); ns = LENGTH(s); switch (TYPEOF(s)) { ... case INTSXP: for (i = 0; i < ns; i++) INTEGER(s)[i] = INTEGER(t)[i % nt]; break; ... could that be replaced with : case INTSXP: for (i=0; i<ns/nt; i++) memcpy((char *)DATAPTR(s)+i*nt*sizeof(int),

Michael, Thanks for reaching out. This was brought up by Romaine Francois offline to me as well. What he does as a workaround is #define class klass extern "C" { #include <R_ext/Altrep.h> } #undef class While we consider changing Altrep.h, the above should work for you in the immediate term. Let me know if it doesn't. ~G On Mon, Oct 8, 2018 at 4:17 PM, Michael