R devel - Mar 2017 - Transferring ownership of R-managed buffer

http://www.keittlab.org/

On Wed, Mar 29, 2017 at 1:04 PM, Herv? Pag?s <hpages at fredhutch.org>
wrote:
> Hi Tim,
>
> On 03/29/2017 08:24 AM, Tim Keitt wrote:
>
>> I have a use case where I would like to create an SEXP around an
existing
>> buffer that is managed by R, thus avoiding a copy operation.
>>
>
> What to you mean exactly by "an existing buffer managed by R"?
>
Not allocated from the general heap, but rather allocated using R's
allocator and therefore returnable (hypothesis!) to the R interpreter as a
vector (not external pointer) without a copy operation.

>
> If I have
>> something like:
>>
>> void *p = (void*) RAW(PROTECT(Rf_allocVector(RAWSXP, n)));
>> ... additional maniupulation ...
>> SEXP x = somefunc(SXPTYPE, n, p);  // ????
>>
>> Is there a "placement" constructor available?
>>
>
> What's a "placement" constructor?
>
Function that constructs an object using previously allocated memory (for
example provided by a 3rd party library). In some cases, responsibility for
further management of the buffer is transferred to the new object.

>
> (I have arranged for the
>> corresponding UNPROTECT.) I've looked at and experimented with
R_allocator
>> and allocVector3, but can't quite get it right. I know this is odd,
but it
>> makes sense for my use case.
>>
>
> Not sure I follow what you are trying to do. Note that an SEXP is a
> pointer to a C struct called SEXPREC. I think that trying to point an
> SEXPREC struct to data pointed to by an existing SEXPREC struct is very
> likely to lead to a disaster.
>
Possibly (although that is not what I am proposing). I'm still trying to
grok where the state resides (all in the SEXPREC or between that and the
allocator).

>
> Note that if the existing buffer managed by R is an SEXP (e.g. b) and
> your code has access to this SEXP then you don't need to create another
> SEXP around its data. Since SEXPs are pointers doing
>
>   SEXP x = b;
>
> is fine and doesn't generate any copy.
>
The buffer in this case might be resized and reallocated and the original
SEXP will have gone out of scope.

>
> And if your code only has access to a "naked" pointer to the
buffer
> managed by R (e.g. to RAW(b) is the buffer is actually in an SEXP)
> then why would you need to wrap it inside an SEXP?
>
I'm working with 3rd party structures that can work with a pointer and
buffer length, but cannot track the original SEXP. I just need to construct
an SEXP (SEXPREC) and point it at the buffer, set the vector length and
data type. I was checking to see if there is an API for that (and that wont
interfere with Rcpp declarations as it pulls in parts of Rinternals.h, but
as far as I can tell, not the parts that I need to work with SEXPRECs at a
low-level).

THK

> H.
>
>
>> Thanks for any pointers.
>>
>> THK
>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.keit
>>
tlab.org_&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWd
>> GbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=Ceu5dLU_mdGwmpIk_iUqE0dPNjqw
>>
y8wiRy6hS_lWF9k&s=h04DJujKDfqzbLz4FmP3_fZ5bYS3t7UEjSwpLrW5mL0&e>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.et
>> hz.ch_mailman_listinfo_r-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84V
>> tBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=Ceu5d
>> LU_mdGwmpIk_iUqE0dPNjqwy8wiRy6hS_lWF9k&s=WpLxtWWwjoZ7TjW6oW-
>> vxE6s3LY5kZCG1H5h0xb0Bbs&e>>
>>
> --
> Herv? Pag?s
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: hpages at fredhutch.org
> Phone:  (206) 667-5791
> Fax:    (206) 667-1319
>
	[[alternative HTML version deleted]]

Tim,

What you're describing is a special case of the ALTREP framework/API that
Luke Tierney, Tomas Kalibera, and I are working on putting into R. See my
initial proposal to the DSC here:
https://www.r-project.org/dsc/2016/slides/customvectors.html and the
subsequent branch here: https://svn.r-project.org/R/branches/ALTREP/ where
Luke and I have merged the ideas from that proposal with work he had been
pursuing independently.  The framework is implemented and passes basic
testing (make check all minus the internet checks, from what I recall).

The concept of having a vector which is a "window" into another vector
without duplication ( which I suspect is at least related to your use-case,
though I could be wrong) is a special case of one of alt-representations I
 have implemented there.

Unfortunately we were not able to get it ready in time for this upcoming
release (it is a rather sizable change to the internals), but we hope to
merge it into R-devel not too long after (there isn't a specific timeline
currently AFAIK). I know that the release after next is quite a long wait
but unfortunately that is how it played out. So anyway, in the future this
will be easy to do, and what you are trying to do may even happen
automatically.

More comments inline.

On Wed, Mar 29, 2017 at 11:36 AM, Tim Keitt <tkeitt at utexas.edu> wrote:
> http://www.keittlab.org/
>
> On Wed, Mar 29, 2017 at 1:04 PM, Herv? Pag?s <hpages at
fredhutch.org> wrote:
>
> > Hi Tim,
> >
> > On 03/29/2017 08:24 AM, Tim Keitt wrote:
> >
> >> I have a use case where I would like to create an SEXP around an
> existing
> >> buffer that is managed by R, thus avoiding a copy operation.
> >>
> >
> > What to you mean exactly by "an existing buffer managed by
R"?
> >
>
> Not allocated from the general heap, but rather allocated using R's
> allocator and therefore returnable (hypothesis!) to the R interpreter as a
> vector (not external pointer) without a copy operation.
>
> >
> > If I have
> >> something like:
> >>
> >> void *p = (void*) RAW(PROTECT(Rf_allocVector(RAWSXP, n)));
> >> ... additional maniupulation ...
> >> SEXP x = somefunc(SXPTYPE, n, p);  // ????
> >>
> >> Is there a "placement" constructor available?
> >>
> >
> > What's a "placement" constructor?
> >
>
> Function that constructs an object using previously allocated memory (for
> example provided by a 3rd party library). In some cases, responsibility for
> further management of the buffer is transferred to the new object.
>
I'm not sure what you mean by this last part, but the rest is one of the
primary use cases of the ALTREP stuff.  I don't think you can do what
you're describing now. See my comments below.

>
> >
> > (I have arranged for the
> >> corresponding UNPROTECT.) I've looked at and experimented with
> R_allocator
> >> and allocVector3, but can't quite get it right. I know this is
odd, but
> it
> >> makes sense for my use case.
> >>
> >
> > Not sure I follow what you are trying to do. Note that an SEXP is a
> > pointer to a C struct called SEXPREC. I think that trying to point an
> > SEXPREC struct to data pointed to by an existing SEXPREC struct is
very
> > likely to lead to a disaster.
> >
>
> Possibly (although that is not what I am proposing). I'm still trying
to
> grok where the state resides (all in the SEXPREC or between that and the
> allocator).
>
>
> >
> > Note that if the existing buffer managed by R is an SEXP (e.g. b) and
> > your code has access to this SEXP then you don't need to create
another
> > SEXP around its data. Since SEXPs are pointers doing
> >
> >   SEXP x = b;
> >
> > is fine and doesn't generate any copy.
> >
>
> The buffer in this case might be resized and reallocated and the original
> SEXP will have gone out of scope.
>
I'm a little unclear what you're trying to do here. Traditional atomic
vectors are a SEXP header plus a C array contiguous in memory, where the
SEXP header knows the length of the array. Note that resizing the array is
AFAIK not supported (it's not part of the blessed C api) and the internal
macros that code uses to do this now will have to go away when the ALTREP
branch is merged. That is in fact one of the major reasons we couldn't get
it prepared in time for this release.


>
>
> >
> > And if your code only has access to a "naked" pointer to the
buffer
> > managed by R (e.g. to RAW(b) is the buffer is actually in an SEXP)
> > then why would you need to wrap it inside an SEXP?
> >
>
> I'm working with 3rd party structures that can work with a pointer and
> buffer length, but cannot track the original SEXP. I just need to construct
> an SEXP (SEXPREC) and point it at the buffer, set the vector length and
> data type. I was checking to see if there is an API for that (and that wont
> interfere with Rcpp declarations as it pulls in parts of Rinternals.h, but
> as far as I can tell, not the parts that I need to work with SEXPRECs at a
> low-level).
>
Atomic vector SEXPRECs are (and must be) contiguous in memory (header plus
payload), so I don't think what you're wanting is possible without the
ALTREP framework being in place (decoupling header and payload is a major
motivating desire for us). I may misunderstand what you want though.

Best,
~G

> THK
>
>
> > H.
> >
> >
> >> Thanks for any pointers.
> >>
> >> THK
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.keit
> >>
tlab.org_&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWd
> >> GbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=Ceu5dLU_mdGwmpIk_iUqE0dPNjqw
> >>
y8wiRy6hS_lWF9k&s=h04DJujKDfqzbLz4FmP3_fZ5bYS3t7UEjSwpLrW5mL0&e>
>>
> >>         [[alternative HTML version deleted]]
> >>
> >> ______________________________________________
> >> R-devel at r-project.org mailing list
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.et
> >>
hz.ch_mailman_listinfo_r-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84V
> >>
tBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=Ceu5d
> >> LU_mdGwmpIk_iUqE0dPNjqwy8wiRy6hS_lWF9k&s=WpLxtWWwjoZ7TjW6oW-
> >> vxE6s3LY5kZCG1H5h0xb0Bbs&e> >>
> >>
> > --
> > Herv? Pag?s
> >
> > Program in Computational Biology
> > Division of Public Health Sciences
> > Fred Hutchinson Cancer Research Center
> > 1100 Fairview Ave. N, M1-B514
> > P.O. Box 19024
> > Seattle, WA 98109-1024
> >
> > E-mail: hpages at fredhutch.org
> > Phone:  (206) 667-5791
> > Fax:    (206) 667-1319
> >
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel



-- 
Gabriel Becker, PhD
Associate Scientist (Bioinformatics)
Genentech Research

	[[alternative HTML version deleted]]

On Wed, Mar 29, 2017 at 4:56 PM, Gabriel Becker <gmbecker at ucdavis.edu>
wrote:
> The concept of having a vector which is a "window" into another
vector
> without duplication ( which I suspect is at least related to your use-case,
> though I could be wrong) is a special case of one of alt-representations I
>  have implemented there.
>
Nice. That is exactly what I need. I was trying out a little experiment,
but I think its not possible right now because the length of an R vector is
identical to the buffer length. I can do everything I need except force the
length of the returned R vector to be only a portion of the allocated
storage.

THK

http://www.keittlab.org/

	[[alternative HTML version deleted]]