R devel - Apr 2014 - Is it possible to shrink an R object in place?

Suppose I generate an integer vector with e.g.

    SEXP iv = PROTECT(allocVector(INTSXP, 100));

and later want to shrink the object, e.g.

    shrink(iv, 50);

would simply re-set the length to 50, and allow R to reclaim the
memory that was previously used.

Is it possible to do this while respecting how R manages memory?

The motivation: there are many operations where the length of the
output is not known ahead of time, and in such cases one typically
uses a data structure that can grow efficiently. Unfortunately, IIUC
SEXPRECs cannot do this; however, an alternative possibility would
involve reserving extra memory, and then shrinking to fit after the
operation is complete.

There have been some discussions previously that defaulted to answers
of the form "you should probably just copy", e.g.
https://stat.ethz.ch/pipermail/r-devel/2008-March/048593.html, but I
wanted to ping and see if others had ideas, or if perhaps there was
code in the R sources that might be relevant.

Another reason why this is interesting is due to C++11 and
multi-threading: if I can pre-allocate SEXPs that will contain results
in the main thread, and then fill these SEXPs asynchronously (without
touching R, and hence not getting in the way of the GC or otherwise),
I can then fill these SEXPs in place and shrink-to-fit after the
computations have been completed. With C++11 support coming with R
3.1.0, functionality like this is very attractive.

The obvious alternatives are to 1) determine the length of the output
first and hence generate SEXPs of appropriate size right off the bat
(potentially expensive), and 2) fill thread-safe containers and copy
to an R object (definitely expensive).

I am probably missing something subtle (or obvious) as to why this may
not work, or be recommended, so I appreciate any comments.

Thanks,
Kevin

Kevin,
Kevin,

On Apr 10, 2014, at 4:57 PM, Kevin Ushey <kevinushey at gmail.com> wrote:
> Suppose I generate an integer vector with e.g.
> 
>    SEXP iv = PROTECT(allocVector(INTSXP, 100));
> 
> and later want to shrink the object, e.g.
> 
>    shrink(iv, 50);
> 
> would simply re-set the length to 50, and allow R to reclaim the
> memory that was previously used.
> 
> Is it possible to do this while respecting how R manages memory?
> 
The short answer is, no.

There are several problems with this, one of the main ones being that there is
simply no way to release the "excess" memory, so the vector still has
the full length in memory. There is the SETLENGTH() function, but it's not
part of the API and it has been proposed for elimination because of the inherent
issues it causes (discrepancy in allocated and reported length).

> The motivation: there are many operations where the length of the
> output is not known ahead of time, and in such cases one typically
> uses a data structure that can grow efficiently. Unfortunately, IIUC
> SEXPRECs cannot do this; however, an alternative possibility would
> involve reserving extra memory, and then shrinking to fit after the
> operation is complete.
> 
> There have been some discussions previously that defaulted to answers
> of the form "you should probably just copy", e.g.
> https://stat.ethz.ch/pipermail/r-devel/2008-March/048593.html, but I
> wanted to ping and see if others had ideas, or if perhaps there was
> code in the R sources that might be relevant.
> 
> Another reason why this is interesting is due to C++11 and
> multi-threading: if I can pre-allocate SEXPs that will contain results
> in the main thread, and then fill these SEXPs asynchronously (without
> touching R, and hence not getting in the way of the GC or otherwise),
> I can then fill these SEXPs in place and shrink-to-fit after the
> computations have been completed. With C++11 support coming with R
> 3.1.0, functionality like this is very attractive.
> 
I don't see how this is related to the question - it was always possible to
fill SEXPs from parallel threads and has been routinely used even in R itself
(most commonly via OpenMP).

> The obvious alternatives are to 1) determine the length of the output
> first and hence generate SEXPs of appropriate size right off the bat
> (potentially expensive), and 2) fill thread-safe containers and copy
> to an R object (definitely expensive).
> 
In most current OSes, it is impossible to shrink allocated memory in-place, so
if you really don't know the size of the object, it will be copied anyway.
As mentioned above, the only case where shrinking may work is if you only need
to strip a few elements of a large vector so that keeping the same allocation
has no significant effect.

Cheers,
Simon



> I am probably missing something subtle (or obvious) as to why this may
> not work, or be recommended, so I appreciate any comments.
> 
> Thanks,
> Kevin
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>