R devel - Mar 2005 - R_alloc with more than 2GB (PR#7721)

Full_Name: Wolfgang Huber
Version: R-devel_2005-03-10
OS: alphaev68-dec-osf4.0f
Submission from: (NULL) (62.253.128.15)


This report concerns allocation of large (>2^31 byte) chunks of memory with
R_alloc. I suspect it is a bug/typo but please don't hate me if it's
actually a
feature:

In R, I can happily create large matrices:
> a= matrix(0, nrow=191481, ncol=3063)
> dim(a)
[1] 191481   3063
> length(a)*8 / 2^30
[1] 4.369812

but when I call R_alloc in some of my C code, I get
"negative length vectors are not allowed":

    maxcp = 3063; n = 191481;
    vs = (long) maxcp * (long) n;
    mI = (double*) R_alloc(vs, sizeof(double));


I suspect this caused by allocString(int), which is called by R_alloc, see
below. Would it be possible to have allocString take a long argument as well?

These code excerpts are from R-devel_2005-03-10.tar.gz:

char *R_alloc(long nelem, int eltsize) {
 R_size_t size = nelem * eltsize;
 SEXP s = allocString(size);
 ...
}


SEXP allocString(int length) {
   return allocVector(CHARSXP, length);
}


SEXP allocVector(SEXPTYPE type, R_len_t length) {
...    
   case CHARSXP:
	size = BYTE2VEC(length + 1);
...
   malloc(sizeof(SEXPREC_ALIGN) + size * sizeof(VECREC)))
...
}

It is a feature.  Other parts of R expect a CHARSXP to have length less 
than or equal to 2^31 - 1.

Because of the use of Fortran, it is hard to see how to allow internal 
lengths (in elements, not necessarily bytes) to exceed that value.  We 
need to return to that, but it is not straightforward and last time we 
discussed it we agreed to defer it.

We can manage a better error message, but I am afraid nothing else in the 
near future.

Could you not use x = allocVector(REALSXP, vs) and REAL(x)[i]?  That will 
get you up to 2^31 - 1 elements, which is the R limit AFAIK.

On Thu, 10 Mar 2005 huber@ebi.ac.uk wrote:
> Full_Name: Wolfgang Huber
> Version: R-devel_2005-03-10
> OS: alphaev68-dec-osf4.0f
> Submission from: (NULL) (62.253.128.15)
>
>
> This report concerns allocation of large (>2^31 byte) chunks of memory
with
> R_alloc. I suspect it is a bug/typo but please don't hate me if
it's actually a
> feature:
>
> In R, I can happily create large matrices:
>
>> a= matrix(0, nrow=191481, ncol=3063)
>> dim(a)
> [1] 191481   3063
>> length(a)*8 / 2^30
> [1] 4.369812
>
> but when I call R_alloc in some of my C code, I get
> "negative length vectors are not allowed":
>
>    maxcp = 3063; n = 191481;
>    vs = (long) maxcp * (long) n;
>    mI = (double*) R_alloc(vs, sizeof(double));
>
>
> I suspect this caused by allocString(int), which is called by R_alloc, see
> below. Would it be possible to have allocString take a long argument as
well?
>
> These code excerpts are from R-devel_2005-03-10.tar.gz:
>
> char *R_alloc(long nelem, int eltsize) {
> R_size_t size = nelem * eltsize;
> SEXP s = allocString(size);
> ...
> }
>
>
> SEXP allocString(int length) {
>   return allocVector(CHARSXP, length);
> }
>
>
> SEXP allocVector(SEXPTYPE type, R_len_t length) {
> ...
>   case CHARSXP:
> 	size = BYTE2VEC(length + 1);
> ...
>   malloc(sizeof(SEXPREC_ALIGN) + size * sizeof(VECREC)))
> ...
> }
>
> ______________________________________________
> R-devel@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595