R help - Nov 2005 - cannot.allocate.memory.again and 32bit<--->64bit

hello!
------

i use 32bit.Linux(SuSe)Server, so i'm limited with 3.5Gb of memory
i demonstrate, that there is times to times a problem with allocating of 
objects of large size, for example


0.state (no objects yet created)
------------------------------------

 > gc()
                  used (Mb)    gc trigger (Mb)    max used (Mb)
Ncells   162070  4.4          350000  9.4          350000  9.4
Vcells      59921  0.5          786432  6.0          281974  2.2



1.state:  let create now a vector of large size
--------------------------------------------------

 > my.vector<-rnorm(100000*500)
 > object.size(my.vector)/1024^2
[1] 381.4698
 > 100000*500*8/1024^2   #calculate object.size directly
[1] 381.4697
 > gc()
                      used   (Mb)      gc trigger  (Mb)      max used  (Mb)
Ncells       162257     4.4             350000   9.4             
350000   9.4
Vcells   50060239 382.0      50412232 384.7     50060419 382.0



3.state:  well, let create a matrix of the same size from this vector
--------------------------------------------------------------------------

 > my.matrix<-matrix(my.vector,nrow=100000,ncol=500)
 > gc()
                     used      (Mb)       gc trigger    (Mb)     max 
used    (Mb)
Ncells       162264       4.4           350000        9.4          
350000        9.4
Vcells  100060241  763.4    150315042 1146.9   150060261 1144.9
 > object.size(my.matrix)/1024^2    #calculate object.size directly
[1] 381.4698


so, the matrix actually - according to the used.Mb - needs the same Mb 
as the vector.
but, the trigger.Mb - and i still have problems with understanding of 
this - grows ennormously.
and i can sure, i had received the "cannot allocate  the vector of  
xxxKb"-error  last time, trying the same experiment.

if we know, that the matrix (or array generally) is acctually alloccated 
as a vector (with removed dimensions), why do we need so much trigger.Mb 
for it?

is it a problem for R only on a 32bit? what is the difference with 
recpect to trigger.memory  if i use 64bit (i didn't yet)?


thanks for your advice
--------------------------

You need to understand that your process has (normally) 3GB of user 
address space, not of memory.

The `gc trigger' is the value at which an automated gc is triggered, but 
one is also triggered if there is no large enough memory block left to 
allocate.  So it is unrelated to the message about not being able to 
allocate memory.  And it does need to be bigger than the maximum actual 
use: you have had three copies of your object in use at once (and now have 
two).

The problem with a small address space is that it can easily become 
fragmented, and a 64-bit system avoids that.  Generally we would not want 
more than about 1/4 of the address space used to avoid fragmentation.

BTW, setting the dim on your vector is a much more efficient way to do this
> my.vector<-rnorm(100000*500)
> dim(my.vector) <- c(100000,500)
> gc()
            used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells   169775   4.6     350000   9.4   350000   9.4
Vcells 50063257 382.0   50415499 384.7 50064028 382.0

so perhaps it is time for you to learn some of these tricks.  If you 
study matrix() you will see where the extra copy comes from.  (Hint: it is 
more-or-less needed if byrow=TRUE.)

On Tue, 15 Nov 2005, someone with a broken shift key wrote:
> hello!
> ------
>
> i use 32bit.Linux(SuSe)Server, so i'm limited with 3.5Gb of memory
> i demonstrate, that there is times to times a problem with allocating of 
> objects of large size, for example
>
>
> 0.state (no objects yet created)
> ------------------------------------
>
>> gc()
>                 used (Mb)    gc trigger (Mb)    max used (Mb)
> Ncells   162070  4.4          350000  9.4          350000  9.4
> Vcells      59921  0.5          786432  6.0          281974  2.2
>
>
>
> 1.state:  let create now a vector of large size
> --------------------------------------------------
>
>> my.vector<-rnorm(100000*500)
>> object.size(my.vector)/1024^2
> [1] 381.4698
>> 100000*500*8/1024^2   #calculate object.size directly
> [1] 381.4697
>> gc()
>                     used   (Mb)      gc trigger  (Mb)      max used  (Mb)
> Ncells       162257     4.4             350000   9.4             350000  
9.4
> Vcells   50060239 382.0      50412232 384.7     50060419 382.0
>
>
>
> 3.state:  well, let create a matrix of the same size from this vector
> --------------------------------------------------------------------------
>
>> my.matrix<-matrix(my.vector,nrow=100000,ncol=500)
>> gc()
>                    used      (Mb)       gc trigger    (Mb)     max used 
> (Mb)
> Ncells       162264       4.4           350000        9.4          350000 
> 9.4
> Vcells  100060241  763.4    150315042 1146.9   150060261 1144.9
>> object.size(my.matrix)/1024^2    #calculate object.size directly
> [1] 381.4698
>
>
> so, the matrix actually - according to the used.Mb - needs the same Mb as
the
> vector.
> but, the trigger.Mb - and i still have problems with understanding of this
-
> grows ennormously.
> and i can sure, i had received the "cannot allocate  the vector of 
> xxxKb"-error  last time, trying the same experiment.
>
> if we know, that the matrix (or array generally) is acctually alloccated as
a
> vector (with removed dimensions), why do we need so much trigger.Mb for it?
>
> is it a problem for R only on a 32bit? what is the difference with recpect
to
> trigger.memory  if i use 64bit (i didn't yet)?
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595