Thanks a lot Jim. Forwarded to Aaron (who rised the question) and the
R-help list...
Patrick
jim holtman a 侀crit :
> The size matrix you are allocating (18227 x 18227) would require 2.6GB
> of memory and that is what the error message is saying that you only
> have 0.5GB (512MB) available. You also need about 3-4x the largest
> object so that you can do any calculations on it due to extra copies
> that might be made of the data.
>
> Now your 2000 x 2000 will require about 32MB of memory that that is
> why it is comfortable in running on your system with 512MB.
>
> If you are running on a Windows machine, you may want to use the
> --max-memory command line parameter to set the memory size. I have
> 1GB and set the max to 800MB. You may also want to remove any extra
> objects and do 'gc()' before working with a large object to try and
> get memory cleared up.
>
>
> On 1/5/06, *Patrick Giraudoux* <patrick.giraudoux@univ-fcomte.fr
> <mailto:patrick.giraudoux@univ-fcomte.fr>> wrote:
>
> Dear Aaron,
>
> I am really a tool user and not a tool maker (actually an ecologist
> doing some biostatistics)... so, I take the liberty of sending a
> copy of
> this e-mail to the r-help list where capable computer persons and true
> statisticians may provide more relevant information and also to Paulo
> Ribeiro and Peter Diggle, the authors of geoR..
>
> I really feel that your huge matrix cannot be handled in R that easy,
> and I get the same kind of error as you:
>
> > m=matrix(1,ncol=18227, nrow=18227)
> Error: cannot allocate vector of size 2595496 Kb
> In addition: Warning messages:
> 1: Reached total allocation of 511Mb: see help( memory.size)
> 2: Reached total allocation of 511Mb: see help(memory.size)
>
> However, if you want to compute a distance matrix, have a look to the
> function:
>
> ?dist
>
> and try it... You will not have to create the distance matrix
> yourself
> from the coordinates file (but you may meet meory problem anyway).
> Still
> more straigthfully, if you intend to use interpolation methods such as
> kriging, you don't need to manage the distance matrix building by
> yourself. See:
>
> library(geoR)
> library(help=geoR)
> ?variog
> ?variofit
> ?likfit?
>
> etc...
>
> If you want further use geostatics (eg via geoR or gstat), you will
> anyway have to manage with memory limits (not due to R). On my
> computer
> (portable HP compaq nx7000) I can hardly manage with more than 2000
> observations using geoR (far from your 18227 observations). I had to
> krige on a dataset with 9000 observations recently and has been led to
> subsample randomly 2000 values. You can however try and increase the
> memory allocated to R on your computer. The size limit is hardware
> dependent, eg:
>
> ?memory.size
> ?memory.limit
> / memory.limit(size=500000000) /
>
> Another way may be to perform local kriging (eg kriging within the
> dataset on a fixed radius), but to my knowldege this cannot be
> done with
> geoR (unfortunately). The library gstat offers this option, but
> has much
> more limited possibilities than geoR considering other issues
> (variogram
> analysis, etc...).
>
> library(gstat)
> ?krige
>
> see argument 'maxdist'
>
> Hope this can help,
>
> Kind regards
>
>
> Patrick Giraudoux
> --
>
> Department of Environmental Biology
> EA3184 usc INRA
> University of Franche-Comte
> 25030 Besancon Cedex
> (France)
>
> tel. +33 381 665 745
> fax +33 381 665 797
> http://lbe.univ-fcomte.fr
>
>
> Aaron Swoboda a 侀crit :
>
> > Dear Sir:
> >
> >
> > I ran across your post to the R-help archive from February 9it is
> > attached below since it was nearly two years ago!). I am
> beginning to
> > learn R and am interested in analyzing some of my data in a
spatial
> > context. I am having a problem that seems similar to the problem
you
> > encountered, trying to work with a large matrix in Windows XP).
> I am
> > wondering if you could help steer me in the direction that
> helped you
> > solve your problem. I am trying to construct a distance matrix
> > containing the all of the distances between my 18,000
observations.
> > Trying to make a matrix that large gets an error message...
> >
> > m=matrix(1,ncol=18227, nrow=18227)
> >
> > Error: cannot allocate vector of size 2595496 Kb
> > In addition: Warning messages:
> > 1: Reached total allocation of 1023Mb: see help(memory.size)
> >
> > Thank you for any help you may be able to send my way.
> >
> > ~Aaron Swoboda
> >
> > Below is your posting to the R-help list...
> >
> >
> > Dear all,
> >
> > I a read with great interest the e-mails related to Arnav Sheth
> about
> > memory limitation when computing a distance matrix. I suspect
> > that I will also meet some memory limitation using GeoR. I am
> > currently running GeoR on a geodata object including 2686
> geographical
> > coordinates.
> >
> > krige.conv() can handle it (it takes 10-15 mn of computing) but
> > requests an increased memory.
> >
> > /> memory.limit(size=500000000) /
> >
> > When the computing is completed, the computer speed is
considerably
> > slowed down for any application. It is thus most necessary to
> > shut it down and restart. I will probably have to handle a set of
> > 5000-6000 coordinates in once in the next few months. I wonder if
> > it will go through it on my plateform (Windows XP and compaq
> nx7000).
> > If not, will the limitation due to R or to Windows? Does an
> > alternate solution exist?
> >
> > Thanks for any hint,
> >
> > Patrick Giraudoux
> >
> >--
> > Dr. Aaron M. Swoboda
> > 3208 Posvar Hall
> > Graduate School of Public and International Affairs
> > University of Pittsburgh
> > Pittsburgh, PA 15260
> >
> > e-mail: aswoboda@gspia.pitt.edu
<mailto:aswoboda@gspia.pitt.edu>
> > Office: 412-648-7604
> > Fax: 412-648-2605
> >
> >
>
> [[alternative HTML version deleted]]
>
>
>
> ______________________________________________
> R-help@stat.math.ethz.ch <mailto:R-help@stat.math.ethz.ch>
mailing
> list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>
>
>
>
> --
> Jim Holtman
> Cincinnati, OH
> +1 513 247 0281
>
> What the problem you are trying to solve?
[[alternative HTML version deleted]]