R help - Sep 2005 - Error in vector("double", length) : vector size specified is too large....VLDs

I have what R seems to consider a very large dataset, a 12MB text file of
lat,long,and height values, 130,000 rows to be exact. 

Here's what I get:


Thomas Colson
North Carolina State University
Department of Forestry and Environmental Resources
(919) 673 8023
tom_colson at ncsu.edu

Calendar:
www4.ncsu.edu/~tpcolson

I have what R seems to consider a very large dataset, a 12MB text file of
lat,long,and height values, 130,000 rows to be exact. 

Here's what I get:
> data1 <-
data.frame(read.table("BE3720078500WC20020828.txt",sep=",",
header=T))
> raw.data <- as.geodata(data1)
> variog.1.b <- variog(raw.data)
variog: computing omnidirectional variogram
Error in vector("double", length) : vector size specified is too
large
> round(memory.limit()/1048576.0, 2)
[1] 4000



The "Vector size specified is too large" seems to be a common error,
but I
haven't seen any workarounds posted...and the help.archive web site seems to
be down. I can plot the dataset, do some elementary stats on it...no
variogram though. 


Any ideas on how to compute variograms on datasets with 100 to 300k points? 
Thanks 

Thomas Colson
North Carolina State University
Department of Forestry and Environmental Resources
(919) 673 8023
tom_colson at ncsu.edu

Calendar:
www4.ncsu.edu/~tpcolson

At 4 GB, I'm at the 32bit windows limit.....

Thomas Colson
North Carolina State University
Department of Forestry and Environmental Resources
(919) 673 8023
tom_colson at ncsu.edu

Calendar:
www4.ncsu.edu/~tpcolson
 

-----Original Message-----
From: Berton Gunter [mailto:gunter.berton at gene.com] 
Sent: Thursday, September 15, 2005 2:34 PM
To: 'Tom Colson'
Subject: RE: [R] Error in vector("double",length) : vector size
specified is
too large....VLDs
> 
> Any ideas on how to compute variograms on datasets with 100 to 300k 
> points?
> Thanks
Get more memory? ... it's cheap! :-)

-- Bert Gunter
Genentech

> rm(data1)
> variog.1.b <- variog(raw.data)
variog: computing omnidirectional variogram
Error in vector("double", length) : vector size specified is too large

Turns out I was wrong re: # of rows...it's 304,000


Same problem. Version is 2.1.1, hardware is Dual Xeon 3.6 4 GB RAM, XP Pro
64 Bit. Can reproduce the problem with 64Bit R 2.1.1 running on Fedora 4,
same hardware. 



Thomas Colson
North Carolina State University
Department of Forestry and Environmental Resources
(919) 673 8023
tom_colson at ncsu.edu

Calendar:
www4.ncsu.edu/~tpcolson
 

-----Original Message-----
From: Douglas Grove [mailto:dgrove at fhcrc.org] 
Sent: Thursday, September 15, 2005 2:23 PM
To: Tom Colson
Subject: Re: [R] Error in vector("double", length) : vector size
specified
is too large....VLDs

Well you could start by removing large objects that you aren't using (e.g.
'data1') and seeing if that helps. 

There may be other suggestions but you haven't told us what platform
you're
working on, as the posting guide requests:
> PLEASE do read the posting guide! 
> http://www.R-project.org/posting-guide.html
Doug


On Thu, 15 Sep 2005, Tom Colson wrote:
> I have what R seems to consider a very large dataset, a 12MB text file 
> of lat,long,and height values, 130,000 rows to be exact.
> 
> Here's what I get:
> > data1 <-
data.frame(read.table("BE3720078500WC20020828.txt",sep=",",
> header=T))
> > raw.data <- as.geodata(data1)
> > variog.1.b <- variog(raw.data)
> variog: computing omnidirectional variogram Error in
vector("double",
> length) : vector size specified is too large
> > round(memory.limit()/1048576.0, 2)
> [1] 4000
> 
> 
> 
> The "Vector size specified is too large" seems to be a common
error,
> but I haven't seen any workarounds posted...and the help.archive web 
> site seems to be down. I can plot the dataset, do some elementary 
> stats on it...no variogram though.
> 
> 
> Any ideas on how to compute variograms on datasets with 100 to 300k
points? 
> Thanks
> 
> Thomas Colson
> North Carolina State University
> Department of Forestry and Environmental Resources
> (919) 673 8023
> tom_colson at ncsu.edu
> 
> Calendar:
> www4.ncsu.edu/~tpcolson
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! 
> http://www.R-project.org/posting-guide.html
>

Tom,

please try to use the variogram function in package gstat;
it doesn't (try to) store all pairwise differences, but rather
accumulates them for distance intervals.

It will take a while to do this, and there is a chance that
you overflow the counter that keeps the number of point
pairs for each interval: 304000^2 > 2^32; it is stored as
a C long, so may work on a 64 bit architecture. Otherwise,
I'd suggest to sample your data set.

I'd be interested to hear whether you succeed (or not).
--
Edzer