Hello All,
I have a 131072x132 matrix for which I need to compute a regular euclidean
distance matrix, which I then need to transform and run agnes() on this
transformed matrix. I am having trouble computing the distance matrix as it is
fairly large and I am sure I have gone over the max.
The specific error I am getting is:
Error in vector("double", length) : negative length vectors are not
allowed
I have increased the memory limit to the maximum capacity of my hard drive
(which is around 20gb), with no success.
I am running the RGUI on Windows XP with 512 mb of RAM.
Would anyone have any suggestions as to how I can overcome this problem? I would
be most grateful for any help.
Thanks,
Arnav
[[alternative HTML version deleted]]
Arnav - A suggestion I have made in the past is to run Chris Fraley and Adrian Raftery's mclust() procedure instead of Rousseuw's agnes(), if you are willing to use a different clustering method. The pdf instruction manual for the mclust package includes explicit suggestions for how to use mclust with large data sets. Naive attempts are guaranteed to fail, so DO read the manual. - tom blackwell - u michigan medical school - ann arbor - On Wed, 25 Feb 2004, Arnav Sheth wrote:> > Hello All, > > I have a 131072x132 matrix for which I need to compute a regular euclidean distance matrix, which I then need to transform and run agnes() on this transformed matrix. I am having trouble computing the distance matrix as it is fairly large and I am sure I have gone over the max. > > The specific error I am getting is: > Error in vector("double", length) : negative length vectors are not allowed > > I have increased the memory limit to the maximum capacity of my hard drive (which is around 20gb), with no success. > > I am running the RGUI on Windows XP with 512 mb of RAM. > > Would anyone have any suggestions as to how I can overcome this problem? I would be most grateful for any help. > > Thanks, > Arnav > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html >
If that's 131072 points, not 132 points, the memory needed to store that is:> 131072^2 * 8 / 1024^3[1] 128 That's in gigabytes. When you have a Windows machine that has at least that much memory, you can try to do that. Before then, try something else. Andy> From: Arnav Sheth > > > Hello All, > > I have a 131072x132 matrix for which I need to compute a > regular euclidean distance matrix, which I then need to > transform and run agnes() on this transformed matrix. I am > having trouble computing the distance matrix as it is fairly > large and I am sure I have gone over the max. > > The specific error I am getting is: > Error in vector("double", length) : negative length vectors > are not allowed > > I have increased the memory limit to the maximum capacity of > my hard drive (which is around 20gb), with no success. > > I am running the RGUI on Windows XP with 512 mb of RAM. > > Would anyone have any suggestions as to how I can overcome > this problem? I would be most grateful for any help. > > Thanks, > Arnav > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html > >------------------------------------------------------------------------------ Notice: This e-mail message, together with any attachments,...{{dropped}}
On Wed, 25 Feb 2004, Arnav Sheth wrote:> > Hello All, > > I have a 131072x132 matrix for which I need to compute a regular euclidean distance matrix, which I then need to transform and run agnes() on this transformed matrix. I am having trouble computing the distance matrix as it is fairly large and I am sure I have gone over the max. > > The specific error I am getting is: > Error in vector("double", length) : negative length vectors are not allowedWe've improved that message for the next release.> I have increased the memory limit to the maximum capacity of my hard > drive (which is around 20gb), with no success.You'd need to increase the 32-bit limit of Windows, too ....> I am running the RGUI on Windows XP with 512 mb of RAM. > > Would anyone have any suggestions as to how I can overcome this problem? > I would be most grateful for any help.No. The distance matrix is 8Gb all by itself. -- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595
> From: Joel Pitt > > Roger D. Peng wrote: > > > The problem is not with R; the problem is with Windows, as > well as the > > x86 architecture. You'll never really be able to access > more than 3GB > > of physical memory (for a single process). Accessing more than 4GB > > requires a 64 bit processor. > > > > -roger > > Alternatively, if you move to Linux you could compile a kernel with > support for large amounts memory (>4GB) which doesn't require > a 64 bit processor.I don't think so. We have a Xeon with 8GB of RAM running a custom-compiled kernel with the large memory support. The kernel does see 8GB, but a single R process cannot go beyond 3GB. OTOH our Opteron box has no such problem. Andy> Joel > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html > >------------------------------------------------------------------------------ Notice: This e-mail message, together with any attachments,...{{dropped}}