Samuelson, Frank*
2004-Feb-11 18:51 UTC
[R] how much memory? was: R does in memory analysis only?
Is there a way to tell how much memory the computer running R has? -Frank -----Original Message----- From: David Smith [mailto:dsmith at insightful.com] Sent: Monday, February 09, 2004 1:32 PM To: Ross Boylan Cc: r-help Subject: RE: [R] R does in memory analysis only? Ross Boylan writes:> R works only on problems that fit into (real or virtual) memory. > ... does S-Plus have the same limitation?S-PLUS, like R, does its computations in-memory. So you're limited to solving problems which can fit in the available RAM (plus available swap space). The OS may impose additional limits (e.g. 2GB of total address space on many Windows systems). However, Insightful Miner, which works with S-PLUS, does include algorithms which can process data sets out of memory. This includes the ability to perform regressions on data sets much larger than the available RAM (the only limit is the availability of disk space to store the results). You can also link S-PLUS with Insightful to perform out-of-memory calculations using S-PLUS functions. This works especially well with operations like predicting from a model, which can be performed on a row-by-row basis. I wrote a long discussion about in-memory and out-of-memory algorithms in the context of S-PLUS and Insightful Miner, which you can download from: http://www.insightful.com/support/whitepaper_download.asp # David Smith -- David M Smith <dsmith at insightful.com> Product Manager, Insightful Corp, Seattle WA Tel: +1 (206) 802 2360 Fax: +1 (206) 283 6310 New Insightful Miner 3! Discover how Pfizer, Bank of America and others are using Insightful Miner -- a highly scalable data analysis workbench. Learn more at http://www.insightful.com/products/iminer> -----Original Message----- > From: Ross Boylan [mailto:ross at biostat.ucsf.edu] > Sent: Saturday, February 07, 2004 2:16 PM > To: r-help > Subject: [R] R does in memory analysis only? > > > I wonder if someone would confirm something I'm 99% sure of from the > docs and discussion on the list, but can't find stated explicitly: > R works only on problems that fit into (real or virtual) memory. > > Thus, even if you have a problem (e.g., simple regression) > that could be > solved by doing some operation on each row of a dataset at a time, you > can't solve it unless the entire dataset and associated intermediate > results fit in memory. > > So if you're in 32 bits, your max problem size is about 2G (regular > Windows and Linux limit your process size to this, though I think some > fancy versions let you go a bit higher). > > Is there any thought of relaxing this limitation? I realize doing so > would be a big job. I also realize that 64 bits makes it much less > pressing. > > Finally, does S-Plus have the same limitation? > > Thanks. > -- > Ross Boylan wk: (415) 502-4031 > 530 Parnassus Avenue (Library) rm 115-4 ross at biostat.ucsf.edu > Dept of Epidemiology and Biostatistics fax: (415) 476-9856 > University of California, San Francisco > San Francisco, CA 94143-0840 hm: (415) 550-1062 > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide!http://www.R-project.org/posting-guide.html ______________________________________________ R-help at stat.math.ethz.ch mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Douglas Bates
2004-Feb-11 21:38 UTC
[R] how much memory? was: R does in memory analysis only?
"Samuelson, Frank*" <FWS4 at CDRH.FDA.GOV> writes:> Is there a way to tell how much memory the computer > running R has?Most Linux distributions have a program called 'free' that reports on the total amount of memory available and the amount used for different purposes. From within R you could use> system("free")total used free shared buffers cached Mem: 1552072 1462044 90028 0 176892 922176 -/+ buffers/cache: 362976 1189096 Swap: 1951888 12360 1939528 which shows that the total amount of memory on this machine is 1.5 GB, of which about 1.2 GB is available if needed. Another method in Linux (depending on your kernel) is> system("cat /proc/meminfo")total: used: free: shared: buffers: cached: Mem: 1589321728 1497116672 92205056 0 181149696 945790976 Swap: 1998733312 12656640 1986076672 MemTotal: 1552072 kB MemFree: 90044 kB MemShared: 0 kB Buffers: 176904 kB Cached: 922172 kB SwapCached: 1452 kB Active: 664200 kB Inactive: 543492 kB HighTotal: 655344 kB HighFree: 10372 kB LowTotal: 896728 kB LowFree: 79672 kB SwapTotal: 1951888 kB SwapFree: 1939528 kB