Hi all, I have a dataset with more than 16800 rows and 2000 columns. I have been trying to run some econometric estimation and regressions as well as some statistical procedures on the dataset and R keeps giving me errors. I tried attaching the data and I was unable to do so. Could anyone please help me by pointing out some useful websites or resources that deals with BIG data on R and econometrics procedures. I googled and could not find anything much helpful. Thanks for your help. Paromita [[alternative HTML version deleted]]
There is a section in the High-Performance Computing (HPC) CRAN Task View on "Large memory and out-of-memory data" ( http://cran.r-project.org/web/views/HighPerformanceComputing.html) that should probably be the first place to start. On Wed, Jan 21, 2015 at 8:18 AM, Paromita Guha <paroguha at gmail.com> wrote:> Hi all, > > I have a dataset with more than 16800 rows and 2000 columns. I have been > trying to run some econometric estimation and regressions as well as some > statistical procedures on the dataset and R keeps giving me errors. > > I tried attaching the data and I was unable to do so. > > > Could anyone please help me by pointing out some useful websites or > resources that deals with BIG data on R and econometrics procedures. I > googled and could not find anything much helpful. > > Thanks for your help. > > Paromita > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- Gregory (Greg) L. Snow Ph.D. 538280 at gmail.com [[alternative HTML version deleted]]
> -----Original Message----- > From: R-help [mailto:r-help-bounces at r-project.org] On Behalf Of > Paromita Guha > Sent: Wednesday, January 21, 2015 7:19 AM > To: r-help at r-project.org > Subject: [R] Need help with BIG Data > > Hi all, > > I have a dataset with more than 16800 rows and 2000 columns. I have > been > trying to run some econometric estimation and regressions as well as > some > statistical procedures on the dataset and R keeps giving me errors. > > I tried attaching the data and I was unable to do so. > > > Could anyone please help me by pointing out some useful websites or > resources that deals with BIG data on R and econometrics procedures. I > googled and could not find anything much helpful. > > Thanks for your help. > > Paromita >Not sure I would call that big data. Can you provide an example of the code you are running and the errors that you are getting that suggests the problem is data size and not programming errors? Also, what OS and hardware do you have (Windows, Linux, ...; 32-bit, 64-bit; amount of RAM; etc.)? Dan Daniel J. Nordlund, PhD Research and Data Analysis Division Services & Enterprise Support Administration Washington State Department of Social and Health Services
On 21/01/2015 10:18 AM, Paromita Guha wrote:> Hi all, > > I have a dataset with more than 16800 rows and 2000 columns. I have been > trying to run some econometric estimation and regressions as well as some > statistical procedures on the dataset and R keeps giving me errors.That's only 33 million entries. If your computer has a couple GB of memory, that should be no problem (though it's not hard to think of models where it really would be a problem, e.g. y ~ .^2). You should post more details (and not in HTML) for more specific help. Duncan Murdoch> > I tried attaching the data and I was unable to do so. > > > Could anyone please help me by pointing out some useful websites or > resources that deals with BIG data on R and econometrics procedures. I > googled and could not find anything much helpful. > > Thanks for your help. > > Paromita > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.