Hi All,
I am reading data file ( > 1B rows) and do some date formatting like
dat=fread(mydatafile)
dat$date1 <- as.Date(ymd(dat$date1))
However, I am getting an error message saying that
Error: cons memory exhausted (limit reached?)
The script was working when the number rows were around 650M.
Is there another way to handle a big data set in R?
Thank you.
Can you tell us what is wrong with the "chunked" package which comes up when you Google "r read large file in chunks"? On November 8, 2024 4:58:18 PM PST, Val <valkremk at gmail.com> wrote:>Hi All, > >I am reading data file ( > 1B rows) and do some date formatting like > dat=fread(mydatafile) > dat$date1 <- as.Date(ymd(dat$date1)) > >However, I am getting an error message saying that > Error: cons memory exhausted (limit reached?) > >The script was working when the number rows were around 650M. > >Is there another way to handle a big data set in R? > > >Thank you. > >______________________________________________ >R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide https://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code.-- Sent from my phone. Please excuse my brevity.
Check the "high performance task view" on CRAN ... https://cran.r-project.org/web/views/HighPerformanceComputing.html On Fri, Nov 8, 2024, 7:58 PM Val <valkremk at gmail.com> wrote:> Hi All, > > I am reading data file ( > 1B rows) and do some date formatting like > dat=fread(mydatafile) > dat$date1 <- as.Date(ymd(dat$date1)) > > However, I am getting an error message saying that > Error: cons memory exhausted (limit reached?) > > The script was working when the number rows were around 650M. > > Is there another way to handle a big data set in R? > > > Thank you. > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > https://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >[[alternative HTML version deleted]]
?s 00:58 de 09/11/2024, Val escreveu:> Hi All, > > I am reading data file ( > 1B rows) and do some date formatting like > dat=fread(mydatafile) > dat$date1 <- as.Date(ymd(dat$date1)) > > However, I am getting an error message saying that > Error: cons memory exhausted (limit reached?) > > The script was working when the number rows were around 650M. > > Is there another way to handle a big data set in R? > > > Thank you. > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide https://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.Hello, fread works but as.Date(ymd(.)) does not? You probably don't need both date coercion functions, get rid of one of them and try again. dat$date1 <- ymd(dat$date1) or dat$date1 <- as.Date(dat$date1) Hope this helps, Rui Barradas -- Este e-mail foi analisado pelo software antiv?rus AVG para verificar a presen?a de v?rus. www.avg.com