similar to: Limit

Check the "high performance task view" on CRAN ... https://cran.r-project.org/web/views/HighPerformanceComputing.html On Fri, Nov 8, 2024, 7:58 PM Val <valkremk at gmail.com> wrote: > Hi All, > > I am reading data file ( > 1B rows) and do some date formatting like > dat=fread(mydatafile) > dat$date1 <- as.Date(ymd(dat$date1)) > > However, I

Can you tell us what is wrong with the "chunked" package which comes up when you Google "r read large file in chunks"? On November 8, 2024 4:58:18 PM PST, Val <valkremk at gmail.com> wrote: >Hi All, > >I am reading data file ( > 1B rows) and do some date formatting like > dat=fread(mydatafile) > dat$date1 <- as.Date(ymd(dat$date1)) >

The data was read. The problem is with processing. On Fri, Nov 8, 2024 at 7:30?PM Bert Gunter <bgunter.4567 at gmail.com> wrote: > > Is the problem reading the file in or processing it after it has been read in? > > Bert > > On Fri, Nov 8, 2024 at 5:13?PM Jeff Newmiller via R-help <r-help at r-project.org> wrote: >> >> Can you tell us what is wrong with

Is the problem reading the file in or processing it after it has been read in? Bert On Fri, Nov 8, 2024 at 5:13?PM Jeff Newmiller via R-help < r-help at r-project.org> wrote: > Can you tell us what is wrong with the "chunked" package which comes up > when you Google "r read large file in chunks"? > > On November 8, 2024 4:58:18 PM PST, Val <valkremk at

Then you don't have enough memory to process the whole thing at once. Not unlike stuffing your mouth with cookies and not being able to chew for lack of space to move the food around in your mouth. Now, can you answer my question? On November 8, 2024 5:38:37 PM PST, Val <valkremk at gmail.com> wrote: >The data was read. The problem is with processing. > >On Fri, Nov 8, 2024

Hi Jeff, Memory was not an issue. The system only used 75% of the memory allocated for the job. I am trying to understand what "r read large file in chunks" is doing. On Fri, Nov 8, 2024 at 7:50?PM Jeff Newmiller <jdnewmil at dcn.davis.ca.us> wrote: > > Then you don't have enough memory to process the whole thing at once. Not unlike stuffing your mouth with cookies

There is always an implied "and do computations on it before writing the processed data out" when reading chunks of a file. And you would almost certainly not be getting that error if you were not out of memory. A good rule of thumb is that you need 4 times as much free memory to process data than you need to read it in. On November 8, 2024 6:08:16 PM PST, Val <valkremk at

Hi All, I have an script in R which accepts user inputs for certain parameters, particularly dates, which the user inputs as character strings. eg: > date1 <- "2009-01-21" The script later parses the input via the as.Date function: > as.Date(date1) However, as.Date encounters an error when the string does not represent an actual date. eg: > date1 <-

Hi, I wanted to learn how to solve a date and time manipulation where i can do the following two 1. difference of two dates eg (differnce between 5th jan 2013 and 1st jan 2013) 2.Suppose i have week number of the year, i want to know if i can find out the day it refers to eg( say week 2 of 2013 would be 6th jan 2013 and the day is sunday) i need my result to tell me that its

Hi Community, I have a question about how to pass command line parameters to R script running in the batch mode. The problem is: there is a banch of data files which are to be processed by R script called from a web-server, i.e. in the batch mode. The web server generates data files and passes their names calling 'R CMD BATCH' one by one for every file. Now the question is how to

Hi all, I'm having a little bit of trouble with some date conversions and am hoping someone can help me out. Thanks in advance. OK, I have two sources of data that provide date info in a csv file differently. I've attached a small zipped file with two text files that illustrate both. (Is it ok to send attachments to this list? Not sure. It's very small.) I need to be able to

Hi everybody, I am beginer in R and I need your precious help. I want to create a small function in R as in sas to retrieve date. I have a file with data that import in R. DATE PAYS nb_pays.ILI. 1 24/04/2009 usa 0 2 24/04/2009 usa 0 3 24/04/2009 Mexique 0 4 24/04/2009

Hi, I have a data.frame like this: y <- rnorm(60) lev <- gl(3,20, labels=paste("lev", 1:3, sep="")) date1 <- as.Date(seq(ISOdate(2007,9,1), ISOdate(2007,11,5), by=60*60*24)) date1 <- date1[-c(3,4,15,34,38,40)] df <- data.frame(lev=lev, date1=date1, y=y) I would like to produce a new data.frame with missing days in df$date1 in each df$lev, like this: lev

I am trying to calculate the median of each row of a data frame where the data frame consist of columns of class Date. Below are my test data and best attempt at using apply. I didn't see a solution via Google or the Baron search site. I'd be grateful for any suggestions or solutions. I'm using R 2.0.0 on Mac OS X. Thank you, Stephen Weigand ### Test data date1 <- c(1000,

Hi, Try this: el<- read.csv("el.csv",header=TRUE,sep="\t",stringsAsFactors=FALSE) ?elsplit<- split(el,el$st) ? datetrial<-data.frame(date1=seq.Date(as.Date("1930.1.1",format="%Y.%m.%d"),as.Date("2010.12.31",format="%Y.%m.%d"),by="day")) elsplit1<- lapply(elsplit,function(x)

I am currently puzzled by a seach path behavior. I have a library of a dozen routines getlabs(), getssn(), getecg(), ... that interface to local repositories and pull back patient information. All have a the first 6 arguments in common, and immediately call a second routine to do initial processing of these 6. The functions "joe" and "fred" below capture the relevant

Full_Name: Alexander L. Belikoff Version: 2.8.1 OS: Ubuntu 9.04 (x86_64) Submission from: (NULL) (67.244.71.200) I'm trying to reshape the following data frame: ID DATE1 DATE2 VALUE_TYPE VALUE 'abcd1233' 2009-11-12 2009-12-23 'TYPE1' 123.45 ... VALUE_TYPE is a string and is a factor with only 2 values

I have time series observation on daily frequencies : library(zoo) SD=1 date1 = seq(as.Date("01/01/01", format = "%m/%d/%y"), as.Date("12/31/02", format = "%m/%d/%y"), by = 1) len1 = length(date1); data1 = zoo(matrix(rnorm(len1, mean=0, sd=SD*0.5), nrow = len1),  date1) plot(data1) Now I want to calculate 1. Quarterly statistics like mean, variance etc

Hi, I have a query about sqldf, and dates in general. I couldnt find much on the net or on the forums, hence I am here. Here is the issue: I want to write a function that accepts 3 arguments: date1, date2 and a dataframe, say 'df'. Within the function, I want to populate a temp dataframe which essentially contains the output of the query "select * from df where DATE between date1

I Have been trying to convert a vector of dates into julian dates using the following commands: as.date & as.numeric. I can convert a date no problem by doing the following: as.numeric (as.date ("9/21/2004")) but as soon as I try to do an entire vector I am given the following: as.numeric (as.date(date1)) Error in as.date(date1) : Cannot coerce to date format I have tried