Hello, I have a problem. I have to read a HUGE file which has to be line processed, so I would like to use scan like p <- scan(pfile,what=c(0,0,0,"",0,""),nlines=1) but it continues to read the first line, so I have to add a skip and increment the skip after each read. It takes forever to read a test file of 1 Mb, the real one is in 60's Mb Is there a way to read line by line an ASCII file without having to reread the part already read? Thanks for any help. R. Heberto Ghezzo Ph.D. Meakins-Christie Labs McGill University Montreal - Canada heberto at meakins.lan.mcgill.ca -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
On Wed, 14 Mar 2001, Heberto Ghezzo wrote:> Hello, I have a problem. I have to read a HUGE file which has to be > line processed, so I would like to use scan like > p <- scan(pfile,what=c(0,0,0,"",0,""),nlines=1) > but it continues to read the first line, so I have to add a skip and > increment the skip after each read. > It takes forever to read a test file of 1 Mb, the real one is in 60's Mb > Is there a way to read line by line an ASCII file without having to > reread the part already read? > Thanks for any help.There are two fairly easy ways to do this sort of thing, depending on the type of expertise available: 1. Load the huge file into a database with a good text file import capability, like Microsoft Access, then use RODBC to read the data you're interested in with a query. This is what I do for manual analysis; I have lots of huge files. 2. Write a Perl script to extract the relevant data to a smaller file and then read the smaller file directly in R. This is what I do for my *automated* analysis. BTW, I've processed files of 120-odd MB this way on 400 MHz Pentium-class machines with 128 MB of RAM. -- znmeb at aracnet.com (M. Edward Borasky) http://www.aracnet.com/~znmeb What is it about @replace(@string(@log(10000)*@atan(1),9),0,0, at char(112)&@char(105)&@char(61)) that you don't understand? -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
On 14 Mar 01,, Heberto Ghezzo wrote (re: [R] scan):> Hello, I have a problem. I have to read a HUGE file which has to be > line processed, so I would like to use scan like > p <- scan(pfile,what=c(0,0,0,"",0,""),nlines=1) > but it continues to read the first line, so I have to add a skip and > increment the skip after each read. It takes forever to read a test file > of 1 Mb, the real one is in 60's Mb Is there a way to read line by line an > ASCII file without having to reread the part already read? Thanks for any > help. > > R. Heberto Ghezzo Ph.D. > Meakins-Christie Labs > McGill UniversityBonjour Heberto Just one option is to use read.table or read.csv: e.g. for a batch of files called name1A.txt, name1B.txt in the directory h:/data/nov99_tan9913/opc/ having 2 columns called code and value you could sequentially read the files dir <- c('h:/data/nov99_tan9913/opc/') files <- c('1A', '.1B' ,.....) for (i in 1:length(files)) { out <-read.csv(paste(dir, "name", files[i], ".txt", sep=''), header=F, row.names = NULL, col.names = c("code","value")) } To prevent having to reload the ascii files each time, save the object called out save(out, file = "opc.shuttle.transect1") And reload it later load("opc.shuttle.transect1") This works well for me and my files are rather large. Cheers Sam Sam McClatchie, Research scientist (fisheries acoustics)))))))))) NIWA (National Institute of Water & Atmospheric Research Ltd) PO Box 14 901, Kilbirnie, Wellington, New Zealand s.mcclatchie at niwa.cri.nz /\ ...>><xX(?> // \\ /// \\\ //// \\\\ /// <?)Xx><< ///// \\\\\\ ><(((?> >><(((?> ...>><xX(?>O<?)Xx><< -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._