similar to: Error with read.csv.sql on processing large file

Dear R-Listers, I am a Windows user (R 2.6.2) using the development version of sqldf to try to read a 3GB file originally stored in .sas7bdat-format. I convert it to comma-delimited ASCII format with StatTransfer before trying to import just the rows I need into R. The problem is that I get this error: > f <- file("hugedata.csv") > DF <- sqldf("select * from f where

Dear list,   Dear list, I am using read.csv.sql() from the sqldf package to read individual-based data from a csv file that is too large for R. My original file contains a column called “ID” that identifies the individual. I would like to read in data for only one individual at a time, for example "Bobby". read.csv.sql("filename",sql = 'select * from file where ID =

Hi, I'm experimenting with using (R)SQLite to do data management. Here are two little problems that I've encountered: 1. The presence of ',' in string values causes trouble since ',' is also the delimiter used in the SQL statement. 2. A newline '\n' line attached to the last string value of each row. Some examples: > library (RSQLite) Loading required

Hi, I'm working in R 2.11.1 x64 on Windows x86_64-pc-mingw32. I'm trying to insert a very large CSV file into a SQLite database. I'm pretty new to working with databases in R, so I apologize if I'm overlooking something obvious here. I'm trying to work with the American Community Survey data, which is two 1.3GB csv files. I have enough RAM to read one of them into memory,

Dear list, I need to read in a big table with 487 columns and 238,305 rows (row names and column names are supplied). Is there a code to read in the table in a fast way? I tried the read.table() but it seems that it takes forever :( Thanks a lot! Best, Allen

I'm trying to use sqdf's function read.csv.sql to read CSV files in which the missing values are represented by NA's. Plain old read.csv works fine on these files, but they are rather large and I'd like to filter using sql-like statements. However, even if I specify field.types correctly and nrows=-1, it still turns the columns with NA's into chars or 0. I'm trying to make

Dear all, I'm trying to import a csv file into sqlite3 and from there into R. Everything looks fine exepct that R outputs the character values in an odd fashion: they are shown as "\"CHARACTER\"" instead of "CHARACTER", but only if I show the character variable as a vector. Does someone know why this happens? Below is a sample code. The first part is written in

Hi, my friends. When a data file is large, loading the whole file into the memory all together is not feasible. A feasible way is to read one row, process it, store the result, and read the next row. In Fortran, by default, the 'read' command reads one line of a file, which is convenient, and when the same 'read' command is executed the next time, the next row of the same file

I'm sure this has gotten some attention before, but I have two CSV files generated from vmstat and free that are roughly 6-8 Mb (about 80,000 lines) each. When I try to use read.csv(), R allocates all available memory (about 4.9 Gb) when loading the files, which is over 300 times the size of the raw data. Here are the scripts used to generate the CSV files as well as the R code: Scripts (run

Hi List, Does any one know what package I need to use in order to fetch/get a large sized dataframe from SQL? I have already used sqldf package which is good for fetching large sized csv files. Thanks Alireza [[alternative HTML version deleted]]

Dear Bioc and R List Users, I am having trouble analysing illumine data generated from BeadScan. I have .idat files and JPEG images. I realise that i need bead-level summary data to be able to begin quality control followed by normalization. Is there a way i can read .idat files for expression analysis or do i need to go back to BeadScan and generate .txt files/tiff files ? Appreciate any help

Hello Everyone, I''d like to provide a page where the user can click to export some data as a csv file. It would be great to have the page show a progress value for percent complete on generating the file (file generation takes between 1 and 4 minutes currently). Once complete, I can show a link to download the temporary file. I''ve looked into BackgrounDrb and found some

Hello all, I wish to use read.csv to read a google doc spreadsheet. I try using the following code: data_url <- " http://spreadsheets0.google.com/spreadsheet/pub?hl=en&hl=en&key=0AgMhDTVek_sDdGI2YzY2R1ZESDlmZS1VYUxvblQ0REE&single=true&gid=0&output=csv " read.csv(data_url) Which results in the following error: Error in file(file, "rt") : cannot open

hi Jim, if i partition the file, then for further operation like merging the partitioned files and after that doing some analysis on whole data set would again require the same amount of memory. If i am not able to do or if i am not having memory then i feel there should be serious thinking over the issue of memory handling. hence i am also copying this to r-devel list and i would also would

hi Jim, if i partition the file, then for further operation like merging the partitioned files and after that doing some analysis on whole data set would again require the same amount of memory. If i am not able to do or if i am not having memory then i feel there should be serious thinking over the issue of memory handling. hence i am also copying this to r-devel list and i would also would

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Assuming the following: > x <- data.frame(a=1:10, b=runif(10)) > str(x) 'data.frame': 10 obs. of 2 variables: $ a: int 1 2 3 4 5 6 7 8 9 10 $ b: num 0.692 0.325 0.634 0.16 0.873 ... > write.csv(x, "x.csv") > x2 <- read.csv("x.csv") > str(x2) 'data.frame': 10 obs. of 3 variables: $ X:

Dear list, I have a file that is comma delimited but contains some erroneous non-delimiter commas. I would like to replace these commas with semicolons and then read the correct file into R as a data frame. I want to do this from within R, without changing the original data file. My current idea of how to do this would be to use system("sed ...") and feed the result to read.csv(), but

Hi, I have a table which I can read via read.csv: fx1<-read.csv(file="A_FX_M.csv", header=TRUE) TIME REER NTD JPY GBP HKD 1 198001 124.26 36.030 237.96 2.263980 4.8366 2 198002 126.59 36.030 244.05 2.290426 4.8765 3 198003 128.33 36.026 248.62 2.206045 4.9960 4 198004 127.85 36.063 251.67 2.215330 4.9760 5 198005 124.40 36.050 228.35 2.302026 4.8891 6 198006

Good Afternoon I have noticed results similar to the following several times as I have used R over the past several years. My .csv file has a header row and 3073 rows of data. > rskreg<-read.table('D:/data/riskregions.csv',header=T,sep=",") > dim(rskreg) [1] 2722 13 > rskreg<-read.csv('D:/data/riskregions.csv',header=T) > dim(rskreg) [1] 3073

Dear R experts, I have been looking into the help-pages and old questions from the R-Help site, but the options offered there don't seem to work in my case. First of all, I am working on Windows XP, using R version 2.6.2. I am attaching two csv files as an example of how the data I am traying to put together is delivered to us. On the first row of every csv file is the name of the