R help - Apr 2010 - Error with read.csv.sql on processing large file

Hi,

I was trying to read & filter records from a large file using read.csv.sql.
I was successfully able to do that with a ~1 GB file. However I get the
following error with a >2 GB file which has exactly the same structure as
the first file.

*Error in try({ : *
*  RS-DBI driver: (RS_sqlite_import: cannot open file <filename> )*

-I have checked the file permission and the file exists in the directory.
-The error comes instantly, so I am assuming its not due to memory.
-If I skip significant number of rows(using skip= statement) from the > 2GB
file, I don't get the error.

I can do a quick and dirty fix by splitting the file into two and by
performing two separate reads, but may be thats not so elegant. Can somebody
please help?

--

	[[alternative HTML version deleted]]

1.  Are you able to determine if its the filtering step that is the
problem?  Also is it the size of the file or is it a particular record
that is causing the problem?  If you read it in in two pieces do both
pieces work or just one piece?   It might be a problem in the
filtering step (in which case you could try filtering it yourself
outside of R and then try reading that in) or there might be something
wrong with one of the records in which case you need to narrow down
which one by reading in portions.

2. Can you provide more info?  Platform?  Exactly what command you are using?

3. In some cases the problem was the one in FAQ #11
http://code.google.com/p/sqldf/#11._Why_am_I_having_difficulty_reading_a_data_file_using_SQLite


On Tue, Apr 6, 2010 at 1:57 AM, Moiz Saifee <moiz.saifee at gmail.com>
wrote:
> Hi,
>
> I was trying to read & filter records from a large file using
read.csv.sql.
> I was successfully able to do that with a ~1 GB file. However I get the
> following error with a >2 GB file which has exactly the same structure
as
> the first file.
>
> *Error in try({ : *
> * ?RS-DBI driver: (RS_sqlite_import: cannot open file <filename> )*
>
> -I have checked the file permission and the file exists in the directory.
> -The error comes instantly, so I am assuming its not due to memory.
> -If I skip significant number of rows(using skip= statement) from the >
2GB
> file, I don't get the error.
>
> I can do a quick and dirty fix by splitting the file into two and by
> performing two separate reads, but may be thats not so elegant. Can
somebody
> please help?
>
> --
>
> ? ? ? ?[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>