R help - Feb 2007 - read.csv size limits

I have been using the read.csv function for a while now without any problems.
My files are usually 20-50 MBs and they take up to a minute to import. They
have all been under 50,000 rows and under 100 columns.

Recently, I tried importing a file of a similar size (which means about the
same amount of data), but with ~500,000 columns and ~20 rows. The process is
taking forever (~1 hour so far). In Task Manager, I see the CPU is at max,
but memory slows down to a halt at around 50 MBs (far below memory limit).

Is this normal? Is there a way to optimize this operation or at least check
the progress? Will this take 2 hours or 200 hours?

All I was trying to do is transpose my extra-wide table with a process that
I assumed would take 5 minutes. Maybe R is not the solution I am looking
for?

Thanks.

-- 
View this message in context:
http://www.nabble.com/read.csv-size-limits-tf3302366.html#a9186136
Sent from the R help mailing list archive at Nabble.com.

Have you used colClasses to define what each of the columns contain?  Can
you use 'scan'?  I haven't tried anything with 500,000 columns, but
if they
are numeric, this should not take too long.  So I create a 20 line file with
500,000 columns and here is what it took reading it both as numeric and
character:
> system.time(x <- scan('/tempyy.txt', what=0))
Read 10000000 items
[1] 17.57  0.12 18.89    NA    NA
> str(x)
 num [1:10000000] 12345 12345 12345 12345 12345 ...
> system.time(x <- scan('/tempyy.txt', what=''))
Read 10000000 items
[1]  9.03  0.10 11.21    NA    NA
> str(x)
 chr [1:10000000] "12345" "12345" "12345"
"12345" "12345" "12345" "12345"
"12345" ...
>




On 2/27/07, andy1983 <andy1983@excite.com> wrote:
>
>
> I have been using the read.csv function for a while now without any
> problems.
> My files are usually 20-50 MBs and they take up to a minute to import.
> They
> have all been under 50,000 rows and under 100 columns.
>
> Recently, I tried importing a file of a similar size (which means about
> the
> same amount of data), but with ~500,000 columns and ~20 rows. The process
> is
> taking forever (~1 hour so far). In Task Manager, I see the CPU is at max,
> but memory slows down to a halt at around 50 MBs (far below memory limit).
>
> Is this normal? Is there a way to optimize this operation or at least
> check
> the progress? Will this take 2 hours or 200 hours?
>
> All I was trying to do is transpose my extra-wide table with a process
> that
> I assumed would take 5 minutes. Maybe R is not the solution I am looking
> for?
>
> Thanks.
>
> --
> View this message in context:
> http://www.nabble.com/read.csv-size-limits-tf3302366.html#a9186136
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem you are trying to solve?

	[[alternative HTML version deleted]]