R help - Jan 2009 - Process File Line By Line Without Slurping into Object

Dear all,

In general practice one would slurp the whole file
using this method before processing the data:

dat <- read.table(filename)

or variations of it.

Is there a way we can access the file line by line
without slurping/storing them into object?

I am thinking something like this in Perl:

__BEGIN__
open INFILE, '<' , 'filename.txt' or die $!;
while (<INFILE>) {
    my $line = $_;
    # then process line by line
}
__END__

the reason I want to do that is because the data
I am processing are large (~ 5GB), my PC may not
be able to handle that.

- Gundala Viswanath
Jakarta - Indonesia

Gundala Viswanath wrote:
> Dear all,
>
> In general practice one would slurp the whole file
> using this method before processing the data:
>
> dat <- read.table(filename)
>
> or variations of it.
>
> Is there a way we can access the file line by line
> without slurping/storing them into object?
>
> I am thinking something like this in Perl:
>
> __BEGIN__
> open INFILE, '<' , 'filename.txt' or die $!;
> while (<INFILE>) {
>     my $line = $_;
>     # then process line by line
> }
> __END__
>
> the reason I want to do that is because the data
> I am processing are large (~ 5GB), my PC may not
> be able to handle that.
>   
Use a connection.  For example,

con <- file("filename.txt", "rt")
repeat {
    line <- readLines(con, 1)
    if (!length(line)) break
    # process the line, or change the "1" to a bigger number, and 
process the block of lines
}
close(con)

Duncan Murdoch