I am using a "for" loop to read a table row by row and I have to
specify how
many records are there in the table. I need to read row by row because the
table is huge and the memory not large enough for the whole table.:
number.of.records=100
fp=file("abc.csv","r")
pos=seek(fp, rw="read")
for (i in 1:number.of.record){
current.row=scan(file=fp, sep=',', what=list(count=1,
cusip6="", idate=1,
spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab1.1,
cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
I need to know the number of records in the table and put it in the variable
named "number.of.records". When I have a new table that I do not know
how
many records it has, I use excel to open the file to figure it out and put
it in variable " number.of.records". I often have many tables to try
and
every one of them has thousands of records----it takes a lot of time and
trouble to adjust the code every time I read a different table.
I am wondering if I can change the "for" loop to a "while"
loop:
while (the end of the table has not been reached)
{
current.row=scan(file=fp, sep=',', what=list(count=1,
cusip6="", idate=1,
spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab1.1,
cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1,
cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
...
}
The problem is how to articulate "while (the end of the table has not been
reached)", or equivalently, how to signal the end of the table?
Best Wishes
Yuchen Luo
[[alternative HTML version deleted]]
What about trying to use a database system and make queries? If you are at the end of a file, you will read elements of length 0... Uwe Ligges Yuchen Luo wrote:> I am using a "for" loop to read a table row by row and I have to specify how > many records are there in the table. I need to read row by row because the > table is huge and the memory not large enough for the whole table.: > > > > number.of.records=100 > > > > fp=file("abc.csv","r") > > pos=seek(fp, rw="read") > > for (i in 1:number.of.record){ > > current.row=scan(file=fp, sep=',', what=list(count=1, cusip6="", idate=1, > spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab> 1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1, > cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T) > > ... > > } > > > > I need to know the number of records in the table and put it in the variable > named "number.of.records". When I have a new table that I do not know how > many records it has, I use excel to open the file to figure it out and put > it in variable " number.of.records". I often have many tables to try and > every one of them has thousands of records----it takes a lot of time and > trouble to adjust the code every time I read a different table. > > > > I am wondering if I can change the "for" loop to a "while" loop: > > > > while (the end of the table has not been reached) > > { > > current.row=scan(file=fp, sep=',', what=list(count=1, cusip6="", idate=1, > spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab> 1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1, > cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T) > > ... > > } > > > The problem is how to articulate "while (the end of the table has not been > reached)", or equivalently, how to signal the end of the table? > > Best Wishes > Yuchen Luo > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.
Have you tried read.table() and the similar functions? I think they would provide a much more simple solution for your problem. HTH, Thierry ------------------------------------------------------------------------ ---- ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest Cel biometrie, methodologie en kwaliteitszorg / Section biometrics, methodology and quality assurance Gaverstraat 4 9500 Geraardsbergen Belgium tel. + 32 54/436 185 Thierry.Onkelinx op inbo.be www.inbo.be Do not put your faith in what statistics say until you have carefully considered what they do not say. ~William W. Watt A statistical analysis, properly conducted, is a delicate dissection of uncertainties, a surgery of suppositions. ~M.J.Moroney> -----Oorspronkelijk bericht----- > Van: r-help-bounces op stat.math.ethz.ch > [mailto:r-help-bounces op stat.math.ethz.ch] Namens Yuchen Luo > Verzonden: woensdag 29 augustus 2007 19:41 > Aan: r-help op stat.math.ethz.ch > Onderwerp: [R] How to signal the end of the table? > > I am using a "for" loop to read a table row by row and I have > to specify how many records are there in the table. I need to > read row by row because the table is huge and the memory not > large enough for the whole table.: > > > > number.of.records=100 > > > > fp=file("abc.csv","r") > > pos=seek(fp, rw="read") > > for (i in 1:number.of.record){ > > current.row=scan(file=fp, sep=',', what=list(count=1, > cusip6="", idate=1, spread=1.1, vol252=1.1, vol1000=1.1, > st_debt=1.1, lt_debt=1.1 , total_liab> 1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, > sec=1.1, cr3m=1.1, > cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T) > > ... > > } > > > > I need to know the number of records in the table and put it > in the variable named "number.of.records". When I have a new > table that I do not know how many records it has, I use excel > to open the file to figure it out and put it in variable " > number.of.records". I often have many tables to try and every > one of them has thousands of records----it takes a lot of > time and trouble to adjust the code every time I read a > different table. > > > > I am wondering if I can change the "for" loop to a "while" loop: > > > > while (the end of the table has not been reached) > > { > > current.row=scan(file=fp, sep=',', what=list(count=1, > cusip6="", idate=1, spread=1.1, vol252=1.1, vol1000=1.1, > st_debt=1.1, lt_debt=1.1 , total_liab> 1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1, > sec=1.1, cr3m=1.1, > cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T) > > ... > > } > > > The problem is how to articulate "while (the end of the table > has not been reached)", or equivalently, how to signal the > end of the table? > > Best Wishes > Yuchen Luo > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help op stat.math.ethz.ch mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >
Well, i am surprise you have problems to read a table that is "small"
enough to be opened entirely in Excel.
I work with csv tables with hundreds of thousands of rows, and sometimes even
millions .... with no problems except that sometimes i have to wait up to 1 or 2
minutes for R to read the table. I use the command read.csv.
If your tables are always small enough to be read in Excel, the limit of number
of rows in Excel is about 64000 or something of that sort (maybe 65000 ....) so
you can use this number as a limit for your number of rows .... although you may
have to "clean-up" your data.frame afterwards.
I work on a Windows machine with 4 Gb DRAM - just for comparison.
I hope this helps,
Monica
<Message: 88<Date: Wed, 29 Aug 2007 10:41:05 -0700<From: "Yuchen
Luo" <realityrandom@gmail.com><Subject: [R] How to signal the end
of the table?<To:
r-help@stat.math.ethz.ch<Message-ID:<<548b8d440708291041obb96e03t72017bcbe254e3d8@mail.gmail.com><<Content-Type:
text/plain<<I am using a "for" loop to read a table row by row
and I have to specify how<many records are there in the table. I need to read
row by row because the<table is huge and the memory not large enough for the
whole
table.:<<<<number.of.records=100<<<<fp=file("abc.csv","r")<<pos=seek(fp,
rw="read")<<for (i in
1:number.of.record){<<current.row=scan(file=fp, sep=',',
what=list(count=1, cusip6="", idate=1,<spread=1.1, vol252=1.1,
vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab=<1.1, cr=1.1, shrout=1.1,
prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1,<cr5y=1.1, ust3m=1.1,
ust5y=1.1), flush=TRUE, nlines=1,quiet=T)<<...<<}<<<<I
need to know the number of records in the table and put it in the
variable<named "number.of.records". When I have a new table that I
do not know how<many records it has, I use excel to open the file to figure
it out and put<it in variable " number.of.records". I often have
many tables to try and<every one of them has thousands of records----it takes
a lot of time and<trouble to adjust the code every time I read a different
table.<<<<I am wondering if I can change the "for" loop to
a "while" loop:<<<<while (the end of the table has not
been reached)<<{<<current.row=scan(file=fp, sep=',',
what=list(count=1, cusip6="", idate=1,<spread=1.1, vol252=1.1,
vol1000=1.1, st_debt=1.1, lt_debt=1.1 , total_liab=<1.1, cr=1.1, shrout=1.1,
prc=1.1, mkt_cap=1.1, rtng=1.1, sec=1.1, cr3m=1.1,<cr5y=1.1, ust3m=1.1,
ust5y=1.1), flush=TRUE, nlines=1,quiet=T)<<...<<}<<<The
problem is how to articulate "while (the end of the table has not
been<reached)", or equivalently, how to signal the end of the
table?<<Best Wishes<Yuchen Luo<< [[alternative HTML version
deleted]]<
_________________________________________________________________
Discover the new Windows Vista
[[alternative HTML version deleted]]
For example,
fp<-file("abc.csv","r")
c.row<-scan(file=fp,sep=",",nlines=1) # "what" argument
is omitted for
bevity as it doesn't matter
rows<-c.row
while(length(c.row)>0) {
c.row<-scan(file=fp,sep=";",nlines=1);
rows<-rbind(rows,c.row)
}
close(fp)
If you want to read a file by parts, then you do something like the
following (untested)
But this looks like a C-style approach, not R-style
fp<-file("abc.csv","r")
lines.skip<-0
for(i in 1:nparts){
c.row<-scan(file=fp,sep=",",nlines=1,skip=lines.skip)
lines.read<-1
rows<-c.row
while(length(c.row)>0 && lines.read<part.size) {
c.row<-scan(file=fp,sep=";",nlines=1); # nlines can be also
>1
rows<-rbind(rows,c.row)
lines.read<-lines.read+1 # if nlines above is >1 then "+1"
must be
replaced with nlines value
}
lines.skip<-lines.read
# do operations with rows
}
close(fp)
Yuchen Luo wrote:>
> I am using a "for" loop to read a table row by row and I have to
specify
> how
> many records are there in the table. I need to read row by row because the
> table is huge and the memory not large enough for the whole table.:
>
>
>
> number.of.records=100
>
>
>
> fp=file("abc.csv","r")
>
> pos=seek(fp, rw="read")
>
> for (i in 1:number.of.record){
>
> current.row=scan(file=fp, sep=',', what=list(count=1,
cusip6="", idate=1,
> spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 ,
> total_liab> 1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1,
sec=1.1,
> cr3m=1.1,
> cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
>
> ...
>
> }
>
>
>
> I need to know the number of records in the table and put it in the
> variable
> named "number.of.records". When I have a new table that I do not
know how
> many records it has, I use excel to open the file to figure it out and put
> it in variable " number.of.records". I often have many tables to
try and
> every one of them has thousands of records----it takes a lot of time and
> trouble to adjust the code every time I read a different table.
>
>
>
> I am wondering if I can change the "for" loop to a
"while" loop:
>
>
>
> while (the end of the table has not been reached)
>
> {
>
> current.row=scan(file=fp, sep=',', what=list(count=1,
cusip6="", idate=1,
> spread=1.1, vol252=1.1, vol1000=1.1, st_debt=1.1, lt_debt=1.1 ,
> total_liab> 1.1, cr=1.1, shrout=1.1, prc=1.1, mkt_cap=1.1, rtng=1.1,
sec=1.1,
> cr3m=1.1,
> cr5y=1.1, ust3m=1.1, ust5y=1.1), flush=TRUE, nlines=1,quiet=T)
>
> ...
>
> }
>
>
> The problem is how to articulate "while (the end of the table has not
been
> reached)", or equivalently, how to signal the end of the table?
>
> Best Wishes
> Yuchen Luo
>
>
--
View this message in context:
http://www.nabble.com/How-to-signal-the-end-of-the-table--tf4353030.html#a12408586
Sent from the R help mailing list archive at Nabble.com.