I am trying to read in a tab-delimited data file with a 21 row header and 2 row footer using two calls to read.table(). Numbers of rows and columns are variable. The header contains information for calculating the number of rows of data. I can successfully pick this out and calculate the number of rows to read, but cannot get the second read.table() to assign this number to "nrows" (the number is correct; if I enter it manually, the everything works fine). Currently the function reads all the way to the end and crashes on the footer, since the number of fields is different from that of the data. I know this could easily be done with some Perl pre-processing of the file, but it is going to run on a Windows machine and I am trying to minimize the number of packages to download. Nevertheless, there is the general problem of why I cannnot pass a calculated value into the function. Code follows. # # function to read data with header and footer # # pick line 8 with the data layout information and calculate the number of rows ... grid.layout <- read.table(fname, as.is=T, header=F, sep="\t", comment.char="", skip=7, nrows=1) row.ctr <- grid.layout[4]*grid.layout[5]*grid.layout[6]*grid.layout[7] # tells me I have the right dimensions ... print(row.ctr) # but they do not get passed into this read.table() ... tmp.df <- read.table(fname, as.is=T, header=T, sep="\t", comment.char="", skip=20, nrows=row.ctr ) tmp.df # end function Regards, David R. McWilliams dmcwilli at utk.edu -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
I think you should avoid read.table here. It tries to set various parameters (number of columns, etc) by looking at a part of your file. Even if you specify col.names, it can get confused by this looking-ahead. Better look at the definition of read.table in terms of scan and readLines and modify to your needs. Perhaps your problem isn't that nrows doesn't get the right value, but that read.table is using the wrong number of columns. (Actually, I can't create an example like yours that read.table doesn't complain about line 1 not having n elements, where n is the number of elements in the longest line.) If you really want to make read.table work, you can probably use the fill=TRUE option and lots of caution. Reid Huntsinger -----Original Message----- From: David R. McWillliams [mailto:dmcwilli at utk.edu] Sent: Monday, April 22, 2002 1:13 PM To: r-help at stat.math.ethz.ch Subject: [R] Problem passing data into read.table() I am trying to read in a tab-delimited data file with a 21 row header and 2 row footer using two calls to read.table(). Numbers of rows and columns are variable. The header contains information for calculating the number of rows of data. I can successfully pick this out and calculate the number of rows to read, but cannot get the second read.table() to assign this number to "nrows" (the number is correct; if I enter it manually, the everything works fine). Currently the function reads all the way to the end and crashes on the footer, since the number of fields is different from that of the data. I know this could easily be done with some Perl pre-processing of the file, but it is going to run on a Windows machine and I am trying to minimize the number of packages to download. Nevertheless, there is the general problem of why I cannnot pass a calculated value into the function. Code follows. # # function to read data with header and footer # # pick line 8 with the data layout information and calculate the number of rows ... grid.layout <- read.table(fname, as.is=T, header=F, sep="\t", comment.char="", skip=7, nrows=1) row.ctr <- grid.layout[4]*grid.layout[5]*grid.layout[6]*grid.layout[7] # tells me I have the right dimensions ... print(row.ctr) # but they do not get passed into this read.table() ... tmp.df <- read.table(fname, as.is=T, header=T, sep="\t", comment.char="", skip=20, nrows=row.ctr ) tmp.df # end function Regards, David R. McWilliams dmcwilli at utk.edu -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-. -.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._. _._ ------------------------------------------------------------------------------ Notice: This e-mail message, together with any attachments, contains information of Merck & Co., Inc. (Whitehouse Station, New Jersey, USA) that may be confidential, proprietary copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named on this message. If you are not the intended recipient, and have received this message in error, please immediately return this by e-mail and then delete it. ============================================================================= -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
Sorry for the delay, only just now got back to it ...
1. debug() did not help, would not show state of nrows inside read.table()
2. fill=T did help, could have dropped the last 2 rows (footer).
By accident I found the problem. read.table() returns a dataframe and the
result of the calculation of the number of rows has mode "list".
Coercing it
to numeric solves the problem.
Working solution:
function(fname) {
# my.func(fname)
#
# Function to read file with header and footer
# Number of rows calculable from header row 8
#
grid.layout <- read.table(fname, as.is=T, header=F, sep="\t",
comment.char="",
skip=7, nrows=1)
# coerce rows calculation to numeric
row.ctr <-
as.numeric(grid.layout[4]*grid.layout[5]*grid.layout[6]*grid.layout[7])
#read.table now sees row.ctr
tmp.df <- read.table(fname, as.is=T, header=T, sep="\t",
comment.char="",
skip=20, nrows=row.ctr )
# drop empty first column
tmp.df <- tmp.df[, -1 ]
tmp.df
# end of my.func()
}
Thanks to all who helped.
regards,
>===== Original Message From Jason Turner <jasont at
indigoindustrial.co.nz>
====>On Mon, Apr 22, 2002 at 03:21:34PM -0400, David R. McWillliams
wrote:>> I thought about scan(), but I think that would require knowing the
number
>> of columns beforehand.
>
>Sounds like you need file(),readLines(), and strsplit(), with
>a home-brewed data frame filler loop.
>
>Something like (untested - just giving the idea here):
>
>con <- file(fname)
>header <- readLines(con, n=21)
>
>#some magic here, to get the max number of columns
>max.cols <- magic stuff
>
>my.data <- readLines(con)
>
>#and let's be hygenic
>close(con)
>
>#chop off the last 2 lines, as they're footer stuff we don't want
>#pushBack would probably work too, but I'm less familiar with it.
>last.two <- ( length(my.data)-1 ):length(my.data)
>my.data <- my.data[-last.two]
>
>#split by tabs
>my.data <- strsplit(my.data,"\t")
>
>my.mat <- matrix(NA,ncol=max.cols,nrow=length(my.data))
>
>for(i in seq(along=my.data)) {
> for(j in seq(along=my.data[[i]]) {
> my.mat[i][j] <- as.numeric(my.data[[i]][j])
> }
>}
>
>
>> As I noted before, if I put in the number of
>> rows in the second call to read.table(), say 'nrows=2000', it
correctly
>> detects the number of columns and rows. I just can't get it to see
the
>> calculated value as in 'nrows=row.ctr'. The intervening
print() can fetch
>> the value of row.ctr, so I don't understand why read.table()
can't get it.
>
>Not sure. Have you played with debug() ? I've found that very, very
>handy for when I lose values - just walk slowly through the function,
>one step at a time, and print out values as you think they might be
>changing.
>
>Cheers
>
>Jason
>--
>Indigo Industrial Controls Ltd.
>64-21-343-545
>jasont at indigoindustrial.co.nz
David McWilliams
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !) To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._