On 11/7/2006 4:54 PM, Matt Anthony wrote:> Hello,
>
>
>
> I am trying to read in elements out of a very large binary file ... the
> total file is 4 gigs. I want to select rows out of the file, and the
> current procedure I run works but is prohibitively slow (takes more than
> a day to run and still won't complete). Is there any faster way to
> accomplish this?
You are doing several things that are likely to be slow.>
>
>
> My current procedure looks like this:
>
>
>
> readHH <- function(file_name, hhid_list) {
>
> incon=file(file_name, open="rb")
>
> result=data.frame()
>
> tran=list()
>
> byte_mark=0
>
> last_1M_mod=0
>
> file_size=file.info(file_name)$size
>
> write.table(paste("Data pulled from", file_name, sep="
"),
> file="readHH_output.txt", sep=",", row.names=FALSE,
col.names=FALSE,
> append=TRUE)
>
> while (TRUE) {
>
> tran$hh_id <- readBin(incon,integer(),1,size=4)
Why use a function call integer() here, rather than just the character
string "integer"?>
> if(is.element(tran$hh_id, hhid_list)) {
You don't show us the is.element() function, but since it's going to be
called a lot, it might be a place for an optimization.
>
> tran$prov_id <- readBin(incon,integer(),1,size=2)
>
> tran$txn_dn <- readBin(incon,integer(),1,size=2)
>
> tran$total_dollars <- readBin(incon,integer(),1,size=4)
>
> tran$total_items <- readBin(incon,integer(),1,size=4)
>
> tran$order_id <- readBin(incon,integer(),1,size=4)
>
> tran$txn_type <- readChar(incon,1)
>
> tran$gender <- readChar(incon,1)
>
> tran$zip_code <- readChar(incon,5)
>
> tran$region_code <- readChar(incon,1)
>
> tran$county_code <- readChar(incon,1)
>
> tran$state_abbrev <- readChar(incon,2)
>
> tran$channel_code <- readChar(incon,1)
>
> tran$source_code <- readChar(incon,20)
>
> tran$payment_type <- readChar(incon,1)
>
> tran$credit_card <- readChar(incon,1)
>
> tran$promo_type <- readChar(incon,1)
>
> tran$flags <- readChar(incon,1)
You could probably make all of this a lot faster by combining it into
three calls:
readBin(ints2, "integer", 2, size=2)
readBin(ints4, "integer", 4, size=4)
readChar(chars, 36)
and then extracting the elements after reading. The extraction will
probably be pretty fast, especially if you put the results into matrices
rather than data.frames. data.frames are hugely slower than matrices.
>
> write.table(data.frame(tran), file="readHH_output",
sep=",",
> row.names=FALSE, col.names=FALSE, append=TRUE)
This is going to reopen, seek, and close the file each time. Do you
really need to do that? Can't you open the output file once, then just
write the data to it?
>
> result <- rbind(result,data.frame(tran))
This is also very slow. It needs to grow a big list of vectors (which
is how result is stored) every time you read a record. It would be
faster if you could pre-allocate the result, and just assign values into
it, especially if you were assigning into a matrix, not a dataframe.
I don't know which of these suggestions will have the biggest effect.
I'd suggest trying them one by one until things are fast enough, and
then going on to something else.
Duncan Murdoch
>
> }
>
> else {
>
> byte_mark=byte_mark+42
>
> if (byte_mark>=file_size) {break}
>
> else {seek(incon, where=byte_mark)}
>
> }
>
> }
>
> return(result)
>
> }
>
>
>
> Thanks
>
>
>
> Matt
>
>
>
>
>
>
>
>
>
> Matt Anthony | Senior Statistician| 303.327.1761 |
> matt.anthony at NextAction.Net
> 10155 Westmoor Drive | Westminster, CO 80021 | FAX 303.327.1650
>
>
>
>
>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.