[R] rbind wastes memory

Duncan Murdoch
Mon, 30 May 2005 14:08:01 -0000

I'm guessing your problem is fragmented memory.  You are creating big
objects, then making them bigger.  This means R needs to go looking for
large allocations for the replacements, but they won't fit in the spots
left by the things you've deleted, so those are being left empty.

A solution to this is to use two passes:  first figure out how much
space you need, then allocate it and fill it.  E.g.

for (Cycle in 1:10) {
     rows[Cycle] <- .... some calculation based on the data ...
}

ds.tmp <- data.frame(x=double(sum(rows)), y=double(sum(rows)), ...

for (Cycle in 1:10) {
     ds.tmp[ appropriate rows, ] <- new data
}


Duncan Murdoch