ActiveRecord: ar-each_model

tvw
Wed, 11 Jan 2012 00:18:40 -0000

Hi,

today I went into a problem, when I had to iterate over a big result
set. ActiveRecord produces a huge array of model instances, which
consumed 1GB of memory on my machine. A comparable perl DBI script
only used some KB for iterating over the result set.

Then I was suggested to use batch (#find_each) for the problem. But
basically batch does the same thing by splitting the query into
several queries, which only return 1000 model instances in an array,
which still consumes more memory than necessary, but not that much.
The problem: it was much slower (it took 25 minutes, while the version
without batch took 90 seconds and the perl script took only 40
seconds) on my legacy database.

So I was searching for a method, which yields a model instance at a
time from the result set of the query, but I could not find anything.

At the end, I tried to find a solution myself and so I came up to this
simple solution

https://github.com/tvw/ar-each_model

I want to put into discussion here, since I wonder, why similar things
do not exist in ActiveRecord.