Rails - Jul 2006 - ActiveRecord retrieval style? Lazy initialtization?

I''m using ActiveRecord for an Extract and Transform operation, so it is
outside of its native environment of Rails.  I am running into a problem 
with memory consumption when I retrieve large amounts of data.  By large 
amounts I am talking about roughly 12,700 records that each have a one 
to many relationship with about 15 additional types of information.  One 
of those types happens to be a table with binaries in it.

Based on the way the system seems to be responding, its as if 
ActiveRecord is creating an array and then using lazy initialization for 
the records.  Creating the ET app was very easy, but I''m running into a
brick wall here.  I know that ActiveRecord was designed for web systems, 
so being able to process 12,700 complex records is outside of what you 
might consider normal use.

The question I have is whether I can access it in a more fly-weight 
model.  In short, once I''m done with a record I don''t need it
any more.
Most database related stuff I''ve used operates this way.  In short you 
have one object (in Java it would be a ResultSet), and its contents are 
mutated as you iterate over the whole recordset.  Instead of trying to 
create 12,700 objects and its ancillary information, merely creating one 
object and changing its value as you iterate.

Am I going to have to go to a lower level API to do what I need?  If so, 
I would still like to use the ActiveRecord API for each individual 
record.... Using the low level API to get the list of IDs and then 
ActiveRecord to retrieve just the record for that ID.

There''s got to be something I''m missing.

-- 
*Berin Loritsch*
Owner 	

*Work:* 571-215-7708
	*Email:* bloritsch@d-haven.com <mailto:bloritsch@d-haven.com>
*IM:* bloritsch@d-haven.org (MSN)
*http://www.linkedin.com/in/bloritsch
*
* * 		*D-Haven* <http://d-haven.org>
20044 Great Falls Forest Dr.
<http://maps.google.com/maps?q=20044+Great+Falls+Forest+Dr.%2CGreat+Falls%2CVA+22066%2CUSA&hl=en>
Great Falls, VA 22066
USA

See who we know in common <http://www.linkedin.com/e/wwk/5678360/> 	Want 
a signature like this? <http://www.linkedin.com/e/sig/5678360/>

-------------- next part --------------
Skipped content of type multipart/related

Hi Berin,

aren''t you able to use an approach similare to what the paginate method
does?

In a first step cound the records you are going to process then iterate on
them in a paginated way by using ''where id in (id, id, id,
id)''

Cheers,
Jan

On 7/27/06, Berin Loritsch <bloritsch@d-haven.org>
wrote:
>
>  I''m using ActiveRecord for an Extract and Transform operation, so
it is
> outside of its native environment of Rails.  I am running into a problem
> with memory consumption when I retrieve large amounts of data.  By large
> amounts I am talking about roughly 12,700 records that each have a one to
> many relationship with about 15 additional types of information.  One of
> those types happens to be a table with binaries in it.
>
> Based on the way the system seems to be responding, its as if ActiveRecord
> is creating an array and then using lazy initialization for the records.
> Creating the ET app was very easy, but I''m running into a brick
wall here.
> I know that ActiveRecord was designed for web systems, so being able to
> process 12,700 complex records is outside of what you might consider normal
> use.
>
> The question I have is whether I can access it in a more fly-weight
> model.  In short, once I''m done with a record I don''t
need it any more.
> Most database related stuff I''ve used operates this way.  In short
you have
> one object (in Java it would be a ResultSet), and its contents are mutated
> as you iterate over the whole recordset.  Instead of trying to create
12,700
> objects and its ancillary information, merely creating one object and
> changing its value as you iterate.
>
> Am I going to have to go to a lower level API to do what I need?  If so, I
> would still like to use the ActiveRecord API for each individual record....
> Using the low level API to get the list of IDs and then ActiveRecord to
> retrieve just the record for that ID.
>
> There''s got to be something I''m missing.
>
> --
>     *Berin Loritsch*
> Owner
>       *Work:* 571-215-7708
>  *Email:* bloritsch@d-haven.com
> *IM:* bloritsch@d-haven.org (MSN)
>  *http://www.linkedin.com/in/bloritsch
> *
> * *  *D-Haven* <http://d-haven.org>
> 20044 Great Falls Forest Dr.
>
<http://maps.google.com/maps?q=20044+Great+Falls+Forest+Dr.%2CGreat+Falls%2CVA+22066%2CUSA&hl=en>Great
Falls, VA 22066
> USA
>     See who we know in common
<http://www.linkedin.com/e/wwk/5678360/> Want
> a signature like this? <http://www.linkedin.com/e/sig/5678360/>
>
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060728/7fe6320c/attachment.html

So is your problem memory consumption, processing time, excessive SQL 
calls?

Cheers
Dr Nic

-- 
Posted via http://www.ruby-forum.com/.