Berin Loritsch
2006-Jul-27 19:58 UTC
[Rails] ActiveRecord retrieval style? Lazy initialtization?
I''m using ActiveRecord for an Extract and Transform operation, so it is outside of its native environment of Rails. I am running into a problem with memory consumption when I retrieve large amounts of data. By large amounts I am talking about roughly 12,700 records that each have a one to many relationship with about 15 additional types of information. One of those types happens to be a table with binaries in it. Based on the way the system seems to be responding, its as if ActiveRecord is creating an array and then using lazy initialization for the records. Creating the ET app was very easy, but I''m running into a brick wall here. I know that ActiveRecord was designed for web systems, so being able to process 12,700 complex records is outside of what you might consider normal use. The question I have is whether I can access it in a more fly-weight model. In short, once I''m done with a record I don''t need it any more. Most database related stuff I''ve used operates this way. In short you have one object (in Java it would be a ResultSet), and its contents are mutated as you iterate over the whole recordset. Instead of trying to create 12,700 objects and its ancillary information, merely creating one object and changing its value as you iterate. Am I going to have to go to a lower level API to do what I need? If so, I would still like to use the ActiveRecord API for each individual record.... Using the low level API to get the list of IDs and then ActiveRecord to retrieve just the record for that ID. There''s got to be something I''m missing. -- *Berin Loritsch* Owner *Work:* 571-215-7708 *Email:* bloritsch@d-haven.com <mailto:bloritsch@d-haven.com> *IM:* bloritsch@d-haven.org (MSN) *http://www.linkedin.com/in/bloritsch * * * *D-Haven* <http://d-haven.org> 20044 Great Falls Forest Dr. <http://maps.google.com/maps?q=20044+Great+Falls+Forest+Dr.%2CGreat+Falls%2CVA+22066%2CUSA&hl=en> Great Falls, VA 22066 USA See who we know in common <http://www.linkedin.com/e/wwk/5678360/> Want a signature like this? <http://www.linkedin.com/e/sig/5678360/> -------------- next part -------------- Skipped content of type multipart/related
Jan Prill
2006-Jul-28 06:56 UTC
[Rails] ActiveRecord retrieval style? Lazy initialtization?
Hi Berin, aren''t you able to use an approach similare to what the paginate method does? In a first step cound the records you are going to process then iterate on them in a paginated way by using ''where id in (id, id, id, id)'' Cheers, Jan On 7/27/06, Berin Loritsch <bloritsch@d-haven.org> wrote:> > I''m using ActiveRecord for an Extract and Transform operation, so it is > outside of its native environment of Rails. I am running into a problem > with memory consumption when I retrieve large amounts of data. By large > amounts I am talking about roughly 12,700 records that each have a one to > many relationship with about 15 additional types of information. One of > those types happens to be a table with binaries in it. > > Based on the way the system seems to be responding, its as if ActiveRecord > is creating an array and then using lazy initialization for the records. > Creating the ET app was very easy, but I''m running into a brick wall here. > I know that ActiveRecord was designed for web systems, so being able to > process 12,700 complex records is outside of what you might consider normal > use. > > The question I have is whether I can access it in a more fly-weight > model. In short, once I''m done with a record I don''t need it any more. > Most database related stuff I''ve used operates this way. In short you have > one object (in Java it would be a ResultSet), and its contents are mutated > as you iterate over the whole recordset. Instead of trying to create 12,700 > objects and its ancillary information, merely creating one object and > changing its value as you iterate. > > Am I going to have to go to a lower level API to do what I need? If so, I > would still like to use the ActiveRecord API for each individual record.... > Using the low level API to get the list of IDs and then ActiveRecord to > retrieve just the record for that ID. > > There''s got to be something I''m missing. > > -- > *Berin Loritsch* > Owner > *Work:* 571-215-7708 > *Email:* bloritsch@d-haven.com > *IM:* bloritsch@d-haven.org (MSN) > *http://www.linkedin.com/in/bloritsch > * > * * *D-Haven* <http://d-haven.org> > 20044 Great Falls Forest Dr. > <http://maps.google.com/maps?q=20044+Great+Falls+Forest+Dr.%2CGreat+Falls%2CVA+22066%2CUSA&hl=en>Great Falls, VA 22066 > USA > See who we know in common <http://www.linkedin.com/e/wwk/5678360/> Want > a signature like this? <http://www.linkedin.com/e/sig/5678360/> > > _______________________________________________ > Rails mailing list > Rails@lists.rubyonrails.org > http://lists.rubyonrails.org/mailman/listinfo/rails > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://wrath.rubyonrails.org/pipermail/rails/attachments/20060728/7fe6320c/attachment.html