Rails - Feb 2009 - How to get cursor / lazy semantics from find(:all)

I''m trying to figure out it there''s some alternative to
find(:all)
that can avoid instantiating every object at once, when I know that  
I''m just going to iterate over the collection and throw it away.  E.g.

def batch_process
  User.find(:all).each do |user|
    batch_process_one_user(user)
  end
end

Unfortunately, this may instantiate millions of User objects and use  
up GB of memory, which is not acceptable.  I''ve been digging through  
docs and code trying to figure out how I can do this within Rails,  
rather than having to go down to the raw DB connection and build from  
there, or pushing everything into a stored procedure.


Thanks,

-kevin

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

On Feb 23, 8:28 pm, Kevin Scaldeferri
<kevin+goo...-JJkA4VpjLwo0zfciT3NkkQ@public.gmane.org>
wrote:
> I''m trying to figure out it there''s some alternative to
find(:all)  
> that can avoid instantiating every object at once, when I know that  
> I''m just going to iterate over the collection and throw it away.
 E.g.
>
http://github.com/rails/rails/commit/d13623ca46c82120c398f4634e206422fc3ad7ea
?

Fred
> def batch_process
>   User.find(:all).each do |user|
>     batch_process_one_user(user)
>   end
> end
>
> Unfortunately, this may instantiate millions of User objects and use  
> up GB of memory, which is not acceptable.  I''ve been digging
through  
> docs and code trying to figure out how I can do this within Rails,  
> rather than having to go down to the raw DB connection and build from  
> there, or pushing everything into a stored procedure.
>
> Thanks,
>
> -kevin
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Thanks!  That''s close enough and easy to backport.  In principle, a  
solution that did O(1) queries, rather than O(N) would be nice, but  
the reduction in memory use at least gets me back to something  
tolerable.

-kevin


On Feb 23, 2009, at 12:37 PM, Frederick Cheung wrote:
>
>
>
> On Feb 23, 8:28 pm, Kevin Scaldeferri
<kevin+goo...-JJkA4VpjLwo0zfciT3NkkQ@public.gmane.org>
> wrote:
>> I''m trying to figure out it there''s some alternative
to find(:all)
>> that can avoid instantiating every object at once, when I know that
>> I''m just going to iterate over the collection and throw it
away.
>> E.g.
>>
>
>
http://github.com/rails/rails/commit/d13623ca46c82120c398f4634e206422fc3ad7ea
> ?
>
> Fred
>> def batch_process
>>  User.find(:all).each do |user|
>>    batch_process_one_user(user)
>>  end
>> end
>>
>> Unfortunately, this may instantiate millions of User objects and use
>> up GB of memory, which is not acceptable.  I''ve been digging
through
>> docs and code trying to figure out how I can do this within Rails,
>> rather than having to go down to the raw DB connection and build from
>> there, or pushing everything into a stored procedure.
>>
>> Thanks,
>>
>> -kevin
> >

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---