Hi, I''m trying to run a set of migration which brings data from a
legacy project. The legacy project is on production and have lots of
data. So, during the time the migration keep allocating and allocating
memory until it grabs it all. I''m doing pagination to bring the data
step by step. Beside, what is very wear is that the memory never goes
down between migration, for example, I do:
rake db:migrate
== 2 InitialMoveLegacyUsersToUsers: migrating
================================== 2 InitialMoveLegacyUsersToUsers: migrated
(83.3700s)
======================
== 3 AddPasswordResetCodeToUsers: migrating
==================================-- add_column(:users, :password_reset_code,
:string)
-> 0.0877s
== 3 AddPasswordResetCodeToUsers: migrated (0.0883s)
=========================
== 4 AddOpenIdAuthenticationTables: migrating
================================--
create_table(:open_id_authentication_associations, {:force=>true})
-> 0.0054s
-- create_table(:open_id_authentication_nonces, {:force=>true})
-> 0.0044s
== 4 AddOpenIdAuthenticationTables: migrated (0.0105s)
=======================
And watching the behavior of the memory between migrations it never
free memory, always keep growing.
So, it seems to me like a memory leak, but dunno if I''m doing
something wrong.
Is there a way to call the garbage collector to free the memory
between migrations??
I''ll appreciate your help.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---
Rafael Chacon wrote:> > rake db:migrate > > == 2 InitialMoveLegacyUsersToUsers: migrating > ================================> == 2 InitialMoveLegacyUsersToUsers: migrated (83.3700s) > ======================> > == 3 AddPasswordResetCodeToUsers: migrating > ==================================> -- add_column(:users, :password_reset_code, :string) > -> 0.0877s > == 3 AddPasswordResetCodeToUsers: migrated (0.0883s) > =========================> > == 4 AddOpenIdAuthenticationTables: migrating > ================================> -- create_table(:open_id_authentication_associations, {:force=>true}) > -> 0.0054s > -- create_table(:open_id_authentication_nonces, {:force=>true}) > -> 0.0044s > == 4 AddOpenIdAuthenticationTables: migrated (0.0105s) > ======================= > And watching the behavior of the memory between migrations it never > free memory, always keep growing.You probably need to show me what your migrations look like. You must be loading a load of database rows into ruby objects? Maybe you''re setting them to a variable that remains in scope? Don''t know. If you do rake db:migrate VERSION=2 and then seperately do rake db:migrate VERSION=4 does it isolate the problem to the first migration? -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
The most conlficting migration looks like this:
class MoveLegacySugarEntries < ActiveRecord::Migration
class Entries < ActiveRecord::Base;end
class Entry::SugarEntry < Entry;end
def self.up
Legacy::SugarReading.each do |le|
u = User.find_by_login(le.user.login)
unless u== nil
TzTime.zone = u.tz
Entry::SugarEntry.create!(
:registered_at => DateTime.parse("#{le.date}
#{le.time.hour}:#{le.time.min}"),
:comment => le.comment,
:value => le.reading,
:user => u)
end
end
end
Where the active record each is implemented like this:
class <<ActiveRecord::Base
def each(limit=1000)
rows = find(:all, :conditions => ["id > ?", 0], :limit =>
limit)
while rows.any?
rows.each { |record| yield record }
rows = find(:all, :conditions => ["id > ?",
rows.last.id], :limit => limit)
end
self
end
end
So, is getting 1000 rows at a time... Enough to don''t exploit the
memory.
Which variable might not be getting out of scope??
Anyway, how would you think possible to get garbage memory from
previous migrations??
For example, if I ran my migration from 0 when I got to this migration
I already have 1.5 gigas memory occupied, isn''t kind of wear???
Thanks for your help
On Jun 28, 3:37 am, Matthew Rudy Jacobs <rails-mailing-l...@andreas-
s.net> wrote:> Rafael Chacon wrote:
>
> > rake db:migrate
>
> > == 2 InitialMoveLegacyUsersToUsers: migrating
> > ================================> > == 2
InitialMoveLegacyUsersToUsers: migrated (83.3700s)
> > ======================>
> > == 3 AddPasswordResetCodeToUsers: migrating
> > ==================================> > -- add_column(:users,
:password_reset_code, :string)
> > -> 0.0877s
> > == 3 AddPasswordResetCodeToUsers: migrated (0.0883s)
> > =========================>
> > == 4 AddOpenIdAuthenticationTables: migrating
> > ================================> > --
create_table(:open_id_authentication_associations, {:force=>true})
> > -> 0.0054s
> > -- create_table(:open_id_authentication_nonces, {:force=>true})
> > -> 0.0044s
> > == 4 AddOpenIdAuthenticationTables: migrated (0.0105s)
> > =======================> > And watching the behavior of the
memory between migrations it never
> > free memory, always keep growing.
>
> You probably need to show me what your migrations look like.
> You must be loading a load of database rows into ruby objects?
>
> Maybe you''re setting them to a variable that remains in scope?
>
> Don''t know.
>
> If you do rake db:migrate VERSION=2
> and then seperately do rake db:migrate VERSION=4
>
> does it isolate the problem to the first migration?
> --
> Posted viahttp://www.ruby-forum.com/.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---