Adam Fields
2006-Aug-23 16:59 UTC
[Rails] Is there a limit to the practical size of a fixture file?
Is there a limit on how big the fixtures can be? Suppose I want to use csv fixtures and load up a table with a few hundred thousand rows to prepopulate my production DB. I did an initial test, but I canceled it after 35 minutes (with no indication of whether it was working or not, and nothing got changed on the db). Obviously, this won''t work for regular testing because it''ll make the tests take too long, but is there a practical limit on how big fixtures can be? I''ll continue to play with this, but I was just wondering if anyone had done this before with fixtures of this size. -- - Adam ** Expert Technical Project and Business Management **** System Performance Analysis and Architecture ****** [ http://www.adamfields.com ] [ http://www.aquick.org/blog ] ............ Blog [ http://www.adamfields.com/resume.html ].. Experience [ http://www.flickr.com/photos/fields ] ... Photos [ http://www.aquicki.com/wiki ].............Wiki --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk -~----------~----~----~----~------~----~------~--~---
Adam Fields
2006-Aug-24 18:54 UTC
[Rails] Re: Is there a limit to the practical size of a fixture file?
On Wed, Aug 23, 2006 at 12:59:23PM -0400, Adam Fields wrote:> Is there a limit on how big the fixtures can be? > > Suppose I want to use csv fixtures and load up a table with a few > hundred thousand rows to prepopulate my production DB. > > I did an initial test, but I canceled it after 35 minutes (with no > indication of whether it was working or not, and nothing got changed > on the db). > > Obviously, this won''t work for regular testing because it''ll make the > tests take too long, but is there a practical limit on how big > fixtures can be? > > I''ll continue to play with this, but I was just wondering if anyone > had done this before with fixtures of this size.In case anyone else is interested, I''ve successfully imported a csv fixture file with 120k records. It took nearly 2.5 hours with indexes in place (which I realized halfway through but didn''t want to cancel). I''ve not had a chance to fully test it without indexes, but even without them, it took well over an hour to parse the whole csv file into memory before it even started writing to the db. -- - Adam ** Expert Technical Project and Business Management **** System Performance Analysis and Architecture ****** [ http://www.adamfields.com ] [ http://www.aquick.org/blog ] ............ Blog [ http://www.adamfields.com/resume.html ].. Experience [ http://www.flickr.com/photos/fields ] ... Photos [ http://www.aquicki.com/wiki ].............Wiki --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk -~----------~----~----~----~------~----~------~--~---