Hey there, I''ve just tried to load a massive CSV fixture file of about 2700000 lines (cites, locations etc). Not surprisingly it died half way through. Does anyone know how to load such an amount of data? Maybe splitting it into multiple files? Sorry for such a newb question... but, well, I am (to rails). John -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
El 8 Apr 2011, a les 22:45, John Wright <lists-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> va escriure:> Hey there, > > I''ve just tried to load a massive CSV fixture file of about 2700000 > lines (cites, locations etc). Not surprisingly it died half way through. > > Does anyone know how to load such an amount of data? Maybe splitting it > into multiple files? > > Sorry for such a newb question... but, well, I am (to rails).Do you need those many records for your test suite? -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
John Wright wrote in post #991838:> I''ve just tried to load a massive CSV fixture file of about 2700000 > lines (cites, locations etc). Not surprisingly it died half way through. > > Does anyone know how to load such an amount of data? Maybe splitting it > into multiple files? > > Sorry for such a newb question... but, well, I am (to rails).Don use fixtures, not even when just starting out. Instead choose one of the excellent test data factory gems to generate test data. And, only generate test data specifically for the tests you write. You''re not going to write 2700000 test on city data. Just a few examples are all you''ll need for your test suite. Generate those as you need them with your factory. https://github.com/thoughtbot/factory_girl https://github.com/notahat/machinist -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
> Does anyone know how to load such an amount of data? Maybe splitting it > into multiple files? >If you are loading it into a MySQL database, use the LOAD DATA INFILE command. It''s very fast and pretty flexible in terms of allowing you to define the record separators, file separators, and field mappings. - D -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.