Hello! I''m just starting to venture into Ruby on Rails, and I was wondering what the best way to populate my databases with exisiting data in the form of csv files that don''t map exactly to the columns (ie require some kind of manipulation/data cleansing). When I used PHP I would just write a little PHP or Perl script to load directly into a MySQL table, but I was wondering if RoR had a way of abstracting that and I was wondering what the corresponding Ruby code might look like... Thanks! -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Rails doesn''t have a specific method for dealing with this, but you might write Ruby scripts to manipulate the data and insert the manipulated data into the DB. Depending on your data you may be able to use ActiveRecord in your scripts as well. -faisal On Aug 21, 2007, at 12:13 AM, Aaron Smidt wrote:> > Hello! > > I''m just starting to venture into Ruby on Rails, and I was wondering > what the best way to populate my databases with exisiting data in the > form of csv files that don''t map exactly to the columns (ie require > some > kind of manipulation/data cleansing). > > When I used PHP I would just write a little PHP or Perl script to load > directly into a MySQL table, but I was wondering if RoR had a way of > abstracting that and I was wondering what the corresponding Ruby code > might look like... > > Thanks! > -- > Posted via http://www.ruby-forum.com/. > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
gene.tani-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
2007-Aug-21 05:46 UTC
Re: Newb: load lots of data into database
On Aug 20, 9:13 pm, Aaron Smidt <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote:> Hello! > > I''m just starting to venture into Ruby on Rails, and I was wondering > what the best way to populate my databases with exisiting data in the > form of csv files that don''t map exactly to the columns (ie require some > kind of manipulation/data cleansing). > > When I used PHP I would just write a little PHP or Perl script to load > directly into a MySQL table, but I was wondering if RoR had a way of > abstracting that and I was wondering what the corresponding Ruby code > might look like... > > Thanks!LOAD DATA INFILE is pretty flexible. ISTR that it can take CSV''s directly, except that Excel would put in random double quotes around certain cells, and they needed to be cleaned out, if that''s where data''s from http://dev.mysql.com/doc/refman/5.0/en/load-data.html> -- > Posted viahttp://www.ruby-forum.com/.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
If you can clean up the data, perhaps this CSV importer plugin might be useful: http://www.stephenbartholomew.co.uk/2007/7/20/csv-importer-for-rails Regards, Chris -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
This is similar to a problem I had normalizing data for neural network training. Ruby is very good at handling csv data and you can use ActiveRecord outside of rails. My recommendation to you would be to write a script to read in the data (File and IO are the primary classes), using ActiveRecord. Using active record would allow you to use your object models to insert the data into the data base. If you have a lot of data, and going model by model would be too slow, you will want to look at the DBI interface. I don''t know of a good Ruby SQL loader. http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/166724 http://rails.aizatto.com/2007/05/21/activerecord-without-rails/ On Aug 21, 2007, at 12:13 AM, Aaron Smidt wrote:> > Hello! > > I''m just starting to venture into Ruby on Rails, and I was wondering > what the best way to populate my databases with exisiting data in the > form of csv files that don''t map exactly to the columns (ie require > some > kind of manipulation/data cleansing). > > When I used PHP I would just write a little PHP or Perl script to load > directly into a MySQL table, but I was wondering if RoR had a way of > abstracting that and I was wondering what the corresponding Ruby code > might look like... > > Thanks! > -- > Posted via http://www.ruby-forum.com/. > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---