Rails - May 2007 - Best way to convert a legacy database that's full of data?

I need to convert a legacy database - the usual non-Railish schema with
several million rows - to a Rails-compatible schema.

Are there any tools / tutorials / books / blog-posts that would help me
with that?

I need to solve problems like connecting to two databases at the same
time, querying the old non-Railish schema and using the data to create
Rails objects, etc.

Thanks in advance,
Maurice B. Gladwell
A recent Rails convert

-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Maurice,

Which DBMS ?  

Each DBMS has different facilities available and different capabilities, 
especially when it comes to linking two databases together.   If you are 
migrating everything, you''d probably be a lot easier putting it all
into one
database which will save you have to use "federated" support to join
across
databases.   

You also have a chance to think whether you want to change DBMS.

Are you talking about modifying the structures ?   

If so, the book "Refactoring Databases : Evolutionary Database Design"
by
Scott Ambler and Pramodkumar J Sadalage (Addison-Wesley, ISBN-10: 0321293533, 
ISBN-13: 978-0321293534) is a great primer for the types of changes you will 
have to make.

The biggest challenge you are going to face is making modifications while not 
breaking existing applications (I assume you have existing apps).   

If I can be any help then drop me a line : especially if you wish to consider 
DB2 as the DBMS of choice going forward.   DB2 has good Rails support, and 
getting better by the day, and the free DB2 Express-C has no limit on 
database size, only on memory (4 gig) and processors (2 sockets, not cores).

HTH

Phil
(teamdba-RvFq4O/J5Z7QT0dZR+AlfA@public.gmane.org)

On Wednesday 30 May 2007 13:17, Maurice Gladwell wrote:
> I need to convert a legacy database - the usual non-Railish schema with
> several million rows - to a Rails-compatible schema.
>
> Are there any tools / tutorials / books / blog-posts that would help me
> with that?
>
> I need to solve problems like connecting to two databases at the same
> time, querying the old non-Railish schema and using the data to create
> Rails objects, etc.
>
> Thanks in advance,
> Maurice B. Gladwell
> A recent Rails convert
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

The Rails recipes from the Pragmatic Programmers by Chad Flower, has
some tips on connecting to legacy databases.
That should point you in the right direction, or at least give you
some idea of what you''re up against


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi Phil,

The DBMS is MySQL 5. It is serving a single, high-traffic web 
application.

My naive approach was much simpler than what you outlined - I was 
thinking about writing a conversion script, that will:

1) go over the legacy database, pulling data from it

2) use data pulled in #1 to instantiate ActiveRecord objects, which will 
be compatible with the new schema of the new database

3) save those ActiveRecord objects to the new database

Does that make sense?  I thought it would be the most conventional 
approach, so there must be material describing such a procedure.

--
M.

-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

You might also want to look at the ETL (Extract, Transform, Load)
approach found in active_warehouse.  If you can reduce your dependency
to cyclical connectivity (say daily or hourly), you can simplify the
process quite a bit.  Or, if you could ETL the legacy data and just
maintain connectivity for the sake of a few tables, you''ve reduced the
runtime load and complexity of your system quite a bit.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi Maurice,

On May 30, 2:17 pm, Maurice Gladwell <rails-mailing-l...@andreas-
s.net> wrote:
> My naive approach was much simpler than what you outlined - I was
> thinking about writing a conversion script, that will:
>
> 1) go over the legacy database, pulling data from it
>
> 2) use data pulled in #1 to instantiate ActiveRecord objects, which will
> be compatible with the new schema of the new database
>
> 3) save those ActiveRecord objects to the new database
>
> Does that make sense?  I thought it would be the most conventional
> approach, so there must be material describing such a procedure.
This is exactly how we''re doing it for one of our clients.
We''ve
created AR objects for all the old tables, and then we''re doing a find
on the old AR object, and looping over the result, to stick it into
the new AR models.
Something like this:
---
class ImportBibData < ActiveRecord::Base
  establish_connection :import
  set_table_name "BibData"
end
---
Then doing something like this...
---
  i_bib_items = ImportBibItem.find_all_by_doctype("M")
  i_bib_items.each do |i_bib_item|
    new_object NewObject.find_or_create_by_import_id(i_bib_item.bibid)
  end
----
It''s not DRY at all, and it''s quite a bit of a mess - but
it''s going
to be more or less a one-off import once the client freezes their old
(Access!!) DB and we start the import.

Hope that''s of any help.

Cheers,

  JS

PS, my first post, so have mercy with me if I''m doing something wrong!
PPS, Hi all!


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

How are you connecting to Access? I hope this isn''t too OT, but there
are a lot of threads in this and other forums saying it''s a mess/not
worth it.

Ron

> ----
> It''s not DRY at all, and it''s quite a bit of a mess - but
it''s going
> to be more or less a one-off import once the client freezes their old
> (Access!!) DB and we start the import.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

On May 31, 11:57 am, paron
<rphill...-Gbsd49wXnelcb8pDYxsr+SzEyWLYDCHU@public.gmane.org>
wrote:
> How are you connecting to Access? I hope this isn''t too OT, but
there
> are a lot of threads in this and other forums saying it''s a
mess/not
> worth it.
It is a mess - so we don''t! :D We use the mdb-tools package to extract
the data as CSV, and the schema as SQL. We then convert the schema to
MySQL compatible SQL, and dump it into MySQL. From there we run a
mysqlimport on the CSV files to populate the DB with the old data.

It''s a mess for sure, but once it''s in the MySQL DB
it''s much nicer to
work with, and once it''s in the new structure we should be all happy
smiling Railers...

/Jocke


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---