thr3ads.net - Rails core - some love for migrations [Mar 2010]

If this information is useful, please help other people find it:
Share via:

Josh Susser

2010-Mar-07 23:54 UTC

some love for migrations

I have a semi-grand plan for some changes to migrations.  Mostly this  
is motivated by the upcoming 3.0 release and wanting to get migrations  
in engines working.


1. Don''t skip migrations with version numbers lower than
"current".

I only recently tripped over this, but apparently it''s been this way  
since 2.1 when we got timestamps masquerading as large/sparse version  
numbers.  See ticket #4120.

rails.lighthouseapp.com/projects/8994/tickets/4120

I thought the whole point of timestamp version numbers was to be able  
to combine different development branches.  That seems like it should  
run migrations older than the most recent one.  I guess the  
alternative would be a new rake task: db:migrate:all that would run  
the lower-numbered versions that would otherwise be skipped.

I know there is a potential for confusion, but I don''t think that  
confusion is any worse than now with some migrations never being run.


2. Include the contents of schema_migrations in schema.rb

The current approach for db:schema:load is to assume that all  
migrations in db/migrate up to the :version option have been run.   
This is fraught with peril.  Given the issue in #1 above, it''s  
possible to assume some skipped migrations have been run, which could  
really mess up your data.  Since we''re encouraged to rake  
db:schema:load to deploy, the migrations table should be populated  
with accurate information.

It should be pretty simple to add this info to schema.rb.

Full db dumps should probably include the full schema_migrations table  
contents as well.


3. Add a timestamp column to schema_migrations

I almost always want the information about when migrations were run  
when I''m trying to investigate an issue.

Timestamp info should not be included in a schema.rb dump, as that  
would both cause a lot of git thrashing and sort of be a lie when it  
was loaded.


4. Support for migrations in engines.

rails.lighthouseapp.com/projects/8994/tickets/2058 is pretty stale and  
I don''t like that approach either.  Since it never reached consensus I
figured I''d try another approach that''s worked well for
Pivotal Labs''
Desert system (we''ve been doing things that way for several years).

The basic idea is to add a method to the Migration class:  
migrate_engine(engine_name, target_version).  You have to manually add  
a migration to the app''s migrations, but that''s nice and
explicit the
way we want it to be.  The migration looks something like this:

     class UpgradeToMarchRelease < ActiveRecord::Migration
       def self.up
         migrate_engine(''users'', 7)
         migrate_engine(''blogs'', 15)
       end

       def self.down
         migrate_engine(''users'', 6)
         migrate_engine(''blogs'', 13)
       end
     end

Desert''s approach of having a separate table to record plugin  
migrations had some problems, so I think the thing to do is add an  
"engine" column to schema_migrations to track them all in one place.

I''ve looked through the migration code and know just how to make this  
work - it''s very straightforward.  The engine support would work  
better if the changes 1 and 2 above were already done, but it''s not  
dependent on them.


I''m starting to work on some patches for these changes, but I thought  
it might be worth throwing it out there for a reaction before getting  
too deeply into it.

--
Josh Susser
http://blog.hasmanythrough.com



-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en.

Chad Woolley

2010-Mar-08 00:51 UTC

head link

Re: some love for migrations

On Sun, Mar 7, 2010 at 4:54 PM, Josh Susser <josh@hasmanythrough.com>
wrote:> I have a semi-grand plan for some changes to migrations.
+1 on all points.

I reviewed rails.lighthouseapp.com/projects/8994/tickets/2058, and
your solutions seem to cover all bases (someone speak up if it
doesn''t).  I especially like the part where you don''t copy
files to
the app, I always thought that sounded crazy.

Referencing the engines'' migration versions in a manually-created
migration makes sense.  You want to have a single "timeline" for your
app''s migrations, as described in this comment:
https://rails.lighthouseapp.com/projects/8994/tickets/2058#ticket-2058-28
(good analysis even though it recommends the file-copy approach).
Plus, manual creation can always be changed to automatic in the future
with higher-level abstractions.

pre-3.0 is also the right time to add fundamental changes like this,
so please post the ticket in this thread once you have a patch ready.

-- Chad

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en.

Jeremy Kemper

2010-Mar-08 01:49 UTC

head link

Re: some love for migrations

On Sun, Mar 7, 2010 at 3:54 PM, Josh Susser <josh@hasmanythrough.com>
wrote:> I have a semi-grand plan for some changes to migrations.  Mostly this is
> motivated by the upcoming 3.0 release and wanting to get migrations in
> engines working.
>
>
> 1. Don''t skip migrations with version numbers lower than
"current".
>
> I only recently tripped over this, but apparently it''s been this
way since
> 2.1 when we got timestamps masquerading as large/sparse version numbers.
>  See ticket #4120.
>
> rails.lighthouseapp.com/projects/8994/tickets/4120
>
> I thought the whole point of timestamp version numbers was to be able to
> combine different development branches.  That seems like it should run
> migrations older than the most recent one.  I guess the alternative would
be
> a new rake task: db:migrate:all that would run the lower-numbered versions
> that would otherwise be skipped.
>
> I know there is a potential for confusion, but I don''t think that
confusion
> is any worse than now with some migrations never being run.
Agreed.

> 2. Include the contents of schema_migrations in schema.rb
>
> The current approach for db:schema:load is to assume that all migrations in
> db/migrate up to the :version option have been run.  This is fraught with
> peril.  Given the issue in #1 above, it''s possible to assume some
skipped
> migrations have been run, which could really mess up your data.  Since
we''re
> encouraged to rake db:schema:load to deploy, the migrations table should be
> populated with accurate information.
>
> It should be pretty simple to add this info to schema.rb.
>
> Full db dumps should probably include the full schema_migrations table
> contents as well.
Agreed.

> 3. Add a timestamp column to schema_migrations
>
> I almost always want the information about when migrations were run when
I''m
> trying to investigate an issue.
>
> Timestamp info should not be included in a schema.rb dump, as that would
> both cause a lot of git thrashing and sort of be a lie when it was loaded.
This would be convenient, and nice polish, but not necessary.

> 4. Support for migrations in engines.
>
> rails.lighthouseapp.com/projects/8994/tickets/2058 is pretty stale and I
> don''t like that approach either.  Since it never reached consensus
I figured
> I''d try another approach that''s worked well for Pivotal
Labs'' Desert system
> (we''ve been doing things that way for several years).
>
> The basic idea is to add a method to the Migration class:
> migrate_engine(engine_name, target_version).  You have to manually add a
> migration to the app''s migrations, but that''s nice and
explicit the way we
> want it to be.  The migration looks something like this:
>
>    class UpgradeToMarchRelease < ActiveRecord::Migration
>      def self.up
>        migrate_engine(''users'', 7)
>        migrate_engine(''blogs'', 15)
>      end
>
>      def self.down
>        migrate_engine(''users'', 6)
>        migrate_engine(''blogs'', 13)
>      end
>    end
>
> Desert''s approach of having a separate table to record plugin
migrations had
> some problems, so I think the thing to do is add an "engine"
column to
> schema_migrations to track them all in one place.
>
> I''ve looked through the migration code and know just how to make
this work -
> it''s very straightforward.  The engine support would work better
if the
> changes 1 and 2 above were already done, but it''s not dependent on
them.
I''m not sure about this one.

I have a plugin with migrations for its own database. I want those to
run and be managed in their own universe. This works well today using
our existing tools: just override the connection class method on the
migration.

I have another plugin whose migrations I want mixed in with the
application''s. Rather than adding application migrations that compose
the plugin''s, I''d prefer to enlist the plugin into my
app''s
migrations:

class Web20Portal::Application < Rails::Application
  migrate :users, :blogs, :widgets
end

This would simply add these plugins'' migrations paths to the
app''s.
Then the migrator proceeds as usual.
> I''m starting to work on some patches for these changes, but I
thought it
> might be worth throwing it out there for a reaction before getting too
> deeply into it.
Great. Thanks Josh!

jeremy

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en.

Josh Susser

2010-Mar-08 04:28 UTC

head link

Re: some love for migrations

On Mar 7, 2010, at 5:49 PM, Jeremy Kemper wrote:> I''m not sure about this one.
>
> I have a plugin with migrations for its own database. I want those to
> run and be managed in their own universe. This works well today using
> our existing tools: just override the connection class method on the
> migration.
You can still do that with the approach I outlined.  That''s a good  
recommendation to put in the docs.
> I have another plugin whose migrations I want mixed in with the
> application''s. Rather than adding application migrations that
compose
> the plugin''s, I''d prefer to enlist the plugin into my
app''s
> migrations:
>
> class Web20Portal::Application < Rails::Application
>  migrate :users, :blogs, :widgets
> end
>
> This would simply add these plugins'' migrations paths to the
app''s.
> Then the migrator proceeds as usual.
I hope it doesn''t sound too snooty to say I''ve been dealing
with these
kinds of issues for several years and I don''t think that''s
going to
work they way you expect.  Mixing in the engine migrations sounds nice  
in theory.  In practice, I''ve run into situations where you really do  
want more manual control.  Since the engine code might be out of your  
control, you might have to run a few of the engine migrations, one of  
your own in your app to adjust to those changes, then the rest of the  
engine ones, then the rest of your app''s.  And then, what if
you''re
going to remove the engine?  (perhaps replacing it with a better one)   
You need to migrate the engine DOWN to get rid of its stuff.  It''s  
much simpler to drop in a new migration that migrates the engine to  
zero and be done with it.  Otherwise you have to figure out how to  
migrate down just the migrations for the engine without undoing the  
other migrations.  Madness!

There''s also the improbable but pathological case where a migration in
an engine has the same version number or class name as one in the main  
app or another engine.  Now that I''ve said that I realize that  
migrations in engines should be namespaced in the engine''s module to  
avoid class name collisions.



--
Josh Susser
http://blog.hasmanythrough.com



-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en.

Jeremy Evans

2010-Mar-08 06:08 UTC

head link

Re: some love for migrations

On Sun, Mar 7, 2010 at 8:28 PM, Josh Susser <josh@hasmanythrough.com>
wrote:> There''s also the improbable but pathological case where a
migration in an
> engine has the same version number or class name as one in the main app or
> another engine.  Now that I''ve said that I realize that migrations
in
> engines should be namespaced in the engine''s module to avoid class
name
> collisions.
You could also use Sequel''s recommended approach of anonymous classes
for migrations:

Class.new(ActiveRecord::Migration) do
  def self.up
    # ...
  end
  def self.down
    # ...
  end
end

Then you don''t have to worry about collisions.  In addition to the
engine issue, this also fixes the much more common error where people
would copy a previous migration file to create a new one, and would
forget to rename the migration class.

Jeremy

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en.

Chad Woolley

2010-Mar-08 06:52 UTC

head link

Re: some love for migrations

On Sun, Mar 7, 2010 at 6:49 PM, Jeremy Kemper <jeremy@bitsweat.net>
wrote:> I have a plugin with migrations for its own database. I want those to
> run and be managed in their own universe. This works well today using
> our existing tools: just override the connection class method on the
> migration.
Even if a plugin/engine manages its own database, and has nothing to
do with your app''s database, it still operates within the context of
your app and your app''s timeline (you explicitly upgrade the code, you
monkey patch it, etc).  Otherwise, if it has nothing to do with your
app, it is a separate app in itself, and should be managed
independently.  But since it *is* in the context of your app, it makes
sense to explicitly manage the plugin''s migrations within the timeline
of your app''s migrations.  You need to have explicit control over this
timeline, for all the reasons Josh mentioned in his response.

The migrations API should definitely support either approach - running
an engine''s migrations as a standalone app, or invoking them in the
context of a parent app''s migration timeline - and I hope
Josh''s patch
takes this into account.  This allows for the greatest flexibility,
but we must support both.

-- Chad

-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en.

Albert Llop

2010-Mar-08 08:17 UTC

head link

Re: some love for migrations

From the point of view of someone who''s been working with engines and
Pivotal''s Desert system for a few months now (on our own kind of CMS) I
agree completely with Josh. Managing migrations is already a little painful
in a team of developers, and when you add engines it''s even worse. The
desert approach of migrating the plugin when you want has been very helpful
and definatelly what we''d find most useful.
--
{ :name => "Albert Llop" }


On 8 March 2010 07:52, Chad Woolley <thewoolleyman@gmail.com> wrote:
> On Sun, Mar 7, 2010 at 6:49 PM, Jeremy Kemper <jeremy@bitsweat.net>
wrote:
> > I have a plugin with migrations for its own database. I want those to
> > run and be managed in their own universe. This works well today using
> > our existing tools: just override the connection class method on the
> > migration.
>
> Even if a plugin/engine manages its own database, and has nothing to
> do with your app''s database, it still operates within the context
of
> your app and your app''s timeline (you explicitly upgrade the code,
you
> monkey patch it, etc).  Otherwise, if it has nothing to do with your
> app, it is a separate app in itself, and should be managed
> independently.  But since it *is* in the context of your app, it makes
> sense to explicitly manage the plugin''s migrations within the
timeline
> of your app''s migrations.  You need to have explicit control over
this
> timeline, for all the reasons Josh mentioned in his response.
>
> The migrations API should definitely support either approach - running
> an engine''s migrations as a standalone app, or invoking them in
the
> context of a parent app''s migration timeline - and I hope
Josh''s patch
> takes this into account.  This allows for the greatest flexibility,
> but we must support both.
>
> -- Chad
>
> --
> You received this message because you are subscribed to the Google Groups
> "Ruby on Rails: Core" group.
> To post to this group, send email to rubyonrails-core@googlegroups.com.
> To unsubscribe from this group, send email to
>
rubyonrails-core+unsubscribe@googlegroups.com<rubyonrails-core%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/rubyonrails-core?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en.

Maybe Matching Threads

Search for more possibly parallel threads

Rails core - Mar 2010 - some love for migrations

some love for migrations

Re: some love for migrations

Re: some love for migrations

Re: some love for migrations

Re: some love for migrations

Re: some love for migrations

Re: some love for migrations

Maybe Matching Threads