First, long-winded thought #1:
Today I was playing with a Rails app (Typo) and wanted to add a
migration to support a new column in one of the data tables -- purely
for some functionality that''s almost certainly only of interest to me
in
my little Weblogging Kingdom. I immediately realized that if I added
migration #37, with whatever name, that Tobi is likely to come along
next week and add his own migration #37, which is going to conflict with
mine.
In a way, the "total order" that the increasing sequence of migration
numbers defines ends up working against collaboration.
The first instant and egregiously bad idea that occurred to me was that
maybe we should put some space between numbers (like in BASIC "10 print
"hello" "20 goto 10"). No need to reminisce about BASIC.
I thought for a second about it and decided that what the linear
migration numbering is trying to do is resolve dependencies. With one
author (or a tightly coordinated team) a total order is fine, and the
easiest total order is one that could be put in the filenames of the
migrations themselves, viz., the natural numbers we have now.
When the development isn''t so controlled (or, in my case and certainly
others, when local installations wish to add a fork for custom
functionality) the total order imposes integration problems, and using
the natural numbers to implement the order makes things worse (i.e., I
can guarantee Tobi''s going to stomp on any custom migrations I put in).
In such situations it would be nice if we could use a partial order
instead.
I thought about this for a second and realized that we''re already
making
use of partial orders in this way in the project: our Rake tasks
(default and in lib/).
Am I suggesting that we use rake tasks for migrations? No. Just that
Rake is a well-thought out example for how me might use Ruby (which
we''re already using in Migrations) to handle dependent tasks. Perhaps
a
DSL for specifying migration dependencies.
Anyway, I started brainstorming that if someone wanted to add a custom
migration that depended upon migration #36 they''d write their
migration,
not use a number for the naming of it, and at the top of the file use a
DSL statement similar to a Rake dependency rule to say "hey, I''m
dependent on this other migration".
Then I realized that the schema_info table and associated logic rely on
the natural-number total ordering for their succinct assessment of
whether anything needs to be done on a ''rake migrate''. Things
begin to
get complicated (the obvious solution being to go from 1 row in
schema_info to 1 row per migration file) and I''m not clear that
it''s
worth pursuing. I.e., is the problem just an annoyance for me or does
it affect others, do they care, are there other solutions, etc.?
So, I figured I''d pass this along in case this was of interest to
someone else.
Then I realized I had another thing to go on about (hopefully
not-so-longwinded thought #2):
I''ve got a chunk of Rails code that was written back around Rails 0.0.0
or so
that I want to upgrade. No big deal, really, that''s just file
manipulations.
Having used this thing (it''s the so-called "Accountomatic")
for years, starting
back in PHP-land, and migrating to Rails, etc., I''ve come to realize
that the
data model needs certain specific changes. So the tack I''m going to
take is to
put the thing onto Migrations and then migrate the data step by step to the new
model. Cool.
Actually, I''ve already started this process, and I''ve run into
a headache. I
think /I/ have an out, but it points to a bigger issue. I think this is going
to be a headache for a number of people from time to time so it''s
probably
worth covering.
I''ve got a number of accounting models (Account, Person, Tran
(transaction),
Budget, Period, etc.) in the system, all with data in a live database. I need
to refactor the data model so that instead of simply having The Simplest Thing
That Could Possibly Work Circa 1999, I''ve instead got a reasonable data
model.
So a Tran object which stores a "transaction" (which should be rightly
called
an "entry" since it''s not transactional in any real sense),
meaning an amount,
a time, a Person, and an Account will be turned into two Entry objects which
are linked to a Transaction object -- the goal being to go to a double-entry
bookkeeping system with multi-legged transactions, memo accounts, etc.
So... I begin writing the migrations one by one to start transforming the data
I''ve got into the data I want. Something comes up, though.
If I''m going to add models (Entry, Transaction, etc.) to replace old
models
(Tran in this case, but there are other things happening to Account) then I
should presumably:
(1) add the new model classes so I can do
def self.up
# ...
Trans.find(:all).each do |t|
Entry.create ...
Transaction.create ...
end
end
(2) get rid of the old model classes
But, here''s the problem: we make these migrations so that we can
upgrade
systems that can be anywhere along the upgrade path. Let''s say at
revision
#200 I add the migrations to convert over Tran -> Entry + Transaction. Then
at
revision #201 I go ahead and get rid of the Tran model class, since
it''s no
longer needed. Then I go on for a few more revisions doing coding, adding
migrations, etc.
Now, a month later some user comes along and decides to upgrade his install.
He''s at revision #199, which was the stable revision for months (true
in this
case, modulo exact numbering). He does a pull and gets himself up to current
at revision #215. He runs ''rake migrate''.
Boom.
What happened?
Well, the first migration says "Trans.find", but Trans is gone, so
that''s not
going to fly.
That''s really sort of the simple case -- imagine if, instead of doing
model
replacement I were adding a ''before_save'' to a model, and that
before_save was
doing some work on a column (for instance you might have a
''body_html'' column
to store rendered content, or a cache of some type...). Well, that column is
going to be added by a migration at some point (basically, concurrent with the
addition of the before_save). But, if you''re back on an old revision,
do an
svn pull, and then run rake migrate, if any of the earlier migrations do things
to the model with the new before_save on it... Boom. That column''s
not there
yet, but the before_save is already in the code, and it needs that column.
(I''m suspicious that Typo has this particular problem, fwiw, depending
on when
the end user does his/her pulls.)
If the end user insteaad pulls down every SVN revision one by one (#200, #201,
#202, ...) and runs ''rake migrate'' each time then s/he should
probably be safe,
otherwise the trunk code can get out of sync with the earlier migrations,
causing problems.
One way the developer can deal with this (what I''ve been muddling
through in a
tough spot in my case, in fact) is to do things like:
def self.up
# ...
ActiveRecord::Base.connection.select_all(...).each do |foo|
execute "insert into bar (...) values (...)"
end
Blech.
But, this doesn''t catch everything (the before_save example is
unhelped, e.g.).
Ultimately, the developer can''t (and shouldn''t have to)
predict what the future
is going to bring, and shouldn''t have to code around this sort of
problem.
Best,
Rick
--
http://www.rickbradley.com MUPRN: 64
| that they collect the
random email haiku | sales tax but don''t ever
| pay it to our state.
> So, I figured I''d pass this along in case this was of interest to > someone else.I appreciate your thinking on this, Rick, but I think that the trouble migrations display in this case is simply a symptom of infrequent integration. Just like you get svn conflicts if you stay off the committing for too long. I think the solution is simply to integrate more often, not to add additional software.> But, this doesn''t catch everything (the before_save example is unhelped, e.g.). > Ultimately, the developer can''t (and shouldn''t have to) predict what the future > is going to bring, and shouldn''t have to code around this sort of problem.If this turns out to be a real problem for you (I''ve never seen it in the wild), I''d encourage tying marrying migrations with svn updates. So you could have a script that does "svn co rev 1; rake migrate; svn co rev 2; rake migrate". If that''s something that turns out to be useful, please do turn it into a plugin. -- David Heinemeier Hansson http://www.loudthinking.com -- Broadcasting Brain http://www.basecamphq.com -- Online project management http://www.backpackit.com -- Personal information manager http://www.rubyonrails.com -- Web-application framework
> One way the developer can deal with this (what I''ve been muddling through in a > tough spot in my case, in fact) is to do things like: > > def self.up > # ... > ActiveRecord::Base.connection.select_all(...).each do |foo| > execute "insert into bar (...) values (...)" > end > > Blech. > > But, this doesn''t catch everything (the before_save example is unhelped, e.g.). > Ultimately, the developer can''t (and shouldn''t have to) predict what the future > is going to bring, and shouldn''t have to code around this sort of problem.RIck, I''ve been running into this with Mephisto, a little blogging app I''ve been writing on the side. After a pretty intense weekend of hacking, I found myself doing a lot of db restructuring. Obviously, the earlier migrations weren''t working, because the model and or table has been renamed. To work around this, I just started creating temporary models per migration. I find that much nicer than reverting to ActiveRecord::Base.connection for my delicate sql needs. It might be best to create the temporary model classes inside the migration class to keep them from clashing. This temp model gives me a nicer way to manage the data, and frees me from worrying about filters, validations, changing method names, etc. I may throw together some task that takes my initial schema and runs through all the migrations eventually. I think this kind ability is important for packaged apps, so I''ll certainly share any progress I make in Mephisto. Take a look: http://techno-weenie.net/svn/projects/mephisto/trunk/db/migrate/ -- Rick Olson http://techno-weenie.net
* Rick Olson (technoweenie@gmail.com) [060220 01:15]:> To work around this, I just started creating temporary models per > migration. I find that much nicer than reverting to > ActiveRecord::Base.connection for my delicate sql needs. It might be > best to create the temporary model classes inside the migration class > to keep them from clashing. This temp model gives me a nicer way to > manage the data, and frees me from worrying about filters, > validations, changing method names, etc.Rick -- That''s a really slick way to handle this problem! I''m going to have to give that a try. That makes my day. So -- any similarly slick ideas on how to handle the "I want to add a local migration without getting stomped by the upstream" issue I was jabbering about at the beginning? Thanks, Rick -- http://www.rickbradley.com MUPRN: 497 | all those negative random email haiku | vibes get you? <fade out> Seen | both movies. They rock.
> Rick -- > That''s a really slick way to handle this problem! I''m going to have to > give that a try. That makes my day. > > So -- any similarly slick ideas on how to handle the "I want to add a > local migration without getting stomped by the upstream" issue I was > jabbering about at the beginning? > > Thanks, > Rick > -- > http://www.rickbradley.com MUPRN: 497 > | all those negative > random email haiku | vibes get you? <fade out> Seen > | both movies. They rock.That''s a tricky issue. Off the top of my head, I''d suggest creating an alternate migration directory for local migrations, as well as a separate table for that. I can see something like that used for an extensible app, such as Typo. Each component or plugin would have it''s own directory of migrations, with its own row in a plugin_migrations table to keep the current version number. "Uninstall" a plugin would run those backwards to completely deactivate it from the system, while "installing" would run some kind of master migration (similar to the schema.rb for your application). Even if you could utilize migrations and somehow sneak a change or two in, how would you roll your changes back? You''d have to rollback all the recent typo changes, then your local changes, and finally re-run the typo migrations. Yuck. That''s why I suggested a separate location for the localized migrations. It sounds like Too Much Software for Rails to handle out of the box. It might make a good plugin, however. I think Rails Engines accomplishes this, but I''m not sure how. -- Rick Olson http://techno-weenie.net
What if, instead of a simple integer sequence, migrations used timestamps? -Kyle _______________________________________________ Rails-core mailing list Rails-core@lists.rubyonrails.org http://lists.rubyonrails.org/mailman/listinfo/rails-core
On 2/20/06, Kyle Maxwell <kyle@kylemaxwell.com> wrote:> What if, instead of a simple integer sequence, migrations used timestamps?Could we do that in a backwards compatible way? The problem of local, uncommited migrations isn''t something that''s worth breaking backwards compatibility -- Cheers Koz
* Michael Koziarski (michael@koziarski.com) [060220 14:09]:> On 2/20/06, Kyle Maxwell <kyle@kylemaxwell.com> wrote: > > What if, instead of a simple integer sequence, migrations used timestamps? > > Could we do that in a backwards compatible way? The problem of local, > uncommited migrations isn''t something that''s worth breaking backwards > compatibilityWell, if it were done as an epoch timestamp (i.e., number of seconds since 1/1/1970) it would probably be pretty simple. Rick -- http://www.rickbradley.com MUPRN: 640 | the discipline of random email haiku | the folks that wrote it than | the language itself.
On 2/20/06, Kyle Maxwell <kyle@kylemaxwell.com> wrote:> What if, instead of a simple integer sequence, migrations used timestamps?The problem is the dependence tree. If you do it with timestamps, by inserting your own migrations you may change things later migrations depend on. Similarly, we lose the simplicity of VER#_description because you wouldn''t be able to use filesystem timestamps which would break in a variety of cases. Kev
* Kevin Clark (kevin.clark@gmail.com) [060220 14:54]:> The problem is the dependence tree. If you do it with timestamps, by > inserting your own migrations you may change things later migrations > depend on. Similarly, we lose the simplicity of VER#_description > because you wouldn''t be able to use filesystem timestamps which would > break in a variety of cases.The dependence tree is actually linear (hence one of the problems I was dragging on about :-), but your point is taken. If we were just talking code then the conflicts are the same as an SVN conflict resolution, which happens from time to time -- but we''re not, the database is involved. With migrations one would have to back out the migration (i.e., rake migrate to a prior version) then sort out the problems, then migrate forward. Actually the more I think about it the more it really is like an SVN conflict merge. On your second point I think you''re right that there''s a pretty baby in the bathwater, being able to do rake migrate VERSION=n and having the file''s named with the version numbers are both good features. I''m not necessarily recommending the adoption of an epoch timestamp version number instead of a natural number version, but I might toy with it at home if I can get a chance soon. Rick -- http://www.rickbradley.com MUPRN: 738 | and grab content, which random email haiku | you see happening in the | "window" to the right.
Rails engines do indeed let you migrate sections of your application independently. It''s pretty simple to achieve; the implementation can be seen in these files: http://opensvn.csie.org/rails_engines/engines/trunk/lib/engines/migration_extensions.rb http://opensvn.csie.org/rails_engines/engines/trunk/lib/engines/active_record_extensions.rb http://opensvn.csie.org/rails_engines/engines/trunk/tasks/engines.rake It might make a good starting point for some custom work (and of course any improvements will be welcomed back into the plugin). - james On 2/20/06, Rick Olson <technoweenie@gmail.com> wrote:> > Rick -- > > That''s a really slick way to handle this problem! I''m going to have to > > give that a try. That makes my day. > > > > So -- any similarly slick ideas on how to handle the "I want to add a > > local migration without getting stomped by the upstream" issue I was > > jabbering about at the beginning? > > > > Thanks, > > Rick > > -- > > http://www.rickbradley.com MUPRN: 497 > > | all those negative > > random email haiku | vibes get you? <fade out> Seen > > | both movies. They rock. > > That''s a tricky issue. Off the top of my head, I''d suggest creating > an alternate migration directory for local migrations, as well as a > separate table for that. I can see something like that used for an > extensible app, such as Typo. Each component or plugin would have > it''s own directory of migrations, with its own row in a > plugin_migrations table to keep the current version number. > "Uninstall" a plugin would run those backwards to completely > deactivate it from the system, while "installing" would run some kind > of master migration (similar to the schema.rb for your application). > > Even if you could utilize migrations and somehow sneak a change or two > in, how would you roll your changes back? You''d have to rollback all > the recent typo changes, then your local changes, and finally re-run > the typo migrations. Yuck. That''s why I suggested a separate > location for the localized migrations. > > It sounds like Too Much Software for Rails to handle out of the box. > It might make a good plugin, however. I think Rails Engines > accomplishes this, but I''m not sure how. > > -- > Rick Olson > http://techno-weenie.net > _______________________________________________ > Rails-core mailing list > Rails-core@lists.rubyonrails.org > http://lists.rubyonrails.org/mailman/listinfo/rails-core >-- * J * ~
Rick Olson wrote:>>One way the developer can deal with this (what I''ve been muddling through in a >>tough spot in my case, in fact) is to do things like: >> >> def self.up >> # ... >> ActiveRecord::Base.connection.select_all(...).each do |foo| >> execute "insert into bar (...) values (...)" >> end >> >>Blech. >> >>But, this doesn''t catch everything (the before_save example is unhelped, e.g.). >>Ultimately, the developer can''t (and shouldn''t have to) predict what the future >>is going to bring, and shouldn''t have to code around this sort of problem. > > > RIck, I''ve been running into this with Mephisto, a little blogging app > I''ve been writing on the side. After a pretty intense weekend of > hacking, I found myself doing a lot of db restructuring. Obviously, > the earlier migrations weren''t working, because the model and or table > has been renamed. > > To work around this, I just started creating temporary models per > migration. I find that much nicer than reverting to > ActiveRecord::Base.connection for my delicate sql needs. It might be > best to create the temporary model classes inside the migration class > to keep them from clashing. This temp model gives me a nicer way to > manage the data, and frees me from worrying about filters, > validations, changing method names, etc. > > I may throw together some task that takes my initial schema and runs > through all the migrations eventually. I think this kind ability is > important for packaged apps, so I''ll certainly share any progress I > make in Mephisto. > > Take a look: http://techno-weenie.net/svn/projects/mephisto/trunk/db/migrate/That''s what I do too, like in http://techno-weenie.net/svn/projects/mephisto/trunk/db/migrate/012_rename_categories_to_sections.rb The only difference, is that I use two classes instead of one, one for the up and one for the down, in case they are different. And I also give them a unique name that uses the migration''s name. This is in case all the migrations are run from 1-N to allow for the possibility that the same table will be modified in different migrations. class Section_Migrate003Up < ActiveRecord::Base set_table_name ''output_state'' end class Section_Migrate003Down < ActiveRecord::Base set_table_name ''output_state'' end Regards, Blair -- Blair Zajac, Ph.D. <blair@orcaware.com> Subversion training, consulting and support http://www.orcaware.com/svn/
> That''s what I do too, like in > > http://techno-weenie.net/svn/projects/mephisto/trunk/db/migrate/012_rename_categories_to_sections.rb > > The only difference, is that I use two classes instead of one, one for the up > and one for the down, in case they are different. And I also give them a unique > name that uses the migration''s name. This is in case all the migrations are run > from 1-N to allow for the possibility that the same table will be modified in > different migrations. > > class Section_Migrate003Up < ActiveRecord::Base > set_table_name ''output_state'' > end > > class Section_Migrate003Down < ActiveRecord::Base > set_table_name ''output_state'' > end > > Regards, > Blair > > -- > Blair Zajac, Ph.D. > <blair@orcaware.com> > Subversion training, consulting and support > http://www.orcaware.com/svn/Good point. I realize my OldAttachment model name wasn''t optimal in that case. I was going to try something like this in my next migration: class RenameCategoriesToSections < ActiveRecord::Migration class Attachment < AR::Base set_table_name ''foo'' end def self.up ... end end That way I don''t need weird names, and I can still access the model as ''Attachment.'' Though, I like the idea of Up and Down migrations too. I''ll try it out next time the issue comes up. -- Rick Olson http://techno-weenie.net
On Mon, 2006-02-20 at 00:14 -0600, Rick Olson wrote:> > But, this doesn''t catch everything (the before_save example is unhelped, e.g.). > > Ultimately, the developer can''t (and shouldn''t have to) predict what the future > > is going to bring, and shouldn''t have to code around this sort of problem. > > To work around this, I just started creating temporary models per > migration. I find that much nicer than reverting to > ActiveRecord::Base.connection for my delicate sql needs.> Take a look: http://techno-weenie.net/svn/projects/mephisto/trunk/db/migrate/FWIW, I also used this technique for Typo''s migrations. It works really well. http://www.typosphere.org/trac/ticket/680 (I got bogged down with some odd typoisms around the teens... I''ll clean it up and hopefully merge it this week).
Jumping in on the migration issue, this also happens with customized "products" We sell a package to a customer that requires customer specific modifications to the database. The core product continues development. At some point in the future the customer requests a change/improvment that is in the core product. We want to remerge to core to the customer -- yes as DHH mentioned this is an infrequent integration issue, but it happens often enough in practice to be a concern (if you sell products and not services). One solution would be to allow local migrations, tied to existing migrations -- possibly in like named subdirectories? We would still have the concept of individual version numbers, but also support minor version numbers (the default being the most recent?) if I migrate to version 4 I know all local modifications are in place as well. The version table would need a second column to store the local minor version number. This opens a door migration failures (a local migration adds a column that is later added in "trunk" migration), but would certainly provide value for us. Would a patch providing this functionality be accepted? Additional levels could be argued, but would be more complex than I have encountered in practice. pth