Rails - Oct 2009 - Migrating Rails App to New Host/IP with No Downtime?

Hello Rails Community,

In the spirit of being agile with your Rails deployment and scaling,
can you recommend a process for moving your application to another
hosting provider where a new IP address will be assigned?

A wep app such as a Radiant CMS may be ok with a short downtime in the
early morning, but a paid SAS service where users call a Restful API
would not enjoy any downtime during a migration.

I''m in search of a solution that provides:
1. Little to no downtime to customers.
2. Considers Internet DNS propogation time for finding the new host
IP.
3. Considers database migration.

I''m not looking for a step-by-step, command line answer, only a higher-
level point of view that can point me in the proper direction.

Thank you.

-Jim

On Mon, Oct 12, 2009 at 11:24 AM, JimCifarelli
<cifarelli-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> In the spirit of being agile with your Rails deployment and scaling,
> can you recommend a process for moving your application to another
> hosting provider where a new IP address will be assigned?
> I''m in search of a solution that provides:
> 1. Little to no downtime to customers.
> 2. Considers Internet DNS propogation time for finding the new host
> IP.
> 3. Considers database migration.
I''d think #3 is the only question -- you can certainly have both sites
running simultaneously, providing zero apparent downtime.

How you handle replication of *data* depends on the details of your
app, what DB you''re using, etc.

FWIW,
-- 
Hassan Schroeder ------------------------
hassan.schroeder-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
twitter: @hassan

Thank you Hassan for your input.  I image the old site would remain up
for a few days as current users my still have old DNS caches pointing
to it.  I have not decided between MySQL or Postgres.  It sounds like
DB Replication is a must in developing a good practice for a safe
migration to new servers?

-Jim

On Oct 12, 2:37 pm, Hassan Schroeder
<hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On Mon, Oct 12, 2009 at 11:24 AM, JimCifarelli
<cifare...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > In the spirit of being agile with your Rails deployment and scaling,
> > can you recommend a process for moving your application to another
> > hosting provider where a new IP address will be assigned?
> > I''m in search of a solution that provides:
> > 1. Little to no downtime to customers.
> > 2. Considers Internet DNS propogation time for finding the new host
> > IP.
> > 3. Considers database migration.
>
> I''d think #3 is the only question -- you can certainly have both
sites
> running simultaneously, providing zero apparent downtime.
>
> How you handle replication of *data* depends on the details of your
> app, what DB you''re using, etc.
>
> FWIW,
> --
> Hassan Schroeder ------------------------
hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> twitter: @hassan

> 2. Considers Internet DNS propogation time for finding the new host
> IP.
With regard to this particular point, I can say that my experience in 
doing exactly what you are asking about has shown that there are quite a 
few ISPs that simply ignore TTLs below some minimum value.  For example, 
when I have done moves as you mention, we always start by cranking TTLs 
down to 5 minutes, a week before the scheduled move date.  At the move 
time, we turn up the new site, put the old site into "maintenance
mode"
(showing our standard "please excuse our dust while we do something 
behind the scenes" screen), then edit the DNS entries to point to the 
new IPs.

In theory, this should ensure no more than 5 minutes of unreachability 
for anyone, and, no worries about DB replication, because the old site 
is not accepting any new input.

In practice, I have watched in amusement and disgust as requests have 
continued to appear on the old IPs for a week or more.  My guess, 
although unverified, is that some caching nameservers (probably 
belonging to ISPs) simply ignore TTLs.  If memory serves correctly, it 
seemed like most of the requests were coming from Asia, so, maybe some 
ISPs there impose some minimum TTL value to throttle trans-oceanic DNS 
requests.

Don''t know for sure, but, that''s my experience, on more than
one
occasion, so, thought I''d throw that out there.

Best,

Danny
-- 
Posted via http://www.ruby-forum.com/.

On Mon, Oct 12, 2009 at 11:55 AM, JimCifarelli
<cifarelli-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I have not decided between MySQL or Postgres.  It sounds like
> DB Replication is a must in developing a good practice for a safe
> migration to new servers?
I would think if there''s the possibility of users continuing to hit the
old system for some time -- and I''ve seen just ridiculously long DNS
caching in the past -- that you would need something like a circular
(master <-> master) replication to keep primary keys from getting
reused. But that may depend on your ratio of writes to reads, etc.

If this were my project, I would set up a comparable configuration
with pair of test machines/VMWare instances and experiment with
different DB replication types and timing.

HTH,
-- 
Hassan Schroeder ------------------------
hassan.schroeder-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
twitter: @hassan

Thank you Danny and Hassan.  Your experience here is just what I was
seeking.  Great advice.

-Jim

On Oct 12, 7:24 pm, JimCifarelli
<cifare...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
.
>
> I''m in search of a solution that provides:
> 1. Little to no downtime to customers.
> 2. Considers Internet DNS propogation time for finding the new host
> IP.
> 3. Considers database migration.
>
> I''m not looking for a step-by-step, command line answer, only a
higher-
> level point of view that can point me in the proper direction.

You''ve had some good answers, here are some other ones:

- register a new domain new.foo.com and point it at where your new
hosting will be
- setup database replication between your 2 sites
- turn off mongrels on old site
- stop replication
- turn on the mongrels on new site
- install a redirect on old.foo.com that redirects people to
new.foo.com
- update old.foo.com to point to new.foo.com

You can lose the database replication step at the expense of some time
by just dumping the db to a file and copying it over once you take
your old site down (or do stuff like directly copy the database''s
files) - how practical that is depends on the size of your data.

We''re doing something similar at work at the moment.

Option two is almost the same as option 1 except that instead of doing
a redirect from old.foo.com you actually proxy traffic between the two
(so unlike option one, users never see new.foo.com in their address
bar)


Fred

Thank you Fred for your response.

1. I like the use of the "temporary" new.foo.com domain name and
redirect.
Do you know how well Ruby libraries like Net::HTTP, ActiveResource, or
RestClient will handle 302 Redirects?
I''m concerned that user''s "posts" to the application
will fail when
confronted with a redirect.

2. Great idea on the transparent forward proxy.  I''ll look into this.

3. My end goals include phasing out the old server (and stop paying
for it)...just keeping it running long enough to validate users
ability to transition to the new server without issue.  For a fair
amount of time, I can also monitor for continued API use of the old
server and perhaps warn valid user attempts of their DNS issues.

Thank you again Fred for all the time and quality answers you
contribute to this community.

-Jim


On Oct 13, 4:25 am, Frederick Cheung
<frederick.che...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On Oct 12, 7:24 pm, JimCifarelli
<cifare...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> .
>
>
>
> > I''m in search of a solution that provides:
> > 1. Little to no downtime to customers.
> > 2. Considers Internet DNS propogation time for finding the new host
> > IP.
> > 3. Considers database migration.
>
> > I''m not looking for a step-by-step, command line answer, only
a higher-
> > level point of view that can point me in the proper direction.
>
> You''ve had some good answers, here are some other ones:
>
> - register a new domain new.foo.com and point it at where your new
> hosting will be
> - setup database replication between your 2 sites
> - turn off mongrels on old site
> - stop replication
> - turn on the mongrels on new site
> - install a redirect on old.foo.com that redirects people to
> new.foo.com
> - update old.foo.com to point to new.foo.com
>
> You can lose the database replication step at the expense of some time
> by just dumping the db to a file and copying it over once you take
> your old site down (or do stuff like directly copy the database''s
> files) - how practical that is depends on the size of your data.
>
> We''re doing something similar at work at the moment.
>
> Option two is almost the same as option 1 except that instead of doing
> a redirect from old.foo.com you actually proxy traffic between the two
> (so unlike option one, users never see new.foo.com in their address
> bar)
>
> Fred

On Oct 13, 5:00 pm, JimCifarelli
<cifare...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> Thank you Fred for your response.
>
> 1. I like the use of the "temporary" new.foo.com domain name and
> redirect.
> Do you know how well Ruby libraries like Net::HTTP, ActiveResource, or
> RestClient will handle 302 Redirects?
> I''m concerned that user''s "posts" to the
application will fail when
> confronted with a redirect.
I''m not sure. It''s certainly possible that some of your api
clients
wouldn''t understand them

Fred
>
> 2. Great idea on the transparent forward proxy.  I''ll look into
this.
>
> 3. My end goals include phasing out the old server (and stop paying
> for it)...just keeping it running long enough to validate users
> ability to transition to the new server without issue.  For a fair
> amount of time, I can also monitor for continued API use of the old
> server and perhaps warn valid user attempts of their DNS issues.
>
> Thank you again Fred for all the time and quality answers you
> contribute to this community.
>
> -Jim
>
> On Oct 13, 4:25 am, Frederick Cheung
<frederick.che...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> wrote:
>
>
>
> > On Oct 12, 7:24 pm, JimCifarelli
<cifare...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > .
>
> > > I''m in search of a solution that provides:
> > > 1. Little to no downtime to customers.
> > > 2. Considers Internet DNS propogation time for finding the new
host
> > > IP.
> > > 3. Considers database migration.
>
> > > I''m not looking for a step-by-step, command line answer,
only a higher-
> > > level point of view that can point me in the proper direction.
>
> > You''ve had some good answers, here are some other ones:
>
> > - register a new domain new.foo.com and point it at where your new
> > hosting will be
> > - setup database replication between your 2 sites
> > - turn off mongrels on old site
> > - stop replication
> > - turn on the mongrels on new site
> > - install a redirect on old.foo.com that redirects people to
> > new.foo.com
> > - update old.foo.com to point to new.foo.com
>
> > You can lose the database replication step at the expense of some time
> > by just dumping the db to a file and copying it over once you take
> > your old site down (or do stuff like directly copy the
database''s
> > files) - how practical that is depends on the size of your data.
>
> > We''re doing something similar at work at the moment.
>
> > Option two is almost the same as option 1 except that instead of doing
> > a redirect from old.foo.com you actually proxy traffic between the two
> > (so unlike option one, users never see new.foo.com in their address
> > bar)
>
> > Fred

On Tue, Oct 13, 2009 at 10:17 AM, Frederick Cheung
<frederick.cheung-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> 1. I like the use of the "temporary" new.foo.com domain name
and
>> redirect.
>> Do you know how well Ruby libraries like Net::HTTP, ActiveResource, or
>> RestClient will handle 302 Redirects?
>> I''m concerned that user''s "posts" to the
application will fail when
>> confronted with a redirect.
>
> I''m not sure. It''s certainly possible that some of your
api clients
> wouldn''t understand them
Yeah, redirects work great with human-oriented browser clients, but
I''d be extremely surprised to see them work with an API client even
for a GET, and I''d bet not at all for POST data.

Using a proxy is probably a better idea if you want to avoid the DB
circular replication complexity.

-- 
Hassan Schroeder ------------------------
hassan.schroeder-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
twitter: @hassan

Thanks Fred and Hassan.

For anyone following this thread, here'' a tidy means for performing
the proxy with Apache:
http://blog.vacs.fr/index.php?post/2009/01/25/21-transparent-web-server-migration-with-apache-proxy

I''m not sure I''ll want to bring services down in order to
dump/restore
the production db onto the new server, so I''m investigating RubyRep
for an easy method of db replication.  They have a short screen cast
to watch if interested (http://www.rubyrep.org/)

So we''re down to just a few "easy" steps:

1. A week ahead of the scheduled migration, throttle down the TTL for
your DNS entry
2. Get new server up and running with db/application/.etc (production
ready)
3. Install/Configure/Execute RubyRep on old server to replicate
production data to the new server
4. Restart old server''s apache with a transparent proxy configuration
to new server''s IP
5. Update DNS to point to new server''s IP address with restored 3600,
7200 sec TTL
6. Monitor old server''s use over a few days unil comfortable with
shutting it down.

Possible Flaws:
1.  A user flip flops between servers and the replication to the new
server hasn''t completed.  Assuming that a user hits the new server
with their DNS resolved to the new IP, I can''t see that they again hit
the old server.
2. RubyRep having any trouble or big latency working with a slave
server outside it''s local network.


Thanks again.
-Jim


On Oct 13, 1:43 pm, Hassan Schroeder
<hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On Tue, Oct 13, 2009 at 10:17 AM, Frederick Cheung
>
> <frederick.che...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> 1. I like the use of the "temporary" new.foo.com domain
name and
> >> redirect.
> >> Do you know how well Ruby libraries like Net::HTTP,
ActiveResource, or
> >> RestClient will handle 302 Redirects?
> >> I''m concerned that user''s "posts" to
the application will fail when
> >> confronted with a redirect.
>
> > I''m not sure. It''s certainly possible that some of
your api clients
> > wouldn''t understand them
>
> Yeah, redirects work great with human-oriented browser clients, but
> I''d be extremely surprised to see them work with an API client
even
> for a GET, and I''d bet not at all for POST data.
>
> Using a proxy is probably a better idea if you want to avoid the DB
> circular replication complexity.
>
> --
> Hassan Schroeder ------------------------
hassan.schroe...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> twitter: @hassan