Mongrel users - Oct 2006 - [Slightly OT] Do you use virtualization in production ?

Hi!

I''ve been using parallels, vmware player, or virtual pc (did not try
xen
yet) for development, testing and experimenting with new platforms (and I
must say I love them - copy paste a few files, and I have a clean deployment
box etc)

Since the RailsConf and various talks I''ve seen (like Jason
Hoffmann''s
http://svn.joyent.com/public/JasonHoffman-EuroRailsConf.pdf , or James
Duncan Davidson), I''m willing to apply the benefits to production.

is there anyone here actually using virtualization techniques in production
? (even for single machine setups?)

any feedback would be most welcome!

thanks!

Thibaut
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061002/d430924e/attachment.html

Virtualization rocks!  No problems in our production environment.  I strongly
suggest it for any website, unless you''re hosting Google or Yahoo.

 

We''re using VMware''s ESX 3.0  in production.  I have 37
production VMs spread across 4 ESX3 hosts.  We also have a SAN which allows us
to VMotion (hot migrate) running VMs from server to server.  Most of my VMs are
Windows but I do have 3 linux VMs with CentOS 4.3 running pound and mongrel, as
well as MySQL.  Everything is running very well.  If you don''t have the
cash for ESX 3, you might consider running VMware Server.  VMWare Server is
free, but the performance is less and you can''t get as many VMs on a
single box and no Vmotion :-(.

 

The biggest benefit we see from the VMware is Disaster Recovery options
available.  I also love using the snapshot feature prior to major software
upgrades.  If the upgrade introduces major errors, you can then shutdown the VM,
revert to the earlier snapshot.

 

The one problem I had initially was with time keeping.  The below link is what
fixed my problems.

 

KB Doc ID: 1420 is recommended reading for time sync issues with Linux guest on
vmware.
http://kb.vmware.com/vmtnkb/search.do?cmd=displayKC&docType=kc&externalId=1420&sliceId=SAL_Public

 

In addition to instructions in the above KB document, I also needed to add a one
line script to  /etc/cron.hourly/,  "ntpdate
myntpserver.mydomain.com".  This corrects the remaining 1-2 seconds that
I''m ahead each hour.

 

 

 

 

________________________________

From: mongrel-users-bounces at rubyforge.org [mailto:mongrel-users-bounces at
rubyforge.org] On Behalf Of Thibaut Barr?re
Sent: Monday, October 02, 2006 6:41 AM
To: mongrel-users at rubyforge.org
Subject: [Mongrel] [Slightly OT] Do you use virtualization in production ?

 

Hi!

I''ve been using parallels, vmware player, or virtual pc (did not try
xen yet) for development, testing and experimenting with new platforms (and I
must say I love them - copy paste a few files, and I have a clean deployment box
etc)

Since the RailsConf and various talks I''ve seen (like Jason
Hoffmann''s http://svn.joyent.com/public/JasonHoffman-EuroRailsConf.pdf
, or James Duncan Davidson), I''m willing to apply the benefits to
production.

is there anyone here actually using virtualization techniques in production ?
(even for single machine setups?)

any feedback would be most welcome!

thanks!

Thibaut
Confidentiality Notice: This e-mail message, including any attachments, is for
the use of the intended recipient(s), and may contain legally privileged and
confidential information. Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient, please do not
read, copy, or use it, and do not disclose it to others. Please notify the
sender of the delivery error by replying to this message, then delete it from
your system, and destroy all copies. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061002/df810c06/attachment-0001.html

On Oct 2, 2006, at 4:40 AM, Thibaut Barr?re wrote:
> Hi!
>
> I''ve been using parallels, vmware player, or virtual pc (did not  
> try xen yet) for development, testing and experimenting with new  
> platforms (and I must say I love them - copy paste a few files, and  
> I have a clean deployment box etc)
>
> Since the RailsConf and various talks I''ve seen (like Jason  
> Hoffmann''s http://svn.joyent.com/public/JasonHoffman- 
> EuroRailsConf.pdf , or James Duncan Davidson), I''m willing to
apply
> the benefits to production.
>
> is there anyone here actually using virtualization techniques in  
> production ? (even for single machine setups?)
>
> any feedback would be most welcome!
>
> thanks!
>
> Thibaut

	Yeah now that I have been way into Xen, I wouldn''t setup a new  
server without it. Even if you only have one server I would put Xen  
on it, even if its just one huge VM that fills the entire server. The  
performance overhead of Xen is about 5% compared to raw hardware  
which is really nothing compared to  the gains you get by virtualizing.

	It makes it so much easier to scale later. Move your VM''s from  
machine to machine. And separate concerns into their own VM''s.  
Seriously I can''t recommend it enough. We have built our entire new  
business on top of Xen. http://engineyard.com . Its only for ruby/ 
rails hosting and everything runs in Xen. Xen is very cool stuff.


-Ezra

On 10/2/06, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
>
> On Oct 2, 2006, at 4:40 AM, Thibaut Barr?re wrote:
>
> > Hi!
> >
> > I''ve been using parallels, vmware player, or virtual pc (did
not
> > try xen yet) for development, testing and experimenting with new
> > platforms (and I must say I love them - copy paste a few files, and
> > I have a clean deployment box etc)
> >
> > Since the RailsConf and various talks I''ve seen (like Jason
> > Hoffmann''s http://svn.joyent.com/public/JasonHoffman-
> > EuroRailsConf.pdf , or James Duncan Davidson), I''m willing to
apply
> > the benefits to production.
> >
> > is there anyone here actually using virtualization techniques in
> > production ? (even for single machine setups?)
> >
> > any feedback would be most welcome!
> >
> > thanks!
> >
> > Thibaut
>
>
>         Yeah now that I have been way into Xen, I wouldn''t setup a
new
> server without it. Even if you only have one server I would put Xen
> on it, even if its just one huge VM that fills the entire server. The
> performance overhead of Xen is about 5% compared to raw hardware
> which is really nothing compared to  the gains you get by virtualizing.
>
>         It makes it so much easier to scale later. Move your VM''s
from
> machine to machine. And separate concerns into their own VM''s.
> Seriously I can''t recommend it enough. We have built our entire
new
> business on top of Xen. http://engineyard.com . Its only for ruby/
> rails hosting and everything runs in Xen. Xen is very cool stuff.
>
>
Ezra, one thing that still bug me is IP handling. If i have a live
server and decide to do a hot migration to a newer box, how it
manages, and how much downtime will get to the IP switches ges
notified of the migration? (in ms resolution)

Or i''m forced to have a load balancer that handle the problem for me?
> -Ezra
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

On 10/2/06, Luis Lavena <luislavena at gmail.com> wrote:
> Ezra, one thing that still bug me is IP handling. If i have a live
> server and decide to do a hot migration to a newer box, how it
> manages, and how much downtime will get to the IP switches ges
> notified of the migration? (in ms resolution)
>
> Or i''m forced to have a load balancer that handle the problem for
me?
If you''re doing it right, the switches should notice within a few
milliseconds.  Any reasonable virtualization setup will have to assign
a virtual MAC address to each virtual ethernet interface, and the MAC
address should transfer between machines along with the rest of the
VM.  So, assuming that both machines are on the same L2 network, it''s
just a matter of the new machine sending an Ethernet frame or two out
its interface using the virtual MAC address as the source address.  To
the switch, it''ll look like the virtual server was unplugged from one
port and plugged into the new one, which isn''t a big deal. 
I''d be
amazed if VMWare didn''t have code in it to explicitly trip this, but
even without it it''ll happen almost instantly.


Scott

Hey Ezra,

What other gains do you get with virtualizing?

Also, would Xen/virtualizing make these easier?
- restoring root compromised servers
- transferring everything from an old to new server --
I do this periodically and cringe at having to
reinstall everything

Thanks,
Joe

> 	Yeah now that I have been way into Xen, I wouldn''t
setup a new  
> server without it. Even if you only have one server
I would put Xen  
> on it, even if its just one huge VM that fills the
entire server. The  
> performance overhead of Xen is about 5% compared to
raw hardware  
> which is really nothing compared to  the gains you
get by virtualizing.
> 
> 	It makes it so much easier to scale later. Move
your VM''s from  
> machine to machine. And separate concerns into their
own VM''s.  
> Seriously I can''t recommend it enough. We have built
our entire new  
> business on top of Xen. http://engineyard.com . Its
only for ruby/ 
> rails hosting and everything runs in Xen. Xen is
very cool stuff.
>
> -Ezra

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

On 10/3/06, Scott Laird <laird at google.com>
wrote:
> On 10/2/06, Luis Lavena <luislavena at gmail.com> wrote:
>
> > Ezra, one thing that still bug me is IP handling. If i have a live
> > server and decide to do a hot migration to a newer box, how it
> > manages, and how much downtime will get to the IP switches ges
> > notified of the migration? (in ms resolution)
> >
> > Or i''m forced to have a load balancer that handle the problem
for me?
>
> If you''re doing it right, the switches should notice within a few
> milliseconds.  Any reasonable virtualization setup will have to assign
> a virtual MAC address to each virtual ethernet interface, and the MAC
> address should transfer between machines along with the rest of the
> VM.  So, assuming that both machines are on the same L2 network,
it''s
> just a matter of the new machine sending an Ethernet frame or two out
> its interface using the virtual MAC address as the source address.  To
> the switch, it''ll look like the virtual server was unplugged from
one
> port and plugged into the new one, which isn''t a big deal. 
I''d be
> amazed if VMWare didn''t have code in it to explicitly trip this,
but
> even without it it''ll happen almost instantly.
>
Thank you for your answer Scott, I''ll investigate Xen further on my
spare time the following months.

Regards,
>
> Scott
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

Hey Joe-

On Oct 3, 2006, at 6:29 PM, Joe Ruby wrote:
> Hey Ezra,
>
> What other gains do you get with virtualizing?
>
> Also, would Xen/virtualizing make these easier?
> - restoring root compromised servers
Yup, you can take snapshots of a VM and restore back to that point in  
time without a fuss. Of course you have to set this up and its up to  
you how often you take snapshots is up to you.
> - transferring everything from an old to new server --
Yup its pretty much just a few command lines to move a VM from one  
physical box to another physical box *while its running*

> I do this periodically and cringe at having to
> reinstall everything
>
> Thanks,
> Joe
>
-Ezra

On 10/3/06, Joe Ruby <joeat303 at yahoo.com>
wrote:
> Hey Ezra,
>
> What other gains do you get with virtualizing?
>
> Also, would Xen/virtualizing make these easier?
> - restoring root compromised servers
> - transferring everything from an old to new server --
> I do this periodically and cringe at having to
> reinstall everything
>
My guess is like Xen, like parallels or VMWave offer a "rollback" or
readonly functionality. for the virtual machine hard disk.

I have setup many times with parallels our OS disk in read only (or
discard changes mode) and our data with rollback functionality.

But guess xen like vmware have some snapshots capabilities which
simplifies everything (like revert to yesterday snapshot).

But Ezra could get offer better info about Xen.

> Thanks,
> Joe
>
>
> >       Yeah now that I have been way into Xen, I wouldn''t
> setup a new
> > server without it. Even if you only have one server
> I would put Xen
> > on it, even if its just one huge VM that fills the
> entire server. The
> > performance overhead of Xen is about 5% compared to
> raw hardware
> > which is really nothing compared to  the gains you
> get by virtualizing.
> >
> >       It makes it so much easier to scale later. Move
> your VM''s from
> > machine to machine. And separate concerns into their
> own VM''s.
> > Seriously I can''t recommend it enough. We have built
> our entire new
> > business on top of Xen. http://engineyard.com . Its
> only for ruby/
> > rails hosting and everything runs in Xen. Xen is
> very cool stuff.
> >
> > -Ezra
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

This sounds really sweet, I''m definitely going to have to look into
this more. It would simply pretty much everything from deployment to
backups.

-carl

On 10/3/06, Luis Lavena <luislavena at gmail.com>
wrote:
> On 10/3/06, Scott Laird <laird at google.com> wrote:
> > On 10/2/06, Luis Lavena <luislavena at gmail.com> wrote:
> >
> > > Ezra, one thing that still bug me is IP handling. If i have a
live
> > > server and decide to do a hot migration to a newer box, how it
> > > manages, and how much downtime will get to the IP switches ges
> > > notified of the migration? (in ms resolution)
> > >
> > > Or i''m forced to have a load balancer that handle the
problem for me?
> >
> > If you''re doing it right, the switches should notice within a
few
> > milliseconds.  Any reasonable virtualization setup will have to assign
> > a virtual MAC address to each virtual ethernet interface, and the MAC
> > address should transfer between machines along with the rest of the
> > VM.  So, assuming that both machines are on the same L2 network,
it''s
> > just a matter of the new machine sending an Ethernet frame or two out
> > its interface using the virtual MAC address as the source address.  To
> > the switch, it''ll look like the virtual server was unplugged
from one
> > port and plugged into the new one, which isn''t a big deal. 
I''d be
> > amazed if VMWare didn''t have code in it to explicitly trip
this, but
> > even without it it''ll happen almost instantly.
> >
>
> Thank you for your answer Scott, I''ll investigate Xen further on
my
> spare time the following months.
>
> Regards,
>
> >
> > Scott
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>
> --
> Luis Lavena
> Multimedia systems
> -
> Leaders are made, they are not born. They are made by hard effort,
> which is the price which all of us must pay to achieve any goal that
> is worthwhile.
> Vince Lombardi
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

Thanks everyone for the valuable input!

cheers

Thibaut
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061004/08f7a851/attachment-0001.html

On 10/3/06, Ezra Zygmuntowicz <ez at brainspl.at>
wrote:
> Hey Joe-
>
> On Oct 3, 2006, at 6:29 PM, Joe Ruby wrote:
>
> > Hey Ezra,
> >
> > What other gains do you get with virtualizing?
> >
> > Also, would Xen/virtualizing make these easier?
> > - restoring root compromised servers
>
> Yup, you can take snapshots of a VM and restore back to that point in
> time without a fuss. Of course you have to set this up and its up to
> you how often you take snapshots is up to you.
>
> > - transferring everything from an old to new server --
>
> Yup its pretty much just a few command lines to move a VM from one
> physical box to another physical box *while its running*
OK, I''ll take this a bit further off topic.  Ezra, do you plan to
cover Xen in Rails Deployment?

Also, the blurb on the Pragmatic site indicates that the book is going
to focus on lighttpd.  Is that still the case, or is the description
out-of-date?

-- James

On Oct 4, 2006, at 8:14 AM, JDL wrote:
> On 10/3/06, Ezra Zygmuntowicz <ez at brainspl.at> wrote:
>> Hey Joe-
>>
>> On Oct 3, 2006, at 6:29 PM, Joe Ruby wrote:
>>
>>> Hey Ezra,
>>>
>>> What other gains do you get with virtualizing?
>>>
>>> Also, would Xen/virtualizing make these easier?
>>> - restoring root compromised servers
>>
>> Yup, you can take snapshots of a VM and restore back to that point in
>> time without a fuss. Of course you have to set this up and its up to
>> you how often you take snapshots is up to you.
>>
>>> - transferring everything from an old to new server --
>>
>> Yup its pretty much just a few command lines to move a VM from one
>> physical box to another physical box *while its running*
>
> OK, I''ll take this a bit further off topic.  Ezra, do you plan to
> cover Xen in Rails Deployment?
>
> Also, the blurb on the Pragmatic site indicates that the book is going
> to focus on lighttpd.  Is that still the case, or is the description
> out-of-date?
>
> -- James

The description is out of date. I will of course cover lighty but I  
will also focus on ngxin and apache2.2 as fornt ends and mongrels as  
back ends. Although I will cover fcgi as well just to cover the  
bases. I do cover working on your own VPS but I will not be going  
into details about configuring Xen itself as it is outside the scope  
of the book. But who knows maybe in the future or 2.0 or the book I  
will cover setting up your own Xen environment. I will also post some  
Xen tutorials on my blog once I get some free time(which could be any  
year now ;-).

Cheers-
-Ezra

> The description is out of date. I will of course
cover lighty but I  
> will also focus on ngxin and apache2.2 as fornt ends
and mongrels as  
> back ends. Although I will cover fcgi as well just
to cover the  
> bases. I do cover working on your own VPS but I will
not be going  
> into details about configuring Xen itself as it is
outside the scope  
> of the book. But who knows maybe in the future or
2.0 or the book I  
> will cover setting up your own Xen environment. I
will also post some  
> Xen tutorials on my blog once I get some free
time(which could be any  
> year now ;-).
>
> Cheers-
> -Ezra
I look forward to those tutorials! :)

I have some more n00b questions, if you don''t mind. :)

- Is it easy to get Xen working on a server that''s
already in production (currently functioning as a web,
app, and db server)?

- Can Xen be installed/setup/used remotely (I have
servers at LayeredTech and EV1)? Xen''s screenshots
show Linux desktops.

- Know of any quickstart or "Xen for Dummies"
docs/tutorials? I''m currently working my way through
Xen''s official docs.

Thanks,
Joe


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com