CentOS - Jun 2010 - Virtualization as cheap redundancy option?

I'm wondering if virtualization could be used as a cheap redundancy
solution for situations which can tolerate a certain amount of
downtime.

Current recommendations is to run some kind of replication server such
as DRBD. The problem here is cost if there are more than one server
(or servers running on different OS) to be backed up. I'd basically
need to tell my client they need to buy say 2X machines instead of
just X. Not really attractive :D

So I'm wondering if it would be a good, or likely stupid idea, to run
X CentOS machines with VMware. Each running a single instance of
CentOS and in at least one case of Windows for MSSQL.

So if any of the machines physically fails for whatever reasons not
related to disk. I'll just transfer the disk to one of the surviving
server or a cold standby and have things running again within say
30~60 minutes needed to check the filesystem, then mount and copy the
image.

I thought I could also rsync the images so that Server 1 backs up
Server 2 image file and Server 2 backs up Server 3 etc in a round
robin fashion to make this even faster. But reading up indicates that
rsync would attempt to mirror the whole 60gb or 80gb image on any
change. Bad idea.

So while this is not real time HA but in most situations, they can
tolerate an hour's downtime. The cost of the "redundancy" also
stays
constant  no matter how many servers are added to the operation.

Any comments on this or is it like just plain stupid because there are
better options that are equally cost effective?

On Fri, Jun 25, 2010 at 9:04 AM, Emmanuel Noobadmin
<centos.admin at gmail.com> wrote:
> I'm wondering if virtualization could be used as a cheap redundancy
> solution for situations which can tolerate a certain amount of
> downtime.
>
> Current recommendations is to run some kind of replication server such
> as DRBD. The problem here is cost if there are more than one server
> (or servers running on different OS) to be backed up. I'd basically
> need to tell my client they need to buy say 2X machines instead of
> just X. Not really attractive :D
>
> So I'm wondering if it would be a good, or likely stupid idea, to run
> X CentOS machines with VMware. Each running a single instance of
> CentOS and in at least one case of Windows for MSSQL.
>
> So if any of the machines physically fails for whatever reasons not
> related to disk. I'll just transfer the disk to one of the surviving
> server or a cold standby and have things running again within say
> 30~60 minutes needed to check the filesystem, then mount and copy the
> image.
>
> I thought I could also rsync the images so that Server 1 backs up
> Server 2 image file and Server 2 backs up Server 3 etc in a round
> robin fashion to make this even faster. But reading up indicates that
> rsync would attempt to mirror the whole 60gb or 80gb image on any
> change. Bad idea.
>
> So while this is not real time HA but in most situations, they can
> tolerate an hour's downtime. The cost of the "redundancy"
also stays
> constant ?no matter how many servers are added to the operation.
>
> Any comments on this or is it like just plain stupid because there are
> better options that are equally cost effective?
>
This is one of the advantages of using VMs, and I'm sure most people
are using it for this reason in one way or another.  However, there
are a few things you need to worry about:

- When the host crashes, the guests will also, so you'll be in a
recovery situation just like for a physical crash.  This is manageable
and something you'd have to deal with either way.

- Rsyncing the VMs while they are running leaves them in an
inconsistent state.  This state may or may not be worse than a simple
crash situation.  One way I have been getting around this is by
creating a snapshot of the VM before performing the rsync, and when
bringing up the copy after a crash, revert to the snapshot.  That will
at least give you consistent filesystem and memory state, but could
cause issues with network connections.  I usually reboot the VM
cleanly after reverting to the snapshot.

Rsync will not transfer the entire file when transferring over the
network.  It scans the whole thing and only sends changes.  If you
have --progress enabled it will appear to go through the whole file,
but you will see the "speedup" go much higher than a regular transfer.
 However, sometimes this process can take more time than doing a full
copy on a local network.  Rsync is meant to conserve bandwidth, not
necessarily time.  Also, I suggest the you use a GB network if you
have the option.  If not you could directly link the network ports on
2 servers and copy straight from 1 to the other.


If you are looking at VMware Server for this, here are some tips:
- For best performance, search around for "vmware tmpfs".  It will
dramatically increase the performance of the VMs at the expense of
some memory.
- VMware Server seems like it's EOL, even though vmware hasn't
specifically said so yet
- There is a bug in VMware with CentOS that causes guests to slowly
use more CPU until the whole machine is bogged down.  This can be
fixed by restarting or suspend/resume each VM
- At this point I'd look at ESXi for the free VMware option.

On Fri, Jun 25, 2010 at 6:34 PM, Emmanuel Noobadmin
<centos.admin at gmail.com> wrote:
> I'm wondering if virtualization could be used as a cheap redundancy
> solution for situations which can tolerate a certain amount of
> downtime.
Absolutely.  I use the Linux KVM as my virtualization platform.   It
is just not redundancy but also better utilization of your hardware
and saving installation/configuration time by creating templates.

For example, at a couple of client locations, I  have 4 DNS/openLDAP
servers in production  (2 VMs each on 2 different hosts) per location.
  I did the initial install for one VM,  the configured DNS + LDAP
services, tested them and put them in production.  To replicate,  I
simply copy the VM image on to the other host,  reconfigure the
services in the guest VMs to be secondary servers.  The secondary DNS
and  LDAP servers are synced with the primary using their own internal
mechanism for replication.

Now I can use the same image file as a template for any other client
that wants these services in their IT infra.

Do a minimal install of  the host OS (takes less time), make sure it's
clock is in sync with a reliable NTP server.  The keyword is "minimal"
- only those packages that are needed to support virtualization.
Copy the necessary VM files; change the client specific info like
domain names, user entries and the "DNS/Directory server" is
operational within an hour.

-- Arun Khan