Xen users - Nov 2007 - Advice on HighAvailable/Failover Xen machines

Hi!

I want to accomplish the following setup:
 Two (maybe leter more) Xen systems running on two machines with HBAs
 inside connected to a SAN.

 I would like to place the DomU block-devices on logical volumes on
 top of a LUN of the SAN which is available at both (all) Dom0s.

I read about the possibiliy to use EVMS as one solutions which could
fullfil my needs.

Is this a good idea?
How could I do some kind of failover in case one Dom0 has gone?
Can Xen detect this and move the affected DomUs automagically?

Any hints how to set something like this up would be highly
appreaciated.

Thanks
Ciao
Max
-- 
	Follow the white penguin.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Nov 28, 2007 at 02:33:58AM +0100, Maximilian Wilhelm
wrote:
> 
> I want to accomplish the following setup:
>  Two (maybe leter more) Xen systems running on two machines with HBAs
>  inside connected to a SAN.
> 
>  I would like to place the DomU block-devices on logical volumes on
>  top of a LUN of the SAN which is available at both (all) Dom0s.
> 
> I read about the possibiliy to use EVMS as one solutions which could
> fullfil my needs.
> 
> Is this a good idea?
EVMS plus heartbead can provide this, alternatively you could use
- clvm here
- or use a cluster-fs like ocfs2 or gfs and place files on that.
> How could I do some kind of failover in case one Dom0 has gone?
> Can Xen detect this and move the affected DomUs automagically?
Thats entirely up to you to
a) detect this
b) start up the domU you need alive again
c) make sure the domU is started only once on your xen-site.

Heartbeat or other cluster-software can help for a and b,
maybe clvm can do the locking of volumes for point c, or you implement
the locking yourself.


Christian

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

2007/11/28, Maximilian Wilhelm <max@rfc2324.org>:
> Hi!
>
> I want to accomplish the following setup:
>  Two (maybe leter more) Xen systems running on two machines with HBAs
>  inside connected to a SAN.
>
>  I would like to place the DomU block-devices on logical volumes on
>  top of a LUN of the SAN which is available at both (all) Dom0s.
>
> I read about the possibiliy to use EVMS as one solutions which could
> fullfil my needs.
>
> Is this a good idea?
> How could I do some kind of failover in case one Dom0 has gone?
> Can Xen detect this and move the affected DomUs automagically?
>
> Any hints how to set something like this up would be highly
> appreaciated.
I''ve played around a lot on this path.
With EMVS you can either use the cluster segment manager or the OCFS2 Plugin
with SAN Storage, either has it''s advantages, both will nicely
integrate with Linux-HA / heartbeat.

EVMS also is greatly helpful for fully automated/scripted storage
setup etc which comes to mind for HA setups.

But, now for the downside:
EVMS is unfortunately quite dead, the OCFS2 plugin never made it into
mainline, the heartbeat integration needs patches, the BBR patches
need dm patches for the dom0 kernel, there is almost no irc support
left, the devs are.. well, i don''t know where AND as it was designed
by IBM gurus it is really well thought through and perfectly
structured, but INCREDIBLY annoying to use!
Once you go into larger Xen setups you might have to handle 100s of
volumes and on occassion it might be manually - personally i think the
EVMS management is unbearable.

I still run my system on EVMS and i don''t really see a good
alternative for a logical-volume based attempt (yes clvm exists but,
as always, can''t mirror LVs in a feasible way)

for a new HA system i''d try to tie together opensharedroot(.org),
ocfs2 and the san, booting from ocfs2 root, with another volume(s) for
the domU images.
But that implies you don''t use a JBOD on the san but higher end
(active-active) storage.

if your site has SDRF-ed EMC^2 Symmetrix that''s no issue, otherwise it
might well turn into one.

Regards,
Florian


-- 
''Sie brauchen sich um Ihre Zukunft keine Gedanken zu machen''

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

Based on my own experience with EVMS and Heartbeat together, I share
Florian''s assessment: EVMS can be a pain in the butt. But once you
learn its quirks it does the job. I found multipathing one of the
near-showstoppers with EVMS. It took a while to figure out the solution for
different seemingly random problems: just let EVMS ignore all /dev/sd* devices
and use only multipathed devs in /dev/mapper/* (same goes btw for LVM2 as well)

Depending on what you want to achieve, having a cluster aware volume manager is
not a requisite for running highly available Xen domUs. As Florian pointed out,
you can just put disk images on OCFS2 and run the domUs from there. This has
also worked for me quite well.

Even the setup what you''re trying, where each VM has its own logical
volume, does not necessarily require cluster aware volume manager. That is, if
your LV layout is fairly static, even with LVM2 your domUs will run just fine.
Sure, if you add or change a volume, the other node won''t see it
immediately. With careful planning it can be done. I tried this and every time I
let LVM2 rescan the volumes, it found out about the new layout. This might,
however, not be the safest way to do it. On the other hand EVMS destroyed the
layout on more than one occasion.

The option I finally decided on to be perhaps the most flexible, is where each
domU also has its own dedicated LUN. On these LUNs I used old-school PC
partition table. No LVs, so no need for cluster aware volume manager.
It''s more simple, more compartmentized and the trade-offs are something
I could live with. One example for this is that you''ll need more LUNs.
But since it''s a SAN, who cares if it''s 3 LUNs or 30, right?
(unless of course your specific storage system has limitations on the nr. of
LUNs) Resizing filesystems is also more involved this way than with LVs. Btw: if
you take my tip: don''t put anything but OS related stuff on the
domU''s disk. Especially databases should always have their dedicated
disks.

The Heartbeat resource agent for Xen does provide you with the automatic restart
or fail-over of a domU. Another nice thing Heartbeat provides is that
it''s supports up to 16 nodes. (I''ve stuck with 4 btw.)
You don''t mention the distro that you''re using, but if
it''s SLES10, Novell will provide you with enterprise grade support for
any of the above configurations.

Hope this helps.

Regards,
Gábor Nyers

-----Original Message-----

Date: Wed, 28 Nov 2007 12:24:19 +0100
From: "Florian Heigl" <florian.heigl@gmail.com>
Subject: Re: [Xen-users] Advice on HighAvailable/Failover Xen machines
To: xen-users@lists.xensource.com
Message-ID:
	<77abe410711280324i14c7e35et8045605bdc346159@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi,

2007/11/28, Maximilian Wilhelm <max@rfc2324.org>:
> Hi!
>
> I want to accomplish the following setup:
>  Two (maybe leter more) Xen systems running on two machines with HBAs
>  inside connected to a SAN.
>
>  I would like to place the DomU block-devices on logical volumes on
>  top of a LUN of the SAN which is available at both (all) Dom0s.
>
> I read about the possibiliy to use EVMS as one solutions which could
> fullfil my needs.
>
> Is this a good idea?
> How could I do some kind of failover in case one Dom0 has gone?
> Can Xen detect this and move the affected DomUs automagically?
>
> Any hints how to set something like this up would be highly
> appreaciated.
I''ve played around a lot on this path.
With EMVS you can either use the cluster segment manager or the OCFS2 Plugin
with SAN Storage, either has it''s advantages, both will nicely
integrate with Linux-HA / heartbeat.

EVMS also is greatly helpful for fully automated/scripted storage
setup etc which comes to mind for HA setups.

But, now for the downside:
EVMS is unfortunately quite dead, the OCFS2 plugin never made it into
mainline, the heartbeat integration needs patches, the BBR patches
need dm patches for the dom0 kernel, there is almost no irc support
left, the devs are.. well, i don''t know where AND as it was designed
by IBM gurus it is really well thought through and perfectly
structured, but INCREDIBLY annoying to use!
Once you go into larger Xen setups you might have to handle 100s of
volumes and on occassion it might be manually - personally i think the
EVMS management is unbearable.

I still run my system on EVMS and i don''t really see a good
alternative for a logical-volume based attempt (yes clvm exists but,
as always, can''t mirror LVs in a feasible way)

for a new HA system i''d try to tie together opensharedroot(.org),
ocfs2 and the san, booting from ocfs2 root, with another volume(s) for
the domU images.
But that implies you don''t use a JBOD on the san but higher end
(active-active) storage.

if your site has SDRF-ed EMC^2 Symmetrix that''s no issue, otherwise it
might well turn into one.

Regards,
Florian


-- 
''Sie brauchen sich um Ihre Zukunft keine Gedanken zu machen

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Nov 28, 2007 at 12:24:19PM +0100, Florian Heigl
wrote:
> 
> But, now for the downside:
> EVMS is unfortunately quite dead, the OCFS2 plugin never made it into
> mainline, the heartbeat integration needs patches, the BBR patches
> need dm patches for the dom0 kernel, there is almost no irc support
> left, the devs are.. well, i don''t know where AND as it was
designed
> by IBM gurus it is really well thought through and perfectly
> structured, but INCREDIBLY annoying to use!
> Once you go into larger Xen setups you might have to handle 100s of
> volumes and on occassion it might be manually - personally i think the
> EVMS management is unbearable.
Agreed, its really heavy stuff. But at least it has names for each and
everything i was confronted with regarding storages yet  :)
> I still run my system on EVMS and i don''t really see a good
> alternative for a logical-volume based attempt (yes clvm exists but,
> as always, can''t mirror LVs in a feasible way)
The mirroring exists since 4.5 :
http://www.redhat.com/docs/manuals/csgfs/browse/4.5/SAC_Cluster_Logical_Volume_Manager/mirrored_volumes.html
but it has problems and is far from beeing as nice as md-mirroring.

Christian

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users