Xen users - Sep 2007 - Debian Etch Xen Cluster (DRBD, GNBD, OCFS2, iSCSI, Heartbeat?)

Hi,
I am planning to run 4 virtual guests on 2 Dell 2900 Servers. Both Dell 
2900 are configured with:

2x Quad Core Xeon
8 GB RAM
3x 500 GB SATA Raid 5
2x Broadcom Gigabit
2x Intel Pro 1000

I want to build an active/active Cluster running these 4 guests:

A: Windows 2003 Server (Active Directory, MSSQL)
B: Debian Etch Mailserver (Postfix...)
C: Endian Firewall
D: Debian Etch Webserver (Lamp)

Node1 should run A and D while node2 should run B and C. If Node1 fails 
A and D should switch to node2 and switch back after node1 is back 
again. I red a lot about very complex setups with OCFS2, iSCSI or GNBD 
but i think all i need ist DRBD and Heartbeat, isn''t it?

Is it possible to tell Xen 3.1 to use a DRBD device to store the guests 
filesystem on it?
Will it be possible to switch the windows server?
Will the data stay consistent in my database?
Would it be better to use a 3rd node running iSCSI as shared storage?
Any suggestions?

Many thanks for any tips and ideas.

greetings Flo

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Node1 should run A and D while node2 should run B and C. If Node1 fails 
> A and D should switch to node2 and switch back after node1 is back 
> again. I red a lot about very complex setups with OCFS2, iSCSI or GNBD 
> but i think all i need ist DRBD and Heartbeat, isn''t it?
Correct.
> Is it possible to tell Xen 3.1 to use a DRBD device to store the guests 
> filesystem on it?
See http://lists.xensource.com/archives/html/xen-users/2007-09/msg00059.html
> Will it be possible to switch the windows server?
Shutdown on A, start on B should be possible. But beware I never dealt 
with HVM guests. It should be possible though.

PVM can use live-migration if you use drbd two-primary mode.
> Will the data stay consistent in my database?
As long as there is no drbd error: yes.
> Would it be better to use a 3rd node running iSCSI as shared storage?
> Any suggestions?
You would at least have to have 2 more nodes to run iscsi to make 
storage high available, too. HA domU with SPOF storage is not the best 
idea imho.

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > Node1 should run A and D while node2 should run B and C. If Node1
fails
> > A and D should switch to node2 and switch back after node1 is back
> > again. I red a lot about very complex setups with OCFS2, iSCSI or
GNBD
> > but i think all i need ist DRBD and Heartbeat, isn''t it?
> 
> Correct.
> 
Just remember, if something goes wrong in such a way that the domain is
active on both nodes at the same time with read/write access to the
filesystem, you *will* *destroy* the filesystem and will need to restore
from backup. No amount of fscking will help you.

(I speak from bitter experience :)

James

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

James Harper schrieb:
>>> Node1 should run A and D while node2 should run B and C. If Node1
>>>       
> fails
>   
>>> A and D should switch to node2 and switch back after node1 is back
>>> again. I red a lot about very complex setups with OCFS2, iSCSI or
>>>       
> GNBD
>   
>>> but i think all i need ist DRBD and Heartbeat, isn''t it?
>>>       
>> Correct.
>>
>>     
>
> Just remember, if something goes wrong in such a way that the domain is
> active on both nodes at the same time with read/write access to the
> filesystem, you *will* *destroy* the filesystem and will need to restore
> from backup. No amount of fscking will help you.
>
> (I speak from bitter experience :)
>
> James
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>   
Sounds like you got some experience with such a setup. Do you think a 
cluster like this is ready for the productive use? Is Xen HVM fast 
enough to host a MSSQL Server (20 Users)?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Just remember, if something goes wrong in such a way that the domain is
> active on both nodes at the same time with read/write access to the
> filesystem, you *will* *destroy* the filesystem and will need to restore
> from backup. No amount of fscking will help you.
> 
> (I speak from bitter experience :)
Full Ack (without the bitter experience yet though).

That''s why you want to have a functioning STONITH Device and redundant 
cluster communication ways.

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Sep 13, 2007 at 09:05:59PM +0200, Mehdi AMINI
wrote:
> >
> > Just remember, if something goes wrong in such a way that the domain
is
> > active on both nodes at the same time with read/write access to the
> > filesystem, you *will* *destroy* the filesystem and will need to
restore
> > from backup. No amount of fscking will help you.
> 
> 
> This is precisely the goal of OCFS, each node can mount a block device
> read/write at the same time :)
..but its also an additional layer when comparing with just having evms
or clvm on the shared disks. Would be great if those two could also provide
snapshots on shared volumes, but i dont think they do at the moment.

Christian 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Sep 13, 2007 at 03:43:24PM +0200, Dominik Klein
wrote:
> >Just remember, if something goes wrong in such a way that the domain is
> >active on both nodes at the same time with read/write access to the
> >filesystem, you *will* *destroy* the filesystem and will need to
restore
> >from backup. No amount of fscking will help you.
> >
> >(I speak from bitter experience :)
> 
> Full Ack (without the bitter experience yet though).
> 
> That''s why you want to have a functioning STONITH Device and
redundant
> cluster communication ways.
That will only help on clusterside.
Has anyone good ideas on how to prevent domUs that are stored on evms or
clvm from beeing started on multiple dom0s?
Setting some lock on an ldap-server, in files on a ocfs2-fs or running
through the dom0s via ssh and asking if the domU is running would be
my ideas.. is there something less errorprone?


Christian

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>>
>
> Just remember, if something goes wrong in such a way that the domain is
> active on both nodes at the same time with read/write access to the
> filesystem, you *will* *destroy* the filesystem and will need to restore
> from backup. No amount of fscking will help you.

This is precisely the goal of OCFS, each node can mount a block device
read/write at the same time :)

Mehdi



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> 
> Sounds like you got some experience with such a setup. Do you think a
> cluster like this is ready for the productive use? Is Xen HVM fast
> enough to host a MSSQL Server (20 Users)?
I was running a pair of xen servers over AoE, and it worked pretty well,
but they were only PV, not HVM. I didn''t have any automated cluster
management though, I just migrated the DomU''s manually as required.
This
of course meant that one day I wasn''t paying attention and started up
the same domain on both separate machines simultaneously - that''s the
bitter experience I was referring to :)

Without the PV disk & network drivers, I think you''ll have a hard
time
getting exceptional performance out of Windows HVM. Unfortunately the
Windows PV drivers are only available with commercial versions of Xen,
which doesn''t suit me (I''m more than happy to pay for the
drivers - up
to a few hundred $$$/vm would be more than acceptable - I just need to
be able to have more flexibility wrt Dom0).

James

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Mehdi AMINI schrieb:
>> Just remember, if something goes wrong in such a way that the domain is
>> active on both nodes at the same time with read/write access to the
>> filesystem, you *will* *destroy* the filesystem and will need to
restore
>> from backup. No amount of fscking will help you.
> 
> 
> This is precisely the goal of OCFS, each node can mount a block device
> read/write at the same time :)
But still, you don''t want to have two servers write to the root 
filesystem simultaneusly, do you?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Christian Horn schrieb:
> On Thu, Sep 13, 2007 at 03:43:24PM +0200, Dominik Klein wrote:
>>> Just remember, if something goes wrong in such a way that the
domain is
>>> active on both nodes at the same time with read/write access to the
>>> filesystem, you *will* *destroy* the filesystem and will need to
restore
>> >from backup. No amount of fscking will help you.
>>> (I speak from bitter experience :)
>> Full Ack (without the bitter experience yet though).
>>
>> That''s why you want to have a functioning STONITH Device and
redundant
>> cluster communication ways.
> 
> That will only help on clusterside.
Correct.
Keep your hands off Xen if you have a cluster software manage it. Use 
cluster commands if you need to do anything manually.
> Has anyone good ideas on how to prevent domUs that are stored on evms or
> clvm from beeing started on multiple dom0s?
You could use a wrapper script for xm.

I don''t use anything like that, but this should give you an idea of
what
I am thinking of.

.bashrc:
alias xm=myxm.sh

myxm.sh
#!/bin/bash

if echo $* | grep -q -w create
then
# for this, you need passwordless ssh access to $othermachine
if ssh $othermachine "xm list $machine"
then
	echo "domU $machine already running on $othermachine. Exit"
	exit 5
else
	/usr/sbin/xm $*
fi
else
/usr/sbin/xm $*
fi

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Sep 14, 2007 at 08:37:17AM +0200, Dominik Klein
wrote:
> Christian Horn schrieb:
> 
> >Has anyone good ideas on how to prevent domUs that are stored on evms
or
> >clvm from beeing started on multiple dom0s?
> 
> You could use a wrapper script for xm.
> 
> [...]
> if ssh $othermachine "xm list $machine"
> then
> 	echo "domU $machine already running on $othermachine. Exit"
> 	exit 5
> else
> 	/usr/sbin/xm $*
> fi
Jup, that was one of my thoughts, now that i think more about it it doesnt
look too fault-proof, especially when more than 2 dom0s are involved in 
a ''site'' of dom0s.
Maybe some ldap where dom0s lock the domUs in could be nice.

Btw, from the docs now i have seen that clvm apparently does the locking,
evms isnt doing this currently (and is also heavier, but the preferred
solution i.e. when using SLES).


Christian

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, 2007-09-14 at 11:06 +0200, Christian Horn wrote:
> On Fri, Sep 14, 2007 at 08:37:17AM +0200, Dominik Klein wrote:
> > Christian Horn schrieb:
> > 
> > >Has anyone good ideas on how to prevent domUs that are stored on
evms or
> > >clvm from beeing started on multiple dom0s?
> > 
> > You could use a wrapper script for xm.
> > 
> > [...]
> > if ssh $othermachine "xm list $machine"
> > then
> > 	echo "domU $machine already running on $othermachine. Exit"
> > 	exit 5
> > else
> > 	/usr/sbin/xm $*
> > fi
> 
Try SRCE, http://echoreply.us/hg/srce.hg , its a 100% non blocking
cluster interconnect that was designed to do what your doing.

Once installed (make make install make test) see /etc/srced/classes
and /etc/srced/nodes for the simplified cluster config. 

Make sure a link to ''xm'' is in /var/srced/exec , then do
anything you
want from wherever you want, i.e. 

on node3 xm reboot guest33
on node2 xm migrate --live guest23 someothernode
... or

on all xennodes xm list

''on'' is just a wrapper script to control the client, otherwise
its

srce -k keyfile.txt -s 1.2.3.4 EXEC "command to run remotely"

There''s some documentation to get you going, its very simple. SRCE is
blowfish encrypted and uses characteristics (random) of plain text keys
to authenticate. Its quite safe, tried, tested and tiny, perfect for
dom-0. SSH is such a waste on such things. Under 100k statically linked
with its own encryption and non blocking socket libs. It also has its
own deny/allow tcp wrappers that do not rely on iptables. I''ve got it
in
use on a few hundred machines with no issues.
> Jup, that was one of my thoughts, now that i think more about it it doesnt
> look too fault-proof, especially when more than 2 dom0s are involved in 
> a ''site'' of dom0s.
> Maybe some ldap where dom0s lock the domUs in could be nice.
> 
> Btw, from the docs now i have seen that clvm apparently does the locking,
> evms isnt doing this currently (and is also heavier, but the preferred
> solution i.e. when using SLES).
I have :

http://echoreply.us/hg/xmlpulse.hg which gives you a single iteration of
xm-top in XML format which some central place can gather to know what is
where and what its doing. I used libxenstat because it ships with Xen to
keep things simple, you could easily change it to use libvirt or
something else. 

I use SRCE to control stuff like ocfs2 / drbd / aoe and other goodies on
large farms and similar clusters, its really a nice tool. Hopefully you
find it useful and enjoy it :)

You could, also, just use the Xen API .. I made this stuff when such a
thing did not exist. Since we use several virtualization technologies, I
continue to use SRCE to manage them all from one place/method.

Kindly,
--Tim



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Dominik Klein <dk@in-telegence.net> writes:
> Mehdi AMINI schrieb:
>>> Just remember, if something goes wrong in such a way that the
domain is
>>> active on both nodes at the same time with read/write access to the
>>> filesystem, you *will* *destroy* the filesystem and will need to
restore
>>> from backup. No amount of fscking will help you.
>>
>>
>> This is precisely the goal of OCFS, each node can mount a block device
>> read/write at the same time :)
>
> But still, you don''t want to have two servers write to the root
> filesystem simultaneusly, do you?
2 Things to think about:

1) Do what you would do without xen. Throw the power switch.

That means you have to teach stonit or heartbeat about xen and the
power switch becomes "xm destroy" on the other node instead of a real
power switch.

2) Would it make sense to have a bit in the lvm headers to show that a
volume is active and have cluster lvm respect that bit and prevent a
second activation if run wihtout force?

Maybe instead of a bit an UUID of the activator would be best. If you
can get a UUID that differs between the physical machines running the
domUs.

MfG
        Goswin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Goswin von Brederlow wrote:
> Dominik Klein <dk@in-telegence.net> writes:
>
>   
>> Mehdi AMINI schrieb:
>>     
>>>> Just remember, if something goes wrong in such a way that the
domain is
>>>> active on both nodes at the same time with read/write access to
the
>>>> filesystem, you *will* *destroy* the filesystem and will need
to restore
>>>> from backup. No amount of fscking will help you.
>>>>         
>>> This is precisely the goal of OCFS, each node can mount a block
device
>>> read/write at the same time :)
>>>       
>> But still, you don''t want to have two servers write to the
root
>> filesystem simultaneusly, do you?
>>     
>
> 2 Things to think about:
>
> 1) Do what you would do without xen. Throw the power switch.
>
> That means you have to teach stonit or heartbeat about xen and the
> power switch becomes "xm destroy" on the other node instead of a
real
> power switch.
>
> 2) Would it make sense to have a bit in the lvm headers to show that a
> volume is active and have cluster lvm respect that bit and prevent a
> second activation if run wihtout force?
>
> Maybe instead of a bit an UUID of the activator would be best. If you
> can get a UUID that differs between the physical machines running the
> domUs.
>
> MfG
>         Goswin
>   
Hmm. You know, you can rename an LVM partition while Xen has it mounted. 
This might be a useful trick to prevent another Xen guest from mounting it.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,
> 1) Do what you would do without xen. Throw the power switch.
>
> That means you have to teach stonit or heartbeat about xen and the
> power switch becomes "xm destroy" on the other node instead of a
real
> power switch.
If the network is down, you won''t be able to "xm destroy" the
other node.

> 2) Would it make sense to have a bit in the lvm headers to show that a
> volume is active and have cluster lvm respect that bit and prevent a
> second activation if run wihtout force?
I''m not sure I understand what you mean, but your bit mean only that a
volume is mounted or unmounted (cleanly), it won''t be able to detect
that
a volume has been uncleany umounted.


Mehdi



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

"Mehdi AMINI" <armeni67@hotmail.com> writes:
> Hello,
>
>> 1) Do what you would do without xen. Throw the power switch.
>>
>> That means you have to teach stonit or heartbeat about xen and the
>> power switch becomes "xm destroy" on the other node instead
of a real
>> power switch.
>
> If the network is down, you won''t be able to "xm
destroy" the other node.
Which would mean one of the two network cards or the cable is broken
or the node is crashed. You do use a cross over cable for the
sonit(heartbeat link, right?

In that case you have to kill the other node completly with all its
domains. You throw the power switch. With xen it is just nested. You
have a small (xm destroy) and big (real) power switch.
>> 2) Would it make sense to have a bit in the lvm headers to show that a
>> volume is active and have cluster lvm respect that bit and prevent a
>> second activation if run wihtout force?
>
> I''m not sure I understand what you mean, but your bit mean only
that a
> volume is mounted or unmounted (cleanly), it won''t be able to
detect that
> a volume has been uncleany umounted.
You can alway mount a clean volume. Obviously.

You can also mount an unclean volume if you know the other node is
down (xm destroy) or node power turned off. Mount it with force.

Otherwise you are in a critical failure situation and don''t know what
to do so you scream for an admin.

MfG
        Goswin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nico Kadel-Garcia wrote:
> Goswin von Brederlow wrote:
>> Dominik Klein <dk@in-telegence.net> writes:
>>
>>  
>>> Mehdi AMINI schrieb:
>>>    
>>>>> Just remember, if something goes wrong in such a way that
the
>>>>> domain is
>>>>> active on both nodes at the same time with read/write
access to the
>>>>> filesystem, you *will* *destroy* the filesystem and will
need to
>>>>> restore
>>>>> from backup. No amount of fscking will help you.
>>>>>         
>>>> This is precisely the goal of OCFS, each node can mount a block
device
>>>> read/write at the same time :)
>>>>       
>>> But still, you don''t want to have two servers write to the
root
>>> filesystem simultaneusly, do you?
>>>     
>>
>> 2 Things to think about:
>>
>> 1) Do what you would do without xen. Throw the power switch.
>>
>> That means you have to teach stonit or heartbeat about xen and the
>> power switch becomes "xm destroy" on the other node instead
of a real
>> power switch.
>>
>> 2) Would it make sense to have a bit in the lvm headers to show that a
>> volume is active and have cluster lvm respect that bit and prevent a
>> second activation if run wihtout force?
>>
>> Maybe instead of a bit an UUID of the activator would be best. If you
>> can get a UUID that differs between the physical machines running the
>> domUs.
>>
>> MfG
>>         Goswin
>>   
> Hmm. You know, you can rename an LVM partition while Xen has it mounted. 
> This might be a useful trick to prevent another Xen guest from mounting it.
> 
I assume you would need CLVM on the DomUs to do this, otherwise the lv 
rename would only be local to that particular Dom0.  Does someone that 
uses CLVM know whether you can do this?

Simon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users