Xen users - Jul 2010 - as promised description of my XEN HA setup

Hi all,

In threads posted by I believe Jonathan Tripley I promised to post my new XEN 
HA setup. Hope it can be of some use to some people. 

In this particular case I''m forced to use SLES 10SP3 with XEN 3.2,
which
excludes the possibility of using things like cLVM (which I don''t think
I need
anyway).

So:

Storage:
I use two HP ML370 G5 machines with DRBD and heartbeat on them. They are 
linked by two Gigabit bonded NICs for syncing. They offer IET across two other 
NICs with IPs in different segments. DRBD is on top of LVM and LVM is again on 
top of DRBD to be able to create a LV for each DomU. 

Network:
switches are HP Procurve 1810. Not the fastest switches, but also not the most 
expensive ones. Will report later on if they can handle it all.

Hypervisors:
different machines, but for the moment all having 4 NICs. One NIC is for the 
trusted LAN, two are used for iSCSI initiating. One for DomUs in the DMZ. I 
use multipathing on top of the iSCSI paths for redundancy and supposedbly 
extra speed (his hasn''t been proven yet). The paths run over different 
switches for redundancy.

DomUs:
Currently HVMs. Will have about 10 in the end. They use phy: devices pointing 
to the multipath devices. config files are synced across the Hypervisors (no 
network storage for avoiding SPOF). 

HA:
(to do) Pacemaker will take care of monitoring DomUs and failing them over.

Backup:
It seems the only save way to backup DomUs is by shutting them down, so what I 
do is make sure the storage servers can ssh to the Hypervisors with public key 
auth. They will shut down the guests, create a snapshot volume of the relevant 
LV for that particular machine (a script finds out where it is running), start 
the guests again and dd the snapshot  to a file server over ssh. Next the 
snapshot is deleted. 


There you go, hope this can inspire people.  ;-)

B.




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

This is fascinating because it is almost exactly what I have been
doing (though the design has waffled between a single highly redundant
single storage server and two DRBD-redundant storage servers).  We use
Dell 5448 switches instead of the procurve, crossover 10gbit between
the two DRBD machines, and also add offsite DRBD replication, but
otherwise it''s the same design. Oh, and I use heartbeat over pacemaker
because it''s so easy. :)

IMO this a very logical approach that is both supportable and
scalable.  The only real trick in my experience is getting the
initiator multipath device configuration handled automatically as part
of the IET lun provisioning process.

Well done!  Hopefully others can share their own designs and experiences.

--frank

On Sat, Jul 3, 2010 at 5:58 AM, Bart Coninckx <bart.coninckx@telenet.be>
wrote:
> Hi all,
>
> In threads posted by I believe Jonathan Tripley I promised to post my new
XEN
> HA setup. Hope it can be of some use to some people.
>
> In this particular case I''m forced to use SLES 10SP3 with XEN 3.2,
which
> excludes the possibility of using things like cLVM (which I don''t
think I need
> anyway).
>
> So:
>
> Storage:
> I use two HP ML370 G5 machines with DRBD and heartbeat on them. They are
> linked by two Gigabit bonded NICs for syncing. They offer IET across two
other
> NICs with IPs in different segments. DRBD is on top of LVM and LVM is again
on
> top of DRBD to be able to create a LV for each DomU.
>
> Network:
> switches are HP Procurve 1810. Not the fastest switches, but also not the
most
> expensive ones. Will report later on if they can handle it all.
>
> Hypervisors:
> different machines, but for the moment all having 4 NICs. One NIC is for
the
> trusted LAN, two are used for iSCSI initiating. One for DomUs in the DMZ. I
> use multipathing on top of the iSCSI paths for redundancy and supposedbly
> extra speed (his hasn''t been proven yet). The paths run over
different
> switches for redundancy.
>
> DomUs:
> Currently HVMs. Will have about 10 in the end. They use phy: devices
pointing
> to the multipath devices. config files are synced across the Hypervisors
(no
> network storage for avoiding SPOF).
>
> HA:
> (to do) Pacemaker will take care of monitoring DomUs and failing them over.
>
> Backup:
> It seems the only save way to backup DomUs is by shutting them down, so
what I
> do is make sure the storage servers can ssh to the Hypervisors with public
key
> auth. They will shut down the guests, create a snapshot volume of the
relevant
> LV for that particular machine (a script finds out where it is running),
start
> the guests again and dd the snapshot  to a file server over ssh. Next the
> snapshot is deleted.
>
>
> There you go, hope this can inspire people.  ;-)
>
> B.
>
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Saturday 03 July 2010 14:30:12 Frank S Fejes III
wrote:
> This is fascinating because it is almost exactly what I have been
> doing (though the design has waffled between a single highly redundant
> single storage server and two DRBD-redundant storage servers).  We use
> Dell 5448 switches instead of the procurve, crossover 10gbit between
> the two DRBD machines, and also add offsite DRBD replication, but
> otherwise it''s the same design. Oh, and I use heartbeat over
pacemaker
> because it''s so easy. :)
Hi Frank,

thank you. Actually, I should correct myself: it is Heartbeat, but with the 
CRM. 
> IMO this a very logical approach that is both supportable and
> scalable.  The only real trick in my experience is getting the
> initiator multipath device configuration handled automatically as part
> of the IET lun provisioning process.
Good point. Up till now I do this by hand, following a written procedure, but 
it should not be a big deal to code this is bash. Only would make sense with 
fast adding guests though.
> Well done!  Hopefully others can share their own designs and experiences.
> 
> --frank
thx again, looking forward to other ideas. 

B.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 03/07/10 11:58, Bart Coninckx wrote:
> Hi all,
>
> In threads posted by I believe Jonathan Tripley I promised to post my new
XEN
> HA setup. Hope it can be of some use to some people.
>
> In this particular case I''m forced to use SLES 10SP3 with XEN 3.2,
which
> excludes the possibility of using things like cLVM (which I don''t
think I need
> anyway).
>
> So:
>
> Storage:
> I use two HP ML370 G5 machines with DRBD and heartbeat on them. They are
> linked by two Gigabit bonded NICs for syncing. They offer IET across two
other
> NICs with IPs in different segments. DRBD is on top of LVM and LVM is again
on
> top of DRBD to be able to create a LV for each DomU.
>
> Network:
> switches are HP Procurve 1810. Not the fastest switches, but also not the
most
> expensive ones. Will report later on if they can handle it all.
>
> Hypervisors:
> different machines, but for the moment all having 4 NICs. One NIC is for
the
> trusted LAN, two are used for iSCSI initiating. One for DomUs in the DMZ. I
> use multipathing on top of the iSCSI paths for redundancy and supposedbly
> extra speed (his hasn''t been proven yet). The paths run over
different
> switches for redundancy.
>
> DomUs:
> Currently HVMs. Will have about 10 in the end. They use phy: devices
pointing
> to the multipath devices. config files are synced across the Hypervisors
(no
> network storage for avoiding SPOF).
>
> HA:
> (to do) Pacemaker will take care of monitoring DomUs and failing them over.
>
> Backup:
> It seems the only save way to backup DomUs is by shutting them down, so
what I
> do is make sure the storage servers can ssh to the Hypervisors with public
key
> auth. They will shut down the guests, create a snapshot volume of the
relevant
> LV for that particular machine (a script finds out where it is running),
start
> the guests again and dd the snapshot  to a file server over ssh. Next the
> snapshot is deleted.
>
>
> There you go, hope this can inspire people.  ;-)
>
> B.
>
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>    
Hi Bart,

Thanks for this, I''m sure some of us will find tihs useful :)

Thanks

Jonathan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sunday 04 July 2010 18:15:56 Jonathan Tripathy wrote:
> On 03/07/10 11:58, Bart Coninckx wrote:
> > Hi all,
> >
> > In threads posted by I believe Jonathan Tripley I promised to post my
new
> > XEN HA setup. Hope it can be of some use to some people.
> >
> > In this particular case I''m forced to use SLES 10SP3 with XEN
3.2, which
> > excludes the possibility of using things like cLVM (which I
don''t think I
> > need anyway).
> >
> > So:
> >
> > Storage:
> > I use two HP ML370 G5 machines with DRBD and heartbeat on them. They
are
> > linked by two Gigabit bonded NICs for syncing. They offer IET across
two
> > other NICs with IPs in different segments. DRBD is on top of LVM and
LVM
> > is again on top of DRBD to be able to create a LV for each DomU.
> >
> > Network:
> > switches are HP Procurve 1810. Not the fastest switches, but also not
the
> > most expensive ones. Will report later on if they can handle it all.
> >
> > Hypervisors:
> > different machines, but for the moment all having 4 NICs. One NIC is
for
> > the trusted LAN, two are used for iSCSI initiating. One for DomUs in
the
> > DMZ. I use multipathing on top of the iSCSI paths for redundancy and
> > supposedbly extra speed (his hasn''t been proven yet). The
paths run over
> > different switches for redundancy.
> >
> > DomUs:
> > Currently HVMs. Will have about 10 in the end. They use phy: devices
> > pointing to the multipath devices. config files are synced across the
> > Hypervisors (no network storage for avoiding SPOF).
> >
> > HA:
> > (to do) Pacemaker will take care of monitoring DomUs and failing them
> > over.
> >
> > Backup:
> > It seems the only save way to backup DomUs is by shutting them down,
so
> > what I do is make sure the storage servers can ssh to the Hypervisors
> > with public key auth. They will shut down the guests, create a
snapshot
> > volume of the relevant LV for that particular machine (a script finds
out
> > where it is running), start the guests again and dd the snapshot  to a
> > file server over ssh. Next the snapshot is deleted.
> >
> >
> > There you go, hope this can inspire people.  ;-)
> >
> > B.
> >
> >
> >
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users
> 
> Hi Bart,
> 
> Thanks for this, I''m sure some of us will find tihs useful :)
> 
> Thanks
> 
> Jonathan
> 
Well, sized differently it could do what you were looking for, although the 
amount of DomUs you were looking to run per Dom0 would require upscaling it 
considerably. That kind of setup would probably also benefit from a cloud 
management platform like, let''s say, openQRM.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users