Xen users - May 2010 - High availability and live migration with just LVM and hardware fencing

Hi all,

I''ve been studying some tutorials and howtos about Xen Live Migration
and High
Availability in preparation for my new setup (which should do both).

I like the KISS principle (Keep It Simple, Stupid) and I was thinking about 
using iSCSI LUNs on which I would create LVM logical volumes to be used as 
block devices for Xen hosts after being activated in evry individual host. 
I''ve tested this for Live Migration and that seems to work flawlessly. 

So i figured I could use this as well for HA. Of course in order to prevent 
several Xen hosts actively accessing the same block devices, I would use 
Heartbeat 2 and hardware fencing to kill a host reporting an unstable state.

However, I see a lot of references to using cLVM or EMVS when sharing storage. 
Is this really necessary in my setup? I would like to avoid the added 
complexity. Also, I''m using SLES10 and cLVM is not supported on that
yet,
while EVMS is going to be phased out on this distro. OCFS2 seems only usefull 
when using file based images, which I won''t.


Thx!!!


Bart

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wednesday 19 May 2010 11:46:18 Bart Coninckx wrote:
> Hi all,
> 
> I''ve been studying some tutorials and howtos about Xen Live
Migration and
>  High Availability in preparation for my new setup (which should do both).
> 
> I like the KISS principle (Keep It Simple, Stupid) and I was thinking about
> using iSCSI LUNs on which I would create LVM logical volumes to be used as
> block devices for Xen hosts after being activated in evry individual host.
> I''ve tested this for Live Migration and that seems to work
flawlessly.
> 
> So i figured I could use this as well for HA. Of course in order to prevent
> several Xen hosts actively accessing the same block devices, I would use
> Heartbeat 2 and hardware fencing to kill a host reporting an unstable
>  state.
> 
> However, I see a lot of references to using cLVM or EMVS when sharing
>  storage. Is this really necessary in my setup? I would like to avoid the
>  added complexity. Also, I''m using SLES10 and cLVM is not
supported on that
>  yet, while EVMS is going to be phased out on this distro. OCFS2 seems only
>  usefull when using file based images, which I won''t.
> 
> 
> Thx!!!
> 
> 
> Bart
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
Did some further research in the meantime and so far the best reason I can 
find for using cLVM, is the making changes to the volume groups: with cLVM 
they seem to be "published" to all Xen hosts at once, while with LVM
one needs
to reconnect to the volume group (probably causing weird things to the 
guests).

Still open to your comments,


Bart

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Did some further research in the meantime and so far the best reason I can 
> find for using cLVM, is the making changes to the volume groups: with cLVM 
> they seem to be "published" to all Xen hosts at once, while with
LVM one needs
> to reconnect to the volume group (probably causing weird things to the 
> guests).
Agreed, I wouldn''t touch his setup without clvm being there.  It does 
other nice things like making sure each cluster member can see the block 
devices before allowing changes to LVM state, which is nice when you 
have to zone and mask dozens of disks.  cLVM is a must in my book.

John





-- 
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com [mailto:xen-users-
> bounces@lists.xensource.com] On Behalf Of Bart Coninckx
> Sent: Wednesday, May 19, 2010 9:00 AM
> To: xen-users@lists.xensource.com
> Subject: Re: [Xen-users] High availability and live migration with
just LVMand hardware
> fencing
> 
> Did some further research in the meantime and so far the best reason I
can
> find for using cLVM, is the making changes to the volume groups: with
cLVM
> they seem to be "published" to all Xen hosts at once, while with
LVM
one needs
> to reconnect to the volume group (probably causing weird things to the
> guests).
We''ve done it with and without CLVM.  It''s not a requirement,
but it
works, and can compensate for storage devices that make it cumbersome to
manage the number of LUNs needed for your deployment.

Advantanges of CLVM:

- Fully clustered, uses robust fencing, no single point of failure
- Integration with device mapper allows for intuitive logical volume
names
- Works identically atop any network block storage (iSCSI etc.)
- Supports mirroring of block devices in recent releases
- Synchronizes volume group changes across cluster

Disadvantages:

- No support for volume snapshots (maybe someday--last I tried it
didn''t
work)
- Not available or convenient to use with all OS distributions
- Requires cluster infrastructure--difficult to use with system
partitions (e.g. /, /var)
- No thin provisioning

For my money I prefer the LUN management that ships with certain SAN
products (such as Dell''s Equallogic arrays) over CLVM.  As we consider
moving away from a RHEL-supported dom0 kernel towards something like XCP
or XenServer, it''ll become a necessity.

-Jeff



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wednesday 19 May 2010 15:43:19 Jeff Sturm wrote:
> > -----Original Message-----
> > From: xen-users-bounces@lists.xensource.com [mailto:xen-users-
> > bounces@lists.xensource.com] On Behalf Of Bart Coninckx
> > Sent: Wednesday, May 19, 2010 9:00 AM
> > To: xen-users@lists.xensource.com
> > Subject: Re: [Xen-users] High availability and live migration with
> 
> just LVMand hardware
> 
> > fencing
> >
> > Did some further research in the meantime and so far the best reason I
> 
> can
> 
> > find for using cLVM, is the making changes to the volume groups: with
> 
> cLVM
> 
> > they seem to be "published" to all Xen hosts at once, while
with LVM
> 
> one needs
> 
> > to reconnect to the volume group (probably causing weird things to the
> > guests).
> 
> We''ve done it with and without CLVM.  It''s not a
requirement, but it
> works, and can compensate for storage devices that make it cumbersome to
> manage the number of LUNs needed for your deployment.
> 
> Advantanges of CLVM:
> 
> - Fully clustered, uses robust fencing, no single point of failure
> - Integration with device mapper allows for intuitive logical volume
> names
> - Works identically atop any network block storage (iSCSI etc.)
> - Supports mirroring of block devices in recent releases
> - Synchronizes volume group changes across cluster
> 
> Disadvantages:
> 
> - No support for volume snapshots (maybe someday--last I tried it
didn''t
> work)
> - Not available or convenient to use with all OS distributions
> - Requires cluster infrastructure--difficult to use with system
> partitions (e.g. /, /var)
> - No thin provisioning
> 
> For my money I prefer the LUN management that ships with certain SAN
> products (such as Dell''s Equallogic arrays) over CLVM.  As we
consider
> moving away from a RHEL-supported dom0 kernel towards something like XCP
> or XenServer, it''ll become a necessity.
> 
> -Jeff
> 
Jeff,

a well put conclusion. 
Your second disadvantage particularly is relevant to me, since I need to use 
the commercially supported SLES10SP3, which has no cLVM available in it. 

Also the first one is a, to say the least, disappointing. Having the 
possibility of snapshotting is most valuable in the context of 
updating/rollback. 

By the way: would it be correct to state that cLVM is only an advantage when 
dealing with more than one volume group (being the result of having more than 
one LUN)?


thx!

B.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, May 19, 2010 at 9:08 AM, Bart Coninckx <bart.coninckx@telenet.be>
wrote:
> By the way: would it be correct to state that cLVM is only an advantage
when
> dealing with more than one volume group (being the result of having more
than
> one LUN)?
no.

- as soon as more than one host can R/W a volume group, you need to
coordinate volume management changes somehow.  if you change anything
on one host without (lock-safely) propagating it to the others, you''ll
almost surely corrupt the LVM metadata.  if you rarely change it, you
can get by without cLVM, but then you have to take all hosts offline
while you commit changes.

- you can have many LUNs on a single volume group, just add each as a PV

-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wednesday 19 May 2010 17:49:15 Javier Guerra Giraldez
wrote:
> On Wed, May 19, 2010 at 9:08 AM, Bart Coninckx
<bart.coninckx@telenet.be>
wrote:
> > By the way: would it be correct to state that cLVM is only an
advantage
> > when dealing with more than one volume group (being the result of
having
> > more than one LUN)?
> 
> no.
> 
> - as soon as more than one host can R/W a volume group, you need to
> coordinate volume management changes somehow.  if you change anything
> on one host without (lock-safely) propagating it to the others,
you''ll
> almost surely corrupt the LVM metadata.  if you rarely change it, you
> can get by without cLVM, but then you have to take all hosts offline
> while you commit changes.
> 
> - you can have many LUNs on a single volume group, just add each as a PV
Hi,

What would you define as "change anything"? Add a LV? I suppose a
"lvscan" on
every node would take care of the changes, no? Or does it mean I cannot  take 
a snapshot? Or resize? Or all?

thx!!

B.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, May 19, 2010 at 1:27 PM, Bart Coninckx <bart.coninckx@telenet.be>
wrote:
> What would you define as "change anything"? Add a LV?
for example, or expanding one, or deleting.  or adding a PV...
>  I suppose a "lvscan" on
> every node would take care of the changes, no?
yes, but what about the time between the change and the lvscan on all
nodes?  during that time, nonupdated nodes have a wrong idea of the VG
structure, and that is deadly dangerous.
> Or does it mean I cannot  take
> a snapshot? Or resize? Or all?
nothing that modifies the LVM metadata.


-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wednesday 19 May 2010 21:01:03 Javier Guerra Giraldez
wrote:
> On Wed, May 19, 2010 at 1:27 PM, Bart Coninckx
<bart.coninckx@telenet.be>
wrote:
> > What would you define as "change anything"? Add a LV?
> 
> for example, or expanding one, or deleting.  or adding a PV...
> 
> >  I suppose a "lvscan" on
> > every node would take care of the changes, no?
> 
> yes, but what about the time between the change and the lvscan on all
> nodes?  during that time, nonupdated nodes have a wrong idea of the VG
> structure, and that is deadly dangerous.
> 
> > Or does it mean I cannot  take
> > a snapshot? Or resize? Or all?
> 
> nothing that modifies the LVM metadata.
> 
I see ... Well, there are two good reasons I cannot have cLVM: need snapshots 
and it''s simply not available for SELS10, so I''m going to keep
on asking if
permitted. 

- What if I would use a seperate volume group for every LV (and for every Xen 
guest)? Would that help? I suppose not really, since the Xen guest should be 
migratable, meaning that all nodes would need to have a consistent view no all 
VG''s and LV''s. 
- would snapshotting change the LVM meta data?

Cheers,


Bart

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I see ... Well, there are two good reasons I cannot have cLVM: need
snapshots
> and it''s simply not available for SELS10, so I''m going to
keep on asking if
> permitted. 
clvm doesn''t do snapshots, so you can''t do snapshots from
dom0.
Snapshots aren''t of much use in dom0, though (assuming you''re
handing
LV''s to domU''s, which have their own caches), so why does that
matter?
Do your snapshots from within your domU''s.  cLVM may not be available
on
SLES10 out of the box, but that doesn''t mean you can''t add it.
> - What if I would use a seperate volume group for every LV (and for every
Xen
> guest)? Would that help? I suppose not really, since the Xen guest should
be
> migratable, meaning that all nodes would need to have a consistent view no
all
> VG''s and LV''s. 
Metadata changes cause writes to disks.  Not safe.  I''ve done this 
before with active/passive clusters where you could somewhat assume some 
safety (i.e., lvm isn''t even running on the passive node) but
I''d never
do it on
> - would snapshotting change the LVM meta data?
Yes, a snapshot is a LV.




-- 
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wednesday 19 May 2010 22:31:29 John Madden wrote:
> > I see ... Well, there are two good reasons I cannot have cLVM: need
> > snapshots and it''s simply not available for SELS10, so
I''m going to keep
> > on asking if permitted.
> 
> clvm doesn''t do snapshots, so you can''t do snapshots from
dom0.
> Snapshots aren''t of much use in dom0, though (assuming
you''re handing
> LV''s to domU''s, which have their own caches), so why does
that matter?
> Do your snapshots from within your domU''s.  cLVM may not be
available on
> SLES10 out of the box, but that doesn''t mean you can''t
add it.
Well, for (HVM) Windows guests I see it as a necessity basically: next to 
backups of regular data, snapshotting will allow to take fast complete backups 
of a temprarely suspended OS. 

And indeed, cLVM can be added from source, but while applying for support one 
of the first things Novell asks you to do is to send them a supportconfig 
report, which amongst others lists loaded modules (I suppose cLVM uses a 
module). This might rise problems. 
> > - What if I would use a seperate volume group for every LV (and for
every
> > Xen guest)? Would that help? I suppose not really, since the Xen guest
> > should be migratable, meaning that all nodes would need to have a
> > consistent view no all VG''s and LV''s.
> 
> Metadata changes cause writes to disks.  Not safe.  I''ve done this
> before with active/passive clusters where you could somewhat assume some
> safety (i.e., lvm isn''t even running on the passive node) but
I''d never
> do it on
That would indeed seem safe to me, if you have something like Heartbeat manage 
volume activation.
> > - would snapshotting change the LVM meta data?
> 
> Yes, a snapshot is a LV.
OK, it seems that I will be needing cLVM or EVMS after all. At least
I''m able
to avoid something like OCFS2 that would add another layer of complexity. 


Thank you all for your insight on this.


Rgds,


Bart


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: Bart Coninckx [mailto:bart.coninckx@telenet.be]
> Sent: Wednesday, May 19, 2010 5:53 PM
> To: John Madden
> Cc: Javier Guerra Giraldez; Jeff Sturm; xen-users@lists.xensource.com
> Subject: Re: [Xen-users] High availability and live migration with just
LVMand hardware
> fencing
> 
> Well, for (HVM) Windows guests I see it as a necessity basically: next to
> backups of regular data, snapshotting will allow to take fast complete
backups
> of a temprarely suspended OS.
This may not be any help to you (yet), but I've read that blktap2 in Xen 4.0
can also create snapshots.

I agree snapshots are very useful, although not a substitute for conventional
backups.  Among other places, we use them on GFS filesystems--we run gfs_tool to
freeze the filesystem, create a snapshot on our SAN, then unfreeze.  These
provide a nice recovery point if disaster strikes.

-Jeff


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thursday 20 May 2010 00:48:54 Jeff Sturm wrote:
> > -----Original Message-----
> > From: Bart Coninckx [mailto:bart.coninckx@telenet.be]
> > Sent: Wednesday, May 19, 2010 5:53 PM
> > To: John Madden
> > Cc: Javier Guerra Giraldez; Jeff Sturm; xen-users@lists.xensource.com
> > Subject: Re: [Xen-users] High availability and live migration with
just
> > LVMand hardware fencing
> >
> > Well, for (HVM) Windows guests I see it as a necessity basically: next
to
> > backups of regular data, snapshotting will allow to take fast complete
> > backups of a temprarely suspended OS.
> 
> This may not be any help to you (yet), but I''ve read that blktap2
in Xen
>  4.0 can also create snapshots.
> 
> I agree snapshots are very useful, although not a substitute for
>  conventional backups.  Among other places, we use them on GFS
>  filesystems--we run gfs_tool to freeze the filesystem, create a snapshot
>  on our SAN, then unfreeze.  These provide a nice recovery point if
>  disaster strikes.
> 
> -Jeff
> 
Xen 4.0 is not an option, it being too young for production. I will look into 
GFS. It''s somewhat challenging: choosing in between cLVM, EVMS, GFS,
OCFS2.
Hopefully the best solution for our situation will come surfacing soon  ...


Cheers,

B.
 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thursday 20 May 2010 00:48:54 Jeff Sturm wrote:
> > -----Original Message-----
> > From: Bart Coninckx [mailto:bart.coninckx@telenet.be]
> > Sent: Wednesday, May 19, 2010 5:53 PM
> > To: John Madden
> > Cc: Javier Guerra Giraldez; Jeff Sturm; xen-users@lists.xensource.com
> > Subject: Re: [Xen-users] High availability and live migration with
just
> > LVMand hardware fencing
> >
> > Well, for (HVM) Windows guests I see it as a necessity basically: next
to
> > backups of regular data, snapshotting will allow to take fast complete
> > backups of a temprarely suspended OS.
> 
> This may not be any help to you (yet), but I''ve read that blktap2
in Xen
>  4.0 can also create snapshots.
> 
> I agree snapshots are very useful, although not a substitute for
>  conventional backups.  Among other places, we use them on GFS
>  filesystems--we run gfs_tool to freeze the filesystem, create a snapshot
>  on our SAN, then unfreeze.  These provide a nice recovery point if
>  disaster strikes.
> 
> -Jeff
> 
I did some more thinking in regards to the snapshotting and came up with the 
following: the backend storage is an IET storage target. I could build the 
LUNs on top of LV''s. This would give me snapshotting for the Xen setup.
Granted, not very elegant, since for a restore I need to put the snapshot back 
and as a added action I need to connect to it from an iSCSI initiator in order 
to access the data. 

Would this be a "clever idea"? 

thx,


B.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I did some more thinking in regards to the snapshotting and came up with
the
> following: the backend storage is an IET storage target. I could build the 
> LUNs on top of LV''s. This would give me snapshotting for the Xen
setup.
> Granted, not very elegant, since for a restore I need to put the snapshot
back
> and as a added action I need to connect to it from an iSCSI initiator in
order
> to access the data. 
You''ll have the same problems here as you will snapshotting from dom0
--
filesystem cache and application state.  If the OS hasn''t flushed its 
cache and frozen the filesystem, any snapshot from the block level would 
contain an inconsistent filesystem.  This is not an acceptable backup 
mechanism and I think doing it from IET (i.e., the SAN) is even worse 
because you have yet another layer of abstraction and potential for caches.

Note Jeff''s note on SAN snapshots and gfs_tool.  His snapshots work 
because the OS is made aware of the need to flush caches and temporarily 
queue writes to the disk.

John





-- 
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thursday 20 May 2010 15:21:10 John Madden wrote:
> > I did some more thinking in regards to the snapshotting and came up
with
> > the following: the backend storage is an IET storage target. I could
> > build the LUNs on top of LV''s. This would give me
snapshotting for the
> > Xen setup. Granted, not very elegant, since for a restore I need to
put
> > the snapshot back and as a added action I need to connect to it from
an
> > iSCSI initiator in order to access the data.
> 
> You''ll have the same problems here as you will snapshotting from
dom0 --
> filesystem cache and application state.  If the OS hasn''t flushed
its
> cache and frozen the filesystem, any snapshot from the block level would
> contain an inconsistent filesystem.  This is not an acceptable backup
> mechanism and I think doing it from IET (i.e., the SAN) is even worse
> because you have yet another layer of abstraction and potential for caches.
> 
> Note Jeff''s note on SAN snapshots and gfs_tool.  His snapshots
work
> because the OS is made aware of the need to flush caches and temporarily
> queue writes to the disk.
> 
> John
> 
Would a "xm suspend" before the snapstho not accomplish this? 

Cheers,

Bart

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>> Note Jeff''s note on SAN snapshots and gfs_tool.  His snapshots
work
>> because the OS is made aware of the need to flush caches and
temporarily
>> queue writes to the disk.
>>
>> John
>>
> 
> Would a "xm suspend" before the snapstho not accomplish this? 
I think you mean "xm pause."  That pauses or freezes the domain, but 
does nothing more to it.  Anything in memory on the guest remains in 
memory, not flushed to disk.  You need something inside the guest that 
tells the kernel to flush everything, fsyncs, etc., then makes no 
further changes to the filesystem until un-frozen.

John




-- 
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thursday 20 May 2010 17:24:23 John Madden wrote:
> >> Note Jeff''s note on SAN snapshots and gfs_tool.  His
snapshots work
> >> because the OS is made aware of the need to flush caches and
temporarily
> >> queue writes to the disk.
> >>
> >> John
> >
> > Would a "xm suspend" before the snapstho not accomplish
this?
> 
> I think you mean "xm pause."  That pauses or freezes the domain,
but
> does nothing more to it.  Anything in memory on the guest remains in
> memory, not flushed to disk.  You need something inside the guest that
> tells the kernel to flush everything, fsyncs, etc., then makes no
> further changes to the filesystem until un-frozen.
> 
> John
> 
In case of Windows HVM guests quite a challenge guess ... "suspend" in
Vmware
context I believe really tells the OS to suspend (I think because it relies on 
the installed Vmware Tools)

If GFS can bring something to the table for this, I''ll check it out,
but it''s
again one more layer of unsupported complexity to SLES. 

Thx John, quite insightful!

Rgds,

B.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> In case of Windows HVM guests quite a challenge guess ...
"suspend" in Vmware
> context I believe really tells the OS to suspend (I think because it relies
on
> the installed Vmware Tools)
What''s wrong with backing up from the clients?
> If GFS can bring something to the table for this, I''ll check it
out, but it''s
> again one more layer of unsupported complexity to SLES. 
Doesn''t GFS require cLVM?  Maybe that''s just a requirement
with RHEL.
Or maybe I remembered it wrong.  At any rate, it doesn''t help you with 
your HVM guests because they''ll have no concept or knowledge of the 
underlying disk.

John




-- 
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

----- Original message -----
> > In case of Windows HVM guests quite a challenge guess ...
"suspend" in Vmware
> > context I believe really tells the OS to suspend (I think because it
relies on
> > the installed Vmware Tools)
>
> What''s wrong with backing up from the clients?
You mean the guests? Well, it could be they get corrupted while for instance
failing over because a xen host dies. Quickest restore path is then a snapshot
restore and then a guest data restore.
> > If GFS can bring something to the table for this, I''ll check
it out, but it''s
> > again one more layer of unsupported complexity to SLES.
>
> Doesn''t GFS require cLVM?  Maybe that''s just a
requirement with RHEL.
I guess it does. Neither are available on sles, so would require installation
from source.
> Or maybe I remembered it wrong.  At any rate, it doesn''t help you
with
> your HVM guests because they''ll have no concept or knowledge of
the
> underlying disk.
>
> John
>
Wait a sec, so i still would have no snapshots? How is this achieved then with
hvm guests?
Do they need to be stopped  for backups? Live migration still works, right?
(Provided i have clvm)
>
> --
> John Madden
> Sr UNIX Systems Engineer
> Ivy Tech Community College of Indiana
> jmadden@ivytech.edu
Cheers,

Bart

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>  > What''s wrong with backing up from the clients?
> 
> You mean the guests? Well, it could be they get corrupted while for 
> instance failing over because a xen host dies. Quickest restore path is 
> then a snapshot restore and then a guest data restore.
If you''re doing guest data restore once the guest is up, why bother
with
snapshots at all?  (In other words, you can deploy VMs all day from a 
vanilla install process and then restore data, so snapshots don''t
really
buy you anything.)
> Wait a sec, so i still would have no snapshots? How is this achieved 
> then with hvm guests?
Correct.  I don''t know how anyone accomplishes this or thinks that they
do, none of it adds up as far as I''m concerned.  If you want consistent
backups, they have to happen either FROM the client or somehow in 
concert with it.

You wouldn''t run fsck on a mounted filesystem, would you?  Or assume 
that a ''dd if=/dev/sda'' backup would be consistent while you
were
writing data to the disk, right?  Well, taking snapshots "from the
san"
or "from dom0" without "letting the client know" is the same
thing.
> Do they need to be stopped for backups? Live migration still works, 
> right? (Provided i have clvm)
LVM+cLVM is only one way to accomplish live migration.  You can also do 
it with disk images stored on NFS or a clustered filesystem like OCFS2, 
for example.  I happen to like LVM+cLVM because I prefer the nature and 
performance advantage of phy: disk assignment.

John




-- 
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Friday 21 May 2010 15:32:22 John Madden wrote:
> >  > What''s wrong with backing up from the clients?
> >
> > You mean the guests? Well, it could be they get corrupted while for
> > instance failing over because a xen host dies. Quickest restore path
is
> > then a snapshot restore and then a guest data restore.
> 
> If you''re doing guest data restore once the guest is up, why
bother with
> snapshots at all?  (In other words, you can deploy VMs all day from a
> vanilla install process and then restore data, so snapshots don''t
really
> buy you anything.)
John,

Basically to safe time. Your reasoning makes sense for Linux systems where 
restore is just a copy of files, but for Windows HVM guests the restore 
process could be really lon because of installation of individual programs, 
settings, mailboxes etc. 
> > Wait a sec, so i still would have no snapshots? How is this achieved
> > then with hvm guests?
> 
> Correct.  I don''t know how anyone accomplishes this or thinks that
they
> do, none of it adds up as far as I''m concerned.  If you want
consistent
> backups, they have to happen either FROM the client or somehow in
> concert with it.
> 
> You wouldn''t run fsck on a mounted filesystem, would you?  Or
assume
> that a ''dd if=/dev/sda'' backup would be consistent while
you were
> writing data to the disk, right?  Well, taking snapshots "from the
san"
> or "from dom0" without "letting the client know" is the
same thing.
Well, maybe I''m a bit influenced by my Vmware past, where it is
possible to
put a Windows guest in a suspended state, having it flushed all data to disk 
and remaining in a frozen state, where it is possbile to then take a quick LVM 
snaphost, start the guest again and copy the LVM snapshot for possible future 
restore. I guess I am looking for the same thing with Xen. But it would 
require something like passing a ACPI suspend command to the guest; I believe 
Vmware does this, either by using Vmware tools, either by sending the ACPI 
command.
> > Do they need to be stopped for backups? Live migration still works,
> > right? (Provided i have clvm)
> 
> LVM+cLVM is only one way to accomplish live migration.  You can also do
> it with disk images stored on NFS or a clustered filesystem like OCFS2,
> for example.  I happen to like LVM+cLVM because I prefer the nature and
> performance advantage of phy: disk assignment.
I recently read that Suse changed the Xen scripts to be able to use an iSCSI 
target directly as vdb in the Xen config file, allowing live migration without 
cLVM. One user reported that this worked without problems for him. What are 
your thoughts about that?

Cheers,


Bart
> John
> 
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, May 19, 2010 at 3:31 PM, John Madden <jmadden@ivytech.edu> wrote:
> Metadata changes cause writes to disks.  Not safe.  I''ve done this
before
> with active/passive clusters where you could somewhat assume some safety
> (i.e., lvm isn''t even running on the passive node) but
I''d never do it on
John, I''m interested as to why you feel this is unsafe and what bad
experiences you may have had doing shared lvm in a manual (ie,
non-clvm) fashion.  In clusters of up to six Xen hosts per iscsi
target I''ve been using a combination of scripted lvchange/lvscan
commands in lvm wrappers and have never yet run into corruption.  As
far as I''m aware, there''s nothing magical that clvm is doing
under the
covers besides locking and if all lvm commands are run via the
"clustered" wrappers then the metadata should not be changing
unexpectedly.

Is there something I''m missing?  It''s a fascinating topic and
I''d love
to read what others have done.  Thanks.

--frank

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> John, I''m interested as to why you feel this is unsafe and what
bad
> experiences you may have had doing shared lvm in a manual (ie,
> non-clvm) fashion.  In clusters of up to six Xen hosts per iscsi
> target I''ve been using a combination of scripted lvchange/lvscan
> commands in lvm wrappers and have never yet run into corruption.  As
> far as I''m aware, there''s nothing magical that clvm is
doing under the
> covers besides locking and if all lvm commands are run via the
> "clustered" wrappers then the metadata should not be changing
> unexpectedly.
(Btw, I''m not a LVM expert or anything.)

If you carefully coordinate changes to the metadata and, for example, 
reload the data on all cluster members on every change, I think you 
would be ok.  CLVM takes care of all this for you and uses locking to 
ensure changes on one node can''t clash with other nodes.

John




-- 
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Monday 24 May 2010 19:29:35 John Madden wrote:
> > John, I''m interested as to why you feel this is unsafe and
what bad
> > experiences you may have had doing shared lvm in a manual (ie,
> > non-clvm) fashion.  In clusters of up to six Xen hosts per iscsi
> > target I''ve been using a combination of scripted
lvchange/lvscan
> > commands in lvm wrappers and have never yet run into corruption.  As
> > far as I''m aware, there''s nothing magical that clvm
is doing under the
> > covers besides locking and if all lvm commands are run via the
> > "clustered" wrappers then the metadata should not be
changing
> > unexpectedly.
> 
> (Btw, I''m not a LVM expert or anything.)
> 
> If you carefully coordinate changes to the metadata and, for example,
> reload the data on all cluster members on every change, I think you
> would be ok.  CLVM takes care of all this for you and uses locking to
> ensure changes on one node can''t clash with other nodes.
> 
> John
> 
Quite interesting point of view, since it is so different from the one of 
authors commenting before ...
This is somewhat difficult in regards to taking decisions on setups: one party 
says "don''t", the other says "do".

Rgds,


Bart

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, May 24, 2010 at 1:00 PM, Bart Coninckx <bart.coninckx@telenet.be>
wrote:
> On Monday 24 May 2010 19:29:35 John Madden wrote:
>> > John, I''m interested as to why you feel this is unsafe
and what bad
>> > experiences you may have had doing shared lvm in a manual (ie,
>> > non-clvm) fashion.  In clusters of up to six Xen hosts per iscsi
>> > target I''ve been using a combination of scripted
lvchange/lvscan
>> > commands in lvm wrappers and have never yet run into corruption.
 As
>> > far as I''m aware, there''s nothing magical that
clvm is doing under the
>> > covers besides locking and if all lvm commands are run via the
>> > "clustered" wrappers then the metadata should not be
changing
>> > unexpectedly.
>>
>> If you carefully coordinate changes to the metadata and, for example,
>> reload the data on all cluster members on every change, I think you
>> would be ok.  CLVM takes care of all this for you and uses locking to
>> ensure changes on one node can''t clash with other nodes.
> Quite interesting point of view, since it is so different from the one of
> authors commenting before ...
> This is somewhat difficult in regards to taking decisions on setups: one
party
> says "don''t", the other says "do".
Well, please don''t read my post as a recommendation to "do". 
I''m only
writing to say that I''ve done it and that I do not yet have a
technical reason for why it *shouldn''t* work.  As others have pointed
out, the process to enabling clvm is long and problematic which is why
I originally set out for a way to avoid it altogether.  I think I''ve
done that and I''ve been using it with success so far.

That said, I''ve never been completely satisfied with this solution for
a number of reasons.  For one, it''s completely undocumented and
who''s
to say that a future update to lvm won''t start doing things to the
metadata that breaks my assumptions?  For another, it would seem that
very few other people are doing it which is partly why I posted in the
first place.  I''d love to see if other people such as John are doing
this and what their experiences have been.

What I''d *really* love to see is clvm decoupled from the rhel
clustering beast and packaged as a barebones easy-to-configure and
deploy option to the base lvm package.

--frank

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> What I''d *really* love to see is clvm decoupled from the rhel
> clustering beast and packaged as a barebones easy-to-configure and
> deploy option to the base lvm package.
Agreed.  After doing a few of these I find it pretty easy to set up, but 
there are a few gotchas.  (Far from the least of these gotchas is the 
suite''s use of multicast, which is tends to be problematic.)  I like 
what OCFS2 has done: built the clustering into the clustering 
filesystem.  What a concept.

John





-- 
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tuesday 25 May 2010 16:17:17 Frank S Fejes III wrote:
> On Mon, May 24, 2010 at 1:00 PM, Bart Coninckx
<bart.coninckx@telenet.be>
wrote:
> > On Monday 24 May 2010 19:29:35 John Madden wrote:
> >> > John, I''m interested as to why you feel this is
unsafe and what bad
> >> > experiences you may have had doing shared lvm in a manual
(ie,
> >> > non-clvm) fashion.  In clusters of up to six Xen hosts per
iscsi
> >> > target I''ve been using a combination of scripted
lvchange/lvscan
> >> > commands in lvm wrappers and have never yet run into
corruption.  As
> >> > far as I''m aware, there''s nothing magical
that clvm is doing under the
> >> > covers besides locking and if all lvm commands are run via
the
> >> > "clustered" wrappers then the metadata should not
be changing
> >> > unexpectedly.
> >>
> >> If you carefully coordinate changes to the metadata and, for
example,
> >> reload the data on all cluster members on every change, I think
you
> >> would be ok.  CLVM takes care of all this for you and uses locking
to
> >> ensure changes on one node can''t clash with other nodes.
> >
> > Quite interesting point of view, since it is so different from the one
of
> > authors commenting before ...
> > This is somewhat difficult in regards to taking decisions on setups:
one
> > party says "don''t", the other says "do".
> 
> Well, please don''t read my post as a recommendation to
"do".  I''m only
> writing to say that I''ve done it and that I do not yet have a
> technical reason for why it *shouldn''t* work.  As others have
pointed
> out, the process to enabling clvm is long and problematic which is why
> I originally set out for a way to avoid it altogether.  I think
I''ve
> done that and I''ve been using it with success so far.
> 
> That said, I''ve never been completely satisfied with this solution
for
> a number of reasons.  For one, it''s completely undocumented and
who''s
> to say that a future update to lvm won''t start doing things to the
> metadata that breaks my assumptions?  For another, it would seem that
> very few other people are doing it which is partly why I posted in the
> first place.  I''d love to see if other people such as John are
doing
> this and what their experiences have been.
> 
> What I''d *really* love to see is clvm decoupled from the rhel
> clustering beast and packaged as a barebones easy-to-configure and
> deploy option to the base lvm package.
> 
> --frank
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
I agree cLVM adds some considerable complexity, that''s why I plan to
avoid it
all together. To be able to do both snapshotting and live migration (for which 
I needed LVM), I decided to use iSCSI targets as vbd''s in the config
files and
to snaphost the iSCSI LUNs once the corresponding Dom0 is down. 

Will report back on how well this works.

Rgds,


Bart

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Frank S Fejes III <frank@fejes.net> writes:
> What I''d *really* love to see is clvm decoupled from the rhel
> clustering beast and packaged as a barebones easy-to-configure and
> deploy option to the base lvm package.
You obviously won''t get clvm without some sort of clustering.  But the
"rhel clustering beast" has already been split up, so you can use clvm
over bare OpenAIS or similar.
-- 
Regards,
Feri.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, May 25, 2010 at 1:01 PM, Ferenc Wagner <wferi@niif.hu>
wrote:
> You obviously won''t get clvm without some sort of clustering.  But
the
> "rhel clustering beast" has already been split up, so you can use
clvm
> over bare OpenAIS or similar.
clvm over openais is enormously easier to setup.  i''ve only seen it
packaged on opensuse.... to get on sles you have to fork some extra
$$$

-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Saturday 29 May 2010 18:36:47 Javier Guerra Giraldez
wrote:
> On Tue, May 25, 2010 at 1:01 PM, Ferenc Wagner <wferi@niif.hu> wrote:
> > You obviously won''t get clvm without some sort of clustering.
But the
> > "rhel clustering beast" has already been split up, so you
can use clvm
> > over bare OpenAIS or similar.
> 
> clvm over openais is enormously easier to setup.  i''ve only seen
it
> packaged on opensuse.... to get on sles you have to fork some extra
> $$$
> 
Hard to accept that strategy: it was included in SLES 10, for SLES 11 one has 
to pay. Only justifyable if they did there own development on it and want to 
see return for that. 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users