Xen users - Mar 2007 - How to make network storage high available?

Hello xen-users

I have a test setup with 2 xen-hosts and 1 storage host which works 
nicely. Block devices are exported from the storage host using open 
iscsi, migration works.

So far so good.

Now storage is a single point of failure and I want to avoid such.

I read a lot on the mailing list archives and all solutions I found 
about how to make storage high available consist of at least 3-5 
components like GFS, ocfs2, iscsi, LVM, CLVM, lustre, DRBD, heartbeat 
and so on. The list of possibilities seems almost endless and each one 
is quite a challenge not only to set up, even more to handle later on.

What I would like to know is how people really use Xen guests on network 
storage and how they make storage high available. Any "easier" setups?

I am aware that this is somewhat asking for a lot without seeming to 
want to do a lot for it, but I thought that asking couldn''t mind.

I like things simple and the setups I found in tutorials seem very 
complex to handle. They may work out of the box from such a tutorial, 
but what to do when something fails? So that''s kind of why I ask.

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

As noone answered, I''ll just post what I found in the meantime:

http://www.pcpro.co.uk/realworld/82284/san-on-the-cheap/page1.html

Looks nice. Going to try it out and report here.

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Dominik Klein <dk@in-telegence.net> writes:
> Hello xen-users
>
> I have a test setup with 2 xen-hosts and 1 storage host which works
> nicely. Block devices are exported from the storage host using open
> iscsi, migration works.
>
> So far so good.
>
> Now storage is a single point of failure and I want to avoid such.
>
> I read a lot on the mailing list archives and all solutions I found
> about how to make storage high available consist of at least 3-5
> components like GFS, ocfs2, iscsi, LVM, CLVM, lustre, DRBD, heartbeat
> and so on. The list of possibilities seems almost endless and each one
> is quite a challenge not only to set up, even more to handle later on.
>
> What I would like to know is how people really use Xen guests on
> network storage and how they make storage high available. Any
"easier"
> setups?
>
> I am aware that this is somewhat asking for a lot without seeming to
> want to do a lot for it, but I thought that asking couldn''t mind.
>
> I like things simple and the setups I found in tutorials seem very
> complex to handle. They may work out of the box from such a tutorial,
> but what to do when something fails? So that''s kind of why I ask.
>
> Regards
> Dominik
I''m testing with 2 server with drbd 0.8 in actice-active mode and clvm
for this. But drbd is indeed complex to handle.

You don''t need gfs or ocfs2 if you only ever have one domain accessing
an fs.

MfG
        Goswin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I''m testing with 2 server with drbd 0.8 in actice-active mode and
clvm
> for this. But drbd is indeed complex to handle.
 From what I read, active/active does not seem to be very stable yet.
> You don''t need gfs or ocfs2 if you only ever have one domain
accessing
> an fs.
I think I understood that in the meantime.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com 
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of 
> Dominik Klein
> Sent: Monday, March 26, 2007 9:59 AM
> To: xen-users@lists.xensource.com
> Subject: Re: [Xen-users] How to make network storage high available?
> 
> > I''m testing with 2 server with drbd 0.8 in actice-active 
> mode and clvm
> > for this. But drbd is indeed complex to handle.
> 
>  From what I read, active/active does not seem to be very stable yet.
> 
> > You don''t need gfs or ocfs2 if you only ever have one 
> domain accessing
> > an fs.
> 
> I think I understood that in the meantime.
Unless I really missed something, though, you''re going to need ocfs2 or
gfs2 if you''re looking to do live migration of xen machines from one
host to another...

But then, as I said, I might have missed something...

- Ian

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Unless I really missed something, though, you''re going to need
ocfs2 or
> gfs2 if you''re looking to do live migration of xen machines from
one
> host to another...
As only one xen host accesses the iscsi-target at a time, it is 
sufficient to use a standard journaling file system like ext3, xfs or 
reiserfs.

At least that''s how I understood it.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Monday 26 March 2007, Dominik Klein wrote:
> > Unless I really missed something, though, you''re going to
need ocfs2 or
> > gfs2 if you''re looking to do live migration of xen machines
from one
> > host to another...
>
> As only one xen host accesses the iscsi-target at a time, it is
> sufficient to use a standard journaling file system like ext3, xfs or
> reiserfs.
right, as long as the same volumes are visible from all Dom0''s (using
CLVM or
EVMS-ha), the migrated domU would pick it from the new box.  at this time, 
the original DomU is already suspended, so it wouldn''t be mounted from
two
kernels at any moment.


-- 
Javier


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>> > Unless I really missed something, though, you''re going to
need ocfs2
>> or
>> > gfs2 if you''re looking to do live migration of xen
machines from one
>> > host to another...
>>
>> As only one xen host accesses the iscsi-target at a time, it is
>> sufficient to use a standard journaling file system like ext3, xfs or
>> reiserfs.
>
> right, as long as the same volumes are visible from all Dom0''s
(using CLVM
> or
> EVMS-ha), the migrated domU would pick it from the new box.  at this time,
> the original DomU is already suspended, so it wouldn''t be mounted
from two
> kernels at any moment.
A question about this interesting topic. Is really nedded a cluster aware
volume manager? I''m asking this because I''m thinking in a
scenario where
two Dom0''s access a shared iSCSI disk and the DomU migrates from one
Dom0
to the other one. Do I need a cluster aware filesystem or volume manager
in that iSCSI shared disk? It would be needed if in the migration process
the DomU is executed in both Dom0 at the same time, but I''m not sure
about
this.

Thanks.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Angel Lopez wrote:
>>>> Unless I really missed something, though, you''re going
to need ocfs2
>>>>         
>>> or
>>>       
>>>> gfs2 if you''re looking to do live migration of xen
machines from one
>>>> host to another...
>>>>         
>>> As only one xen host accesses the iscsi-target at a time, it is
>>> sufficient to use a standard journaling file system like ext3, xfs
or
>>> reiserfs.
>>>       
>> right, as long as the same volumes are visible from all Dom0''s
(using CLVM
>> or
>> EVMS-ha), the migrated domU would pick it from the new box.  at this
time,
>> the original DomU is already suspended, so it wouldn''t be
mounted from two
>> kernels at any moment.
>>     
>
> A question about this interesting topic. Is really nedded a cluster aware
> volume manager? I''m asking this because I''m thinking in a
scenario where
> two Dom0''s access a shared iSCSI disk and the DomU migrates from
one Dom0
> to the other one. Do I need a cluster aware filesystem or volume manager
> in that iSCSI shared disk? It would be needed if in the migration process
> the DomU is executed in both Dom0 at the same time, but I''m not
sure about
> this.
>
> Thanks.
>   
Hi

I followed this guide: http://www.linux.com/article.pl?sid=06/07/17/1916214

and it worked very well for me for quite a while. However lately I''ve
seen some weird errors, such as mounting a LV on one node would report corrupted
filesystem, or filesystem not found, where it would work fine to mount the LV
from several other nodes. From what I''ve gathered, even tho no node is
mounting the same LV at the same time, problems could arise with using
"regular" lvm instead of clvm. It is to be said i did export one
raided device with AoE and then pvcreate etc to create the lvm layer ontop of
the aoe share, and maybe, just maybe, it is aoe that is making the problem, but
seems not. I''d like to get some clarification on this matter so if
someone would care to elaborate on this I would be happy :)

I have heard comments such as "that is insane" about the way aoe and
lvm is combined in the guide above..


Also, if anyone has any bright ideas on howto easily manage aoe shares
on the "san" side and then mounting each of the /dev/etherd/eX.Y
shares
in the domUs instead of creating LVs on top of one share please
elaborate on that as well. I find the numbering scheme AoE uses quite
confusing and difficult to keep track of.

For my personal use, clvm seems to induce a substantial overhead on my
nodes which i really would like to avoid.

Ingard

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> A question about this interesting topic. Is really nedded a cluster aware
> volume manager? 
No.
> I''m asking this because I''m thinking in a scenario where
> two Dom0''s access a shared iSCSI disk and the DomU migrates from
one Dom0
> to the other one. Do I need a cluster aware filesystem or volume manager
> in that iSCSI shared disk? It would be needed if in the migration process
> the DomU is executed in both Dom0 at the same time, but I''m not
sure about
> this.
It is not needed. A shared iSCSI disk (with the same name in both dom0s) 
is okay.

You may want to read this:
http://www.performancemagic.com/iscsi-xen-howto/index.html

Regards
Dominik

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tuesday 27 March 2007 07:07, Dominik Klein wrote:
> > A question about this interesting topic. Is really nedded a cluster
aware
> > volume manager?
>
> No.
>
> > I''m asking this because I''m thinking in a scenario
where
> > two Dom0''s access a shared iSCSI disk and the DomU migrates
from one Dom0
> > to the other one. Do I need a cluster aware filesystem or volume
manager
> > in that iSCSI shared disk? It would be needed if in the migration
process
> > the DomU is executed in both Dom0 at the same time, but I''m
not sure
> > about this.
>
> It is not needed. A shared iSCSI disk (with the same name in both dom0s)
> is okay.
>
This is how we have set things up. Nominally I have two storage servers 
connected with drbd, sharing lvm2 volumes through iscsitarget and multiple 
xen dom0 boxes. Each dom0 box connects to the storage box using iSCSI with 
consistent naming of the devices from udev.

Then run reiserfs, swap, etc on these partitioned devices, and live migration 
works a treat. No need for anything cluster aware so far.

Regards

Matthew
-- 
Matthew Wild                       Tel.: +44 (0)1235 445173
M.Wild@rl.ac.uk                    URL http://www.ukssdc.ac.uk/
UK Solar System Data Centre and
World Data Centre - Solar-Terrestrial Physics, Chilton
Rutherford Appleton Laboratory, Chilton, Didcot, Oxon, OX11 0QX

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>> It is not needed. A shared iSCSI disk (with the same name in both
dom0s)
>> is okay.
>>
> This is how we have set things up. Nominally I have two storage servers
> connected with drbd, sharing lvm2 volumes through iscsitarget and multiple
> xen dom0 boxes. Each dom0 box connects to the storage box using iSCSI with
> consistent naming of the devices from udev.
>
> Then run reiserfs, swap, etc on these partitioned devices, and live
> migration
> works a treat. No need for anything cluster aware so far.
Think about a live migration of a VM that is writing to the disk, could it
be possible, during the live migration process, that the VM being migrated
runs in both Dom0 at the same time? If it happens the same DomU will be
writing to the same iSCSI target and no cluster filesystem is in use.

It''s OK if the DomU doesn''t run in both nodes at the same
time, but I''m
not sure about this point in a live migration procedure. I am worried
about this because I did some testing with this and when I performed live
migration of a VM that wasn''t writing to disk, it worked fine, but
whith a
ftp filw downlad it hangs. The VM root disk was a file exported as a iSCSI
target not a LVM logical volume, maybe this was the problem?



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users