Xen users - Mar 2010 - Best way to store domU''s. NFS? NBD?

Hi,

I''m trying to find out what the way is to store the domU disk images,
keeping
live migration and failover in mind. One source [1] says NFS is the most
common. Another source [2] says that NFS is flaky. Then iSCSI is mentioned as
very robust, but the xen wiki makes not a single mention of that. Also, (E)NBD
is mentioned.

I want to setup a cluster where I have a machine for storing the disk images
and then a bunch of machines between which I want to freely migrate my
domU''s
back and forth. The machine storing the images itself should also run
domU''s,
if possible. This is not possible with NBD, according to [2].

What is important, is that the storage server has failover. I could do this
with drbd (distributed remote block device) and NFS, but I have no idea what
would be the best way. 

Any help is appreciated.

[1]
http://www.virtuatopia.com/index.php/Migrating_Xen_domainU_Guests_Between_Host_Systems
[2] http://lists.xensource.com/archives/html/xen-users/2007-08/msg00264.html


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com [mailto:xen-users-
> bounces@lists.xensource.com] On Behalf Of Wiebe Cazemier
> Sent: Wednesday, March 24, 2010 1:15 PM
> To: xen-users@lists.xensource.com
> Subject: [Xen-users] Best way to store domU''s. NFS? NBD?
> 
> Hi,
> 
> I''m trying to find out what the way is to store the domU disk
images,
keeping
> live migration and failover in mind. One source [1] says NFS is the
most
> common. Another source [2] says that NFS is flaky. Then iSCSI is
mentioned as
> very robust, but the xen wiki makes not a single mention of that.
Also, (E)NBD
> is mentioned.
Decide whether you want files or block devices as the backing store of
your domU''s.  There are pros and cons to each.  In my installations I
work solely with block devices, so I won''t discuss NFS further.

DRBD does a great job of providing shared, reliable block devices for
two-node Linux clusters.  It requires a good network connection but no
specialized hardware.

iSCSI is nearly ubiquitous among commercial SAN products these days.
Its main advantage is interoperability--there are many SAN vendors, many
client implementations (Linux and otherwise), and works over any network
that supports TCP/IP.  You can also use a Fibre Channel SAN or even
FCoE, AoE, or any of the lesser-known protocols.  It may make more sense
to choose a good storage vendor, using their recommended and supported
protocol, than the other way around.

GNBD is available for those who don''t wish to invest in a commercial
SAN
(as is DRBD), but I don''t believe GNDB is receiving further
development.

No matter what you choose for shared block storage, you need some kind
of logical volume management so you can easily carve up your large
physical RAID arrays into manageable pieces to store individual disk
images or filesystems.  Many SAN products include management tools that
make this easy.  Some offer handy features such as thin provisioning or
volume snapshots.  For Linux installations, Red Hat''s Clustered LVM
(CLVM) can also provide volume management independent of whatever
network storage you choose, and is simple to deploy on RHEL or CentOS
clusters.

What is "best" may well depend on your exact requirements.  Do you
need
simple failover (2 machines) or might you need to grow to 3 or more
hosts?  How much storage overall do you need today, and are you prepared
to grow this on demand?  Are you running a homogenous Linux environment
or do you need to mix in Windows or other systems?  What will you use to
backup data?  (And so forth.)

-Jeff



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wednesday 24 March 2010 19:34, Jeff Sturm wrote:
> Decide whether you want files or block devices as the backing store of
> your domU''s.  There are pros and cons to each.  In my
installations I
> work solely with block devices, so I won''t discuss NFS further.
Is storing disk images on NFS even an option? I can imagine that this gives
problems with syncs not actually being written when the driver reports it, and
similar problems. I have found out that when you do use NFS, the tap:aio:
back-end should be used, as opposed to the file: back-end.
> 
> DRBD does a great job of providing shared, reliable block devices for
> two-node Linux clusters.  It requires a good network connection but no
> specialized hardware.
> 
> iSCSI is nearly ubiquitous among commercial SAN products these days.
> Its main advantage is interoperability--there are many SAN vendors, many
> client implementations (Linux and otherwise), and works over any network
> that supports TCP/IP.  You can also use a Fibre Channel SAN or even
> FCoE, AoE, or any of the lesser-known protocols.  It may make more sense
> to choose a good storage vendor, using their recommended and supported
> protocol, than the other way around.
> 
> GNBD is available for those who don''t wish to invest in a
commercial SAN
> (as is DRBD), but I don''t believe GNDB is receiving further
development.
> 
> No matter what you choose for shared block storage, you need some kind
> of logical volume management so you can easily carve up your large
> physical RAID arrays into manageable pieces to store individual disk
> images or filesystems.  Many SAN products include management tools that
> make this easy.  Some offer handy features such as thin provisioning or
> volume snapshots.  For Linux installations, Red Hat''s Clustered
LVM
> (CLVM) can also provide volume management independent of whatever
> network storage you choose, and is simple to deploy on RHEL or CentOS
> clusters.
> 
> What is "best" may well depend on your exact requirements.  Do
you need
> simple failover (2 machines) or might you need to grow to 3 or more
> hosts?  How much storage overall do you need today, and are you prepared
> to grow this on demand?  Are you running a homogenous Linux environment
> or do you need to mix in Windows or other systems?  What will you use to
> backup data?  (And so forth.)
Let me put it this way, what are common and reliable storage solutions, using
only Linux hosts? My problem here is that there is documentation for a whole
bunch of methods, but still I don''t really know which of those methods
is
reliable and commonly used, and which are just legacy''s from the past,
and so
forth.

And, if you say iSCSI is ubiquitous, why are there 0 hits when I search for
iscsi on wiki.xensource.org?
> 
> -Jeff
- Wiebe


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com [mailto:xen-users-
> bounces@lists.xensource.com] On Behalf Of Wiebe Cazemier
> Sent: Thursday, March 25, 2010 5:31 AM
> To: xen-users@lists.xensource.com
> Subject: [Xen-users] RE: Best way to store domU''s. NFS? NBD?
> 
> On Wednesday 24 March 2010 19:34, Jeff Sturm wrote:
> > Decide whether you want files or block devices as the backing store
of
> > your domU''s.  There are pros and cons to each.  In my
installations
I
> > work solely with block devices, so I won''t discuss NFS
further.
> 
> Is storing disk images on NFS even an option? I can imagine that this
gives
> problems with syncs not actually being written when the driver reports
it, and
> similar problems. I have found out that when you do use NFS, the
tap:aio:
> back-end should be used, as opposed to the file: back-end.
I confess I don''t really know the answer to your question.  I suspect
NFS provides a "good enough" consistency model for many users.

I''ve been burned by early implementations of NFS (v2) which colors my
opinion somewhat.  Recent implementations of NFSv4 likely fix some of
the problems, but I''m really not an NFS expert, as there are enough
alternatives these days for network storage that I don''t have to be
one.

Use tap:sync: if you are paranoid.  I''ve had consistency problems with
GFS and tap:aio: over block devices.
> Let me put it this way, what are common and reliable storage
solutions, using
> only Linux hosts? My problem here is that there is documentation for a
whole
> bunch of methods, but still I don''t really know which of those
methods
is
> reliable and commonly used, and which are just legacy''s from the
past,
and so
> forth.
My opinion is but a single data point.  In my last reply I''d given you
some guidelines and general info that you could use to do your own
evaluation.

But for my money...

- High-availability block storage on Linux only--DRBD + CLVM.  Make sure
you invest in a good disk subsystem for each storage host.  No need for
15K RPM drives everywhere, but the more spindles, the better.

- SAN storage. EqualLogic for iSCSI--a bit pricey but solid, reliable,
simple to configure and great performance.  Coraid for AoE--simple,
easy, very inexpensive, also reliable.

I wouldn''t bother with the standard protocols like iSCSI unless
I''m also
buying hardware from a commercial storage vendor.  If you''re committed
to Linux only, take a long look at DRBD.  It''s actively developed and
works great.  (There are "soft" appliances for iSCSI like Openfiler.
I''ve had lukewarm results with those.  Commercial products are
generally
much more polished.)
> And, if you say iSCSI is ubiquitous, why are there 0 hits when I
search for
> iscsi on wiki.xensource.org?
Do you mean wiki.xensource.com?  The search
http://www.google.com/#hl=en&q=site%3Awiki.xensource.com+iscsi gives me
quite a few hits.

-Jeff



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello all,

I''m playing with Ebtables to to apply Firewall rules inter-VMs inside
the
same Xen Host.

Unfortunately, ebtables is not stateful and I don''t think it''s
been
developed anymore. I would like to know about other solutions you guys are
playing with.

Regards.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thursday 25 March 2010 18:42, Jeff Sturm wrote:
> My opinion is but a single data point.  In my last reply I''d given
you
> some guidelines and general info that you could use to do your own
> evaluation.
> 
> But for my money...
> 
> - High-availability block storage on Linux only--DRBD + CLVM.  Make sure
> you invest in a good disk subsystem for each storage host.  No need for
> 15K RPM drives everywhere, but the more spindles, the better.
> 
> - SAN storage. EqualLogic for iSCSI--a bit pricey but solid, reliable,
> simple to configure and great performance.  Coraid for AoE--simple,
> easy, very inexpensive, also reliable.
> 
> I wouldn''t bother with the standard protocols like iSCSI unless
I''m also
> buying hardware from a commercial storage vendor.  If you''re
committed
> to Linux only, take a long look at DRBD.  It''s actively developed
and
> works great.  (There are "soft" appliances for iSCSI like
Openfiler.
> I''ve had lukewarm results with those.  Commercial products are
generally
> much more polished.)
DRBD by itself doesn''t provide me with a storage solution. I can make a
primary
and failover storage host with it, but when I want to have my VM''s
running on
other servers, I still need some way to let those servers access it.

But, you''ve given me enough food for thought for now. Thanks.
> 
>> And, if you say iSCSI is ubiquitous, why are there 0 hits when I
> search for
>> iscsi on wiki.xensource.org?
> 
> Do you mean wiki.xensource.com?  The search
> http://www.google.com/#hl=en&q=site%3Awiki.xensource.com+iscsi gives me
> quite a few hits.
I guess I did a title only search on the wiki, with the integrated search
function. So, never mind :)
> 
> -Jeff


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thursday 25 March 2010 18:59, Antonio Pina (antonio.pina) wrote:
> Hello all,
> 
> I''m playing with Ebtables to to apply Firewall rules inter-VMs
inside the
> same Xen Host.
> 
> Unfortunately, ebtables is not stateful and I don''t think
it''s been
> developed anymore. I would like to know about other solutions you guys are
> playing with.
> 
> Regards.
You shouldn''t use "reply" and then remove the subject and
body to start a new
message, because it will remember the reference and appear in the wrong
conversation.

As for you problem, I can''t help you there... 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,

Am 24.03.2010 um 19:34 Uhr schrieb Jeff Sturm
<jeff.sturm@eprize.com>:
> > -----Original Message-----
> > From: xen-users-bounces@lists.xensource.com [mailto:xen-users-
> > bounces@lists.xensource.com] On Behalf Of Wiebe Cazemier
> > Sent: Wednesday, March 24, 2010 1:15 PM
> > To: xen-users@lists.xensource.com
> > Subject: [Xen-users] Best way to store domU''s. NFS? NBD?
[..]
> iSCSI is nearly ubiquitous among commercial SAN products these days.
and it''s also possible to create a completly open source SAN with a
linux
host. Just use lvm2 with iscsi enterprise target.

I''ve done this and i use the storage hosts also as xen hosts.
I''ve all the
nice features, live migration, storage redundancy with software raid1 inside 
the domUs, backup scripts for the whole volume group, can scale from 2 to n 
hosts, ...


-- 
greetings

eMHa


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Friday 26 March 2010 16:36, Markus Hochholdinger wrote:
>> iSCSI is nearly ubiquitous among commercial SAN products these days.
> 
> and it''s also possible to create a completly open source SAN with
a linux
> host. Just use lvm2 with iscsi enterprise target.
> 
> I''ve done this and i use the storage hosts also as xen hosts.
I''ve all the
> nice features, live migration, storage redundancy with software raid1
inside
> the domUs, backup scripts for the whole volume group, can scale from 2 to n
> hosts, ...
> 
> 
I will look into that.

A question though: "software raid1 inside the domU''s"? Why
inside the domU''s? I
would have the storage on a RAID+LVM, but not inside the domU''s.

Can you elaborate a bit more about your storage hosts? I was planning to have
two hosts: a primary host with a failover one. The failover can take over by
bringing up a virtual network interface with the same IP. Their data would be
kept up-to-date by DRBD. 

But, I haven''t figured out how I am going to let the xen hosts access
the data.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,

Am 26.03.2010 um 16:53 Uhr schrieb Wiebe Cazemier
<halfgaar@gmx.net>:
> On Friday 26 March 2010 16:36, Markus Hochholdinger wrote:
> >> iSCSI is nearly ubiquitous among commercial SAN products these
days.
> >
> > and it''s also possible to create a completly open source SAN
with a linux
> > host. Just use lvm2 with iscsi enterprise target.
> >
> > I''ve done this and i use the storage hosts also as xen hosts.
I''ve all
> > the nice features, live migration, storage redundancy with software
raid1
> > inside the domUs, backup scripts for the whole volume group, can scale
> > from 2 to n hosts, ...
> I will look into that.
> A question though: "software raid1 inside the domU''s"?
Why inside the
> domU''s? I would have the storage on a RAID+LVM, but not inside the
domU''s.
inside the domU you will never have a split brain situation for the storage! 
And you can grow the filesystem online without the need to reboot.

> Can you elaborate a bit more about your storage hosts? I was planning to
> have two hosts: a primary host with a failover one. The failover can take
> over by bringing up a virtual network interface with the same IP. Their
> data would be kept up-to-date by DRBD.
There you go with your problems: primary host, failover, bring up network... A 
lot of things that can fail! DRBD needs heartbeat, HA-Cluster, ... These are 
all problems i don''t have!

> But, I haven''t figured out how I am going to let the xen hosts
access the
> data.
Yeah, you see! You could do drbd (raid1 outside domU). Big problem is if you 
mirror the whole volume group, but mirroring each logical volume is also a 
lot of effort. So you should mirror each logical volume and access this 
directly on each host. So you''re limited to 2 hosts. Additionally you
could
export the drbd devices over iscsi, but there, again, you get into split 
brain situations...


-- 
Gruß
                                                          \|/
       eMHa                                              (o o)
------------------------------------------------------oOO--U--OOo--
 Markus Hochholdinger
 e-mail  mailto:Markus@Hochholdinger.net             .oooO
 www     http://www.hochholdinger.net                (   )   Oooo.
------------------------------------------------------\ (----(   )-
Ich will die Welt verändern,                           \_)    ) /
aber Gott gibt mir den Quelltext nicht!                      (_/


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users