Xen users - May 2008 - What''s recommended method for serving data to/among DomU''s?

I''m looking for comments/opinions on the pros & cons of various
ways
of serving data to DomU''s ...

Given a number of DomU VM''s on a given box, data sotrage can take
several forms:

 (1) Data served via NFS from Dom0 to DomU
 (2) Data served from a DomU-dedicated NAS, e.g., via NFS, OpenFiler, etc
 (3) Data stored locally in each DomU

The options above vary, at least, in security, performance, and backup strategy.

I''m interested in what folks here recommend.

Thanks!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I''m looking for comments/opinions on the pros & cons of
various ways
> of serving data to DomU''s ...
>
> Given a number of DomU VM''s on a given box, data sotrage can take
several
> forms:
>
>  (1) Data served via NFS from Dom0 to DomU
>  (2) Data served from a DomU-dedicated NAS, e.g., via NFS, OpenFiler, etc
>  (3) Data stored locally in each DomU
>
> The options above vary, at least, in security, performance, and backup
> strategy.
You could also consider network-based block devices, e.g. NBD or iSCSI to 
provide guest storage.  With these, you can use some kind of network storage 
server which may also be able to do block-based snapshots, backups, etc.  
They can also be used to facilitate live migration of VMs by making the VM 
storage accessible from multiple locations on your network.

Scripts are provided with Xen for setting up ENBD connections automatically.  
You can simply alter a domain''s config file to give details of the ENBD
connection, then dom0 will connect to the ENBD server automatically when the 
domain is started.  If the domain is migrated, the dom0 on the destination 
host will automatically connect to the ENBD server.  Through this technique, 
the use of networked storage is completely transparent to the guest, which 
just sees a normal Xen virtual disk.

My Ubuntu system includes scripts called block-enbd and block-nbd, presumably 
for different variants of the NBD server.

Something similar should be possible with iSCSI in order to make that 
transparent to the guest.  I''ve seen some patches posted on the mailing
list
to add the ability to do this automatically; they''re not merged
upstream,
though.  An alternative would be to get guests to boot explicitly from an 
iSCSI root device; this isn''t transparent to them but does still allow
some
of the same benefits.

Note that for any network based storage, you''ll need to secure the
storage
server from abuse (e.g. domU''s accessing each others storage, snooping 
network traffic, spoofing, etc).

Cheers,
Mark

-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>  You could also consider network-based block devices, e.g. NBD or iSCSI to
>  provide guest storage.
(big snip)

i''d admittedly neither considered this approach, nor even heard of
ENBD! so, something new/interesting ...

reading here, http://www.it.uc3m.es/ptb/nbd/, iiuc, this lends itself
to implementation on a separate box, no?

or, are you suggesting deploying an NBD server in/as a DomU?

thanks for the idea(s)!  your well written comment does a nice job of
intro as to capability; understanding differences wrt
NFS/OpenFiler/other ntwk storage requires some add''l reading on my
part ...

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> >  You could also consider network-based block devices, e.g. NBD or
iSCSI
> > to provide guest storage.
>
> (big snip)
>
> i''d admittedly neither considered this approach, nor even heard of
> ENBD! so, something new/interesting ...
>
> reading here, http://www.it.uc3m.es/ptb/nbd/, iiuc, this lends itself
> to implementation on a separate box, no?
NBD is, IIRC, not safe to "loop back".  So it''s not safe to
connect to and
mount a NBD device which is being served out by the local machine.  This is 
to avoid deadlocks.  I assume this is what you''re referring to?

Really the restriction should be phrased as "it''s not safe to
connect to and
mount a NBD device which is being served out by the local *OPERATING 
SYSTEM*".  So as long as the NBD server and client are in separate domains,
it''s safe to put them on the same physical host.
> or, are you suggesting deploying an NBD server in/as a DomU?
Well, the NBD script shipped by Xen allows you to have an NBD server wherever 
you want.  It could be a domU on the same machine, or it could be a separate 
host.  The point of the script is that it sets up an NBD device *in dom0* 
that connects to the NBD server.  It then re-exports that as a standard Xen 
VBD to the guest.  

The guest sees a normal disk and has no idea that it''s actually network
transparent.  You can shut the guest down, move the disk to an NBD server on 
another machine, then boot the guest again without making any changes to the 
guest itself.

In actual fact, I suspect it is actually safe to export a block device via NBD 
in dom0, then map that NBD device in dom0 and export it to a domU (at least 
for a PV domU.  For HVM I might need a bit more caffeine in the system to 
decide whether it''s a good idea but I suspect might be OK).  This
should be
safe because dom0 wouldn''t actually mounting the device, just passing
it
through.  Doing this would be rather pointless, unless you wanted to 
occasionally migrate the guests off the original machine but leave the 
storage there.  I don''t imagine it''d be good for performance
either, I''m just
making you aware of what could be possible (even if not recommended!).
> thanks for the idea(s)!  your well written comment does a nice job of
> intro as to capability; understanding differences wrt
> NFS/OpenFiler/other ntwk storage requires some add''l reading on my
> part ...
At this point it''s probably worth asking what exactly you''re
hoping to achieve
so I can direct my advice a bit better.  If you want some kind of shared 
filesystem between domains that multiple domains can access concurrently, you 
need either a network filesystem (NFS or CIFS) or a cluster filesystem (GFS 
or OCFS2).  If you want a way of centralising block storage on a server 
there, you want iSCSI or *NBD.  And so on :-)  There are lots of options here 
but if you tell us a bit more we might be able to suggest some further 
options.

Cheers,
Mark

-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>  If you want some kind of shared
>  filesystem between domains that multiple domains can access concurrently,
you
>  need either a network filesystem (NFS or CIFS) or a cluster filesystem
(GFS
>  or OCFS2).
I _just_ read this

  
http://rpmfind.net/linux/RPM/opensuse/oss-factory/x86_64/nbd-2.9.9-32.x86_64.html
	"... nbd can be used to have a filesystem stored on another machine. It
does
	provide a block device, not a file system; so unless you put a
	clustering filesystem on top of it, you can''t access it simultaneously
	from more than one client. Use NFS or a real cluster FS (such as
	ocfs2) if you want to do this. ..."

which I believe says the same.

My goal is to abstract DomU instances from their embodied apps'' data
-- e.g., bind zones, apache web files, etc., and to centralize that
data so as to be easy to backup.

With your explanations (thanks!), and a bit of reading, I think
OpenFiler in a DomU, booted early is my solution -- with subsequent
VM''s reading their apps'' data/conf/etc from
OpenFiler-published NFS
partitions gets me ''there'' ...

Bottom line, I do need -- or at least want -- simultaneous access.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> which I believe says the same.
>
> My goal is to abstract DomU instances from their embodied apps''
data
> -- e.g., bind zones, apache web files, etc., and to centralize that
> data so as to be easy to backup.
I see.
> With your explanations (thanks!), and a bit of reading, I think
> OpenFiler in a DomU, booted early is my solution -- with subsequent
> VM''s reading their apps'' data/conf/etc from
OpenFiler-published NFS
> partitions gets me ''there'' ...
>
> Bottom line, I do need -- or at least want -- simultaneous access.
OK.

nb. block level devices such as NBD, iSCSI and friends are fine for a "read
many, write exclusive" setup - e.g. multiple domains mounting a device and 
reading from it, but they must all unmount in order for a single writer to 
modify the contents.  This works without any fuss, using a normal non-cluster 
filesystem.  For some scenarios, this suffices.

Otherwise, good luck with your setup.  IIRC, OpenFiler offer Xen-enabled 
appliance downloads so that should serve your purposes.

Cheers,
Mark

-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users