Xen users - Aug 2010 - Idea for future xen development: PV file system

Good day.

Few days I have discussion with colleagues about file-based space
provisioning for xen VM''s. Obvious solution is ''pure
NFS'' VM (with root
file system on NFS server). But NFS have two serious disadvantages:
1) NFS required network level. Even network is fast, you need to send
and receive packets for every file operation. That''s create a huge
difference in performance between local FS on network block device and
NFS.
2) NFS is not very secure. Yes, there is  NFS4 with kerberos, but it
changed model of user rights, so NFS-server administrator must be aware
about every user of every VM. It''s opposite to idea of ''here
your
directory do you want to do in NFS3''. 
(I can say third: you need network in VM to works with NFS).

So, here two main problems: network overhead and bad security. ... But
if we can  solve them with nice and fast Xen mechanisms of IDC (inter
domain communications)?

All we need is create a daemon for dom0 to export directories from
''own''
local filesystem (it can resides on local disk, iscsi or be gfs, any
kind of hyper-mega-cluster-fs) and simple PV driver for domU translating
requests from domU to daemon in dom0. It will be like xen block device:
minimalistic and simple. It will offload all real work to well-tested
product filesystems in dom0 (it can be even not a dom0, but any other
stubdomain).

All need do a ''xen pvfs'' driver is translate every request to
fs via
shared mem, xenstore and xen-events to pvfs daemon.

Work of daemon in dom0 (or stubdomain) will be slightly large. It will
need to:
1) check access rights (may be in form like NFS with domID instead
IP''s)
2) translate requests to underlying filesystem.
3) translate f/s replies (include errors and so on).

I see forth job: maintain disk quote for every dom (because access to
filesystem will occur (may be) from root, and root will be differ
between domUs, we need do some more checking at daemon level.

Now about advantages of this system:

1) We will gain file-level access. That''s means easy access to
filesystem of domU without mount/umount process.
2) We will able to use single volume for few virtual machines,
increasing server consolidation.
3) Currently VM is have certain troubles with sudden block device size
changes. We can resize filesystems, but even LVM not ready for PV size
changes. And, if we talking about space reduction, every FS do it
slowly, so it can not be normal process like xenballooning. Using single
FS we will able to use single space for every VM (and quotes will helps
us to keep every VM within limits).
4) Any deduplication capabilities will work more efficiently with file
deduplication instead block-level deduplication.
5) ... and we can start thinging about openvz-like system with container
templates (where duplicated files on filesystem are simply links to
original).


Now about sad thing. I''d like to say ''I will write
it'', but my
programming level is far beyond required, so I simply asking someone to
help do it. Or, at least, say opinion about this feature.


---
wBR, George.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Aug 31, 2010 at 03:35:43PM +0400, George Shuklin
wrote:
> Good day.
> 
> Few days I have discussion with colleagues about file-based space
> provisioning for xen VM''s. Obvious solution is ''pure
NFS'' VM (with root
> file system on NFS server). But NFS have two serious disadvantages:
> 1) NFS required network level. Even network is fast, you need to send
> and receive packets for every file operation. That''s create a huge
> difference in performance between local FS on network block device and
> NFS.
> 2) NFS is not very secure. Yes, there is  NFS4 with kerberos, but it
> changed model of user rights, so NFS-server administrator must be aware
> about every user of every VM. It''s opposite to idea of
''here your
> directory do you want to do in NFS3''. 
> (I can say third: you need network in VM to works with NFS).
> 
> So, here two main problems: network overhead and bad security. ... But
> if we can  solve them with nice and fast Xen mechanisms of IDC (inter
> domain communications)?
> 
> All we need is create a daemon for dom0 to export directories from
''own''
> local filesystem (it can resides on local disk, iscsi or be gfs, any
> kind of hyper-mega-cluster-fs) and simple PV driver for domU translating
> requests from domU to daemon in dom0. It will be like xen block device:
> minimalistic and simple. It will offload all real work to well-tested
> product filesystems in dom0 (it can be even not a dom0, but any other
> stubdomain).
> 
> All need do a ''xen pvfs'' driver is translate every
request to fs via
> shared mem, xenstore and xen-events to pvfs daemon.
> 
> Work of daemon in dom0 (or stubdomain) will be slightly large. It will
> need to:
> 1) check access rights (may be in form like NFS with domID instead
IP''s)
> 2) translate requests to underlying filesystem.
> 3) translate f/s replies (include errors and so on).
> 
> I see forth job: maintain disk quote for every dom (because access to
> filesystem will occur (may be) from root, and root will be differ
> between domUs, we need do some more checking at daemon level.
> 
> Now about advantages of this system:
> 
> 1) We will gain file-level access. That''s means easy access to
> filesystem of domU without mount/umount process.
> 2) We will able to use single volume for few virtual machines,
> increasing server consolidation.
> 3) Currently VM is have certain troubles with sudden block device size
> changes. We can resize filesystems, but even LVM not ready for PV size
> changes. And, if we talking about space reduction, every FS do it
> slowly, so it can not be normal process like xenballooning. Using single
> FS we will able to use single space for every VM (and quotes will helps
> us to keep every VM within limits).
> 4) Any deduplication capabilities will work more efficiently with file
> deduplication instead block-level deduplication.
> 5) ... and we can start thinging about openvz-like system with container
> templates (where duplicated files on filesystem are simply links to
> original).
> 
> 
> Now about sad thing. I''d like to say ''I will write
it'', but my
> programming level is far beyond required, so I simply asking someone to
> help do it. Or, at least, say opinion about this feature.
> 
Did you take a look at VirtFS ?

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > Now about sad thing. I''d like to say ''I will write
it'', but my
> > programming level is far beyond required, so I simply asking someone
to
> > help do it. Or, at least, say opinion about this feature.
> > 
> Did you take a look at VirtFS ?
May be you talking about something else, but this:
http://www.prongs.org/virtfs/ is not I''m talking about....


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Aug 31, 2010 at 6:35 AM, George Shuklin
<george.shuklin@gmail.com> wrote:
> All we need is create a daemon for dom0 to export directories from
''own''
> local filesystem (it can resides on local disk, iscsi or be gfs, any
> kind of hyper-mega-cluster-fs) and simple PV driver for domU translating
> requests from domU to daemon in dom0
in KVM there are periodic mentions of 9p/virtio

http://www.slideshare.net/ericvh/9p-on-kvm

the code has evolved a little since then, and i think it''s a fairly
common addition to qemu.  i''m sure it would be easier to port to Xen
than to develop a new protocol from scratch.

-- 
Javier

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel