Hi, list I''ve been looking at solutions to copy UML''s CoW disks for Xen, but based on the archives and google results, this is pretty much an active work-in-progress for Xen. I''ve looked at CowNFS and it seems promising, but I''m a bit worried of relying on a machine that will run it. If and when the daemon conks out, everything relying on it will die as well (can''t get the clients to reestablish a connection with the server). What I''ve been looking at lately is using NFS Root + UnionFS. Having NFS root means I can use a dedicated machine to act as an NFS server and tweak it to my heart''s content, or maybe even use a NetApp for it. The only challenge this poses is getting the guest OSs to actually do the overlaying by mounting a copy of their work space (read/write) over the exported NFS root (read-only). This way, each domain can happily write to their own "disk" without having to give them their own independent disks (which wastes space). Has anyone gotten something similar to work? Any ideas / tips / comments / suggestions in doing so? I''m tweaking my xen RPM spec file to support unionfs and right now just looking at the boot up process of getting the domain to mount something else on top of the exported root file system (either a file-backed VBD or yet another NFS export). gino ledesma P.S. If there''s anything waaaay simpler than this, I''d really like to know. :-) I''m having fun playing with xen but will need to see how this setup scales. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Gino LV. Ledesma wrote:>Hi, list > >Has anyone gotten something similar to work? Any ideas / tips / >comments / suggestions in doing so? I''m tweaking my xen RPM spec file >to support unionfs and right now just looking at the boot up process >of getting the domain to mount something else on top of the exported >root file system (either a file-backed VBD or yet another NFS export). > >This is actually a large part of the paper for this presentation at OLS: http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117 There''s a number of approaches to solving this problem. unionfs would be ideal but it''s a bit unstable. Another approach is to keep certain directories read only (like /usr, /bin, /lib, /sbin, etc.) and others read write (/etc, /var/, etc.). This will get you pretty far. Regards, Anthony Liguori>gino ledesma > >P.S. If there''s anything waaaay simpler than this, I''d really like to >know. :-) I''m having fun playing with xen but will need to see how >this setup scales. > >_______________________________________________ >Xen-users mailing list >Xen-users@lists.xensource.com >http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 5/25/05, Anthony Liguori <aliguori@us.ibm.com> wrote:> Gino LV. Ledesma wrote: > > >Hi, list > > > >Has anyone gotten something similar to work? Any ideas / tips / > >comments / suggestions in doing so? I''m tweaking my xen RPM spec file > >to support unionfs and right now just looking at the boot up process > >of getting the domain to mount something else on top of the exported > >root file system (either a file-backed VBD or yet another NFS export). > > > > > This is actually a large part of the paper for this presentation at OLS: > > http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117 > > There''s a number of approaches to solving this problem. unionfs would > be ideal but it''s a bit unstable. Another approach is to keep certain > directories read only (like /usr, /bin, /lib, /sbin, etc.) and others > read write (/etc, /var/, etc.). This will get you pretty far.Thanks for the reply. I guess this is one of the most oftens suggested approaches. The two goals that I''d like to meet are: 1. Storage flexibility (resize as necessary) -- LVM looks good to go, though in our setup where we can use NetApps, NFS would be another approach. 2. Ease of administration / maintenance -- Some of the issues I''m wary about when going unionfs is that if we make changes to the underlying filesystem (e.g. OS upgrade) and there are changes on the overlay, conflicts might occur. I''ll look to doing both -- I''d like to keep things simpler, as there''d be less chances of breaking things. Thanks again. gino ledesma _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I''ve managed to come up with something pretty neat -- a shared base image with unionfs providing an overlay for a "delta" filesystem that allows each host to customize the image to their heart''s content while storing only the changes (deltas). I *love* how xen and open source work. :-) xen config exports to the domUs a file-backed master.img which contains a base root filesystem as /dev/hda1. Then each domU gets their own LVM-backed logical volumes for root (/dev/hdb1) and swap (/dev/hdc1). This allows me to grow/shrink partitions as necessary. And because these partitions only contain changes, they don''t use as much data. I customized the initrd process to union mount the base root filesystem with the "delta" filesystem and overlay that as root. This works pretty well, so far -- I''m just trying to make this work with NFS-exported file systems for maximum flexibility. Anyway, this may not really be a xen issue, but I thought I''d ask. I was trying to get several domU''s to share a base LVM-backed filesystem, but when I fire up a second domain using the same fs, it says that the vbd is already in use. I take it isn''t possible to have multiple mounts of an LVM-backed volume? That''s one reason why I had to use a file-backed root fs above. Also... As I''m not too sure about this, is it *safe* for a file-backed root fs to be shared across multiple domUs? I have the initrd mount it as read-only BUT the xen config has to make it writable. Making it read only in the config causes the mount process to spit an error (EXT3 INFO requires write access ...). -gino On 5/25/05, Gino LV. Ledesma <gledesma@gmail.com> wrote:> On 5/25/05, Anthony Liguori <aliguori@us.ibm.com> wrote: > > Gino LV. Ledesma wrote: > > > > >Hi, list > > > > > >Has anyone gotten something similar to work? Any ideas / tips / > > >comments / suggestions in doing so? I''m tweaking my xen RPM spec file > > >to support unionfs and right now just looking at the boot up process > > >of getting the domain to mount something else on top of the exported > > >root file system (either a file-backed VBD or yet another NFS export). > > > > > > > > This is actually a large part of the paper for this presentation at OLS: > > > > http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117 > > > > There''s a number of approaches to solving this problem. unionfs would > > be ideal but it''s a bit unstable. Another approach is to keep certain > > directories read only (like /usr, /bin, /lib, /sbin, etc.) and others > > read write (/etc, /var/, etc.). This will get you pretty far. > > Thanks for the reply. I guess this is one of the most oftens suggested > approaches. The two goals that I''d like to meet are: > 1. Storage flexibility (resize as necessary) -- LVM looks good to go, > though in our setup where we can use NetApps, NFS would be another > approach. > 2. Ease of administration / maintenance -- Some of the issues I''m wary > about when going unionfs is that if we make changes to the underlying > filesystem (e.g. OS upgrade) and there are changes on the overlay, > conflicts might occur. > > I''ll look to doing both -- I''d like to keep things simpler, as there''d > be less chances of breaking things. > > Thanks again. > > gino ledesma >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Anyway, this may not really be a xen issue, but I thought I''d ask. I > was trying to get several domU''s to share a base LVM-backed > filesystem, but when I fire up a second domain using the same fs, it > says that the vbd is already in use. I take it isn''t possible to have > multiple mounts of an LVM-backed volume? That''s one reason why I had > to use a file-backed root fs above.The Xen tools won''t let you have multiple mounts to a filesystem unless they''re all read only. This is just a "safety catch" to avoid people shooting themselves in the foot. If you''re Really Sure you want writeable sharing, stick "w!" as the permission in the config file.> Also... As I''m not too sure about this, is it *safe* for a file-backed > root fs to be shared across multiple domUs?Not if they''re mounting it writeable...> I have the initrd mount it > as read-only BUT the xen config has to make it writable. Making it > read only in the config causes the mount process to spit an error > (EXT3 INFO requires write access ...).If you mount it read only in the guest, it''s probably OK. It''s a bit odd that exporting it read only doesn''t work though - we export /usr read only and then mount it read only in the guest, which seems to work fine. HTH, Mark> -gino > > On 5/25/05, Gino LV. Ledesma <gledesma@gmail.com> wrote: > > On 5/25/05, Anthony Liguori <aliguori@us.ibm.com> wrote: > > > Gino LV. Ledesma wrote: > > > >Hi, list > > > > > > > >Has anyone gotten something similar to work? Any ideas / tips / > > > >comments / suggestions in doing so? I''m tweaking my xen RPM spec file > > > >to support unionfs and right now just looking at the boot up process > > > >of getting the domain to mount something else on top of the exported > > > >root file system (either a file-backed VBD or yet another NFS export). > > > > > > This is actually a large part of the paper for this presentation at > > > OLS: > > > > > > http://www.linuxsymposium.org/2005/view_abstract.php?content_key=117 > > > > > > There''s a number of approaches to solving this problem. unionfs would > > > be ideal but it''s a bit unstable. Another approach is to keep certain > > > directories read only (like /usr, /bin, /lib, /sbin, etc.) and others > > > read write (/etc, /var/, etc.). This will get you pretty far. > > > > Thanks for the reply. I guess this is one of the most oftens suggested > > approaches. The two goals that I''d like to meet are: > > 1. Storage flexibility (resize as necessary) -- LVM looks good to go, > > though in our setup where we can use NetApps, NFS would be another > > approach. > > 2. Ease of administration / maintenance -- Some of the issues I''m wary > > about when going unionfs is that if we make changes to the underlying > > filesystem (e.g. OS upgrade) and there are changes on the overlay, > > conflicts might occur. > > > > I''ll look to doing both -- I''d like to keep things simpler, as there''d > > be less chances of breaking things. > > > > Thanks again. > > > > gino ledesma > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Gino, Mind if I ask a couple questions about your UnionFS setup?> I''ve managed to come up with something pretty neat -- a shared base > image with unionfs providing an overlay for a "delta" filesystem that > allows each host to customize the image to their heart''s content while > storing only the changes (deltas).Google turns up a couple projects using that name. Are you using the one from Stony Brook (http://www.fsl.cs.sunysb.edu/project-unionfs.html), or a different one?> I customized the initrd process to union mount the baseroot filesystem > with the "delta" filesystem and overlay that as root.Are there any pitfalls to watch out for in creating this initrd, or is it pretty straightforward? Thanks, mukesh _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 6/9/05, mukesh agrawal <lists.xensource.com@mukesh.agrawals.org> wrote:> Google turns up a couple projects using that name. Are you using the one > from Stony Brook (http://www.fsl.cs.sunysb.edu/project-unionfs.html), or a > different one?Yes, that''s the one.> > I customized the initrd process to union mount the baseroot filesystem > > with the "delta" filesystem and overlay that as root. > > Are there any pitfalls to watch out for in creating this initrd, or is it > pretty straightforward?The initrd itself is pretty straightforward, especially if you''re using a file-/physical-/LVM-backed block device. Using NFS gets a little bit tricky because of the mount procedure, but nothing too difficult to overcome. The only issues I''m running into are kernel stack traces during certain I/O-heavy operations. This has more to do with unionfs I suppose, rather than the xen kernel itself. Another one I''ve looked at recently is mini_fo <http://www.denx.de/twiki/bin/view/Know/MiniFOHome>, though this doesn''t support overlaying an NFS exported volume yet, and it has some caching issues I''m looking into. - gino _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks for your replies. I''ve used it to start a Wiki page with information on the various possibilities for copy-on-write functionality. See http://wiki.xensource.com/xenwiki/COWHowTo. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users