Hi all, What do you think is the best way to share data between guest domains? Would that be something like NFS? Would appreciate it if someone could give me some advice. Thanks, Rene Kogels _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Joost,
Thanks for your reply.
When you use NFS, does this imply you are using your domain0 for the
NFS-daemon or do you use a dedicated guest domain?
Thanks,
Rene
On Sat, 2006-03-18 at 20:04 +0000, Joost Roeleveld
wrote:> Hi,
>
> On Saturday 18 March 2006 19:33, Rene wrote:
> > Hi all,
> >
> > What do you think is the best way to share data between guest domains?
> > Would that be something like NFS?
>
> I have seen some mails about a XenFS-like filesystem to be in development
> allowing quick and efficient sharing of data.
> But till that comes along, something like NFS, SCP, FTP,... or anything
else
> that you''d use to share data between different physical machines.
>
> A guest domain is in most respects comparable to a physical machine, except
> that it shares the actual physical resources with all the other domains
> running on the physical machine.
>
> > Would appreciate it if someone could give me some advice.
>
> I personally use NFS to share a few filesystems between domains:
> - backup (to store all the backups to so i can write them to DVD inside
> Domain-0)
> - portage + distfiles (to avoid having to download this multiple times,
these
> are for the Gentoo package manager, portage)
>
> But any other Network Filesystem should work as well.
>
> --
> Joost
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Stephan Austermühle
2006-Mar-18 20:53 UTC
Re: [Xen-users] How to share data between guest domains
Hi Rene! Rene wrote:> What do you think is the best way to share data between guest domains? > Would that be something like NFS?I would think of GFS, OCFS, or other cluster/SAN filesystems that allow concurrent write access by many nodes at the same time. Does anybody have experience with those? Stephan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Richard Jones
2006-Mar-18 21:45 UTC
Re: [Xen-users] How to share data between guest domains
On Sat, Mar 18, 2006 at 07:33:56PM +0000, Rene wrote:> What do you think is the best way to share data between guest domains? > Would that be something like NFS? > > Would appreciate it if someone could give me some advice.We''ve set up NFS home directories. One of the domU''s is an NFS server and the others all mount /home. Works well. It was very slow until we started mounting the NFS partitions over TCP, which solved that problem. We use NIS to synchronise the userids/passwd entries. Rich. -- Richard Jones, CTO Merjis Ltd. Merjis - web marketing and technology - http://merjis.com Team Notepad - intranets and extranets for business - http://team-notepad.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Gabor HALASZ
2006-Mar-19 01:18 UTC
Re: [Xen-users] How to share data between guest domains
Stephan Austermühle wrote:> > I would think of GFS, OCFS, or other cluster/SAN filesystems that allow > concurrent write access by many nodes at the same time. Does anybody > have experience with those? >Yes, ocfs2 over gnbd works for me. -- Szerintem. Gabor HALASZ <halasz.g@freemail.hu> _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra
2006-Mar-19 02:19 UTC
Re: [Xen-users] How to share data between guest domains
On Saturday 18 March 2006 8:18 pm, Gabor HALASZ wrote:> Yes, ocfs2 over gnbd works for me.would that work over a shared image/LVM partition? i mean, using the same backing store for two domU''s would be similar to a network/SAN shared block device. i know xm is supposed to forbid it, but is there any way to override the check for some devices? -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Todd D. Esposito
2006-Mar-19 03:00 UTC
Re: [Xen-users] How to share data between guest domains
On Sat, March 18, 2006 20:19, Javier Guerra said:> would that work over a shared image/LVM partition? i mean, using the same > backing store for two domU''s would be similar to a network/SAN shared > block > device. > > i know xm is supposed to forbid it, but is there any way to override the > check > for some devices?That sounds an awfully bad idea. Most (all?) non-networked filesystems assume, correctly, that they have complete control over the filesystem data AND metadata, so they don''t need to worry about two processes updating the same inode, for example, at the same time. NFS and other networked systems DO have to worry about that. However, on that note, I wonder if you could mount the same file system, say something like /usr, into multiple domU''s READ ONLY. That could certainly save some trouble when updating applications, etc. Anyone tried it? - Todd _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra
2006-Mar-19 03:22 UTC
Re: [Xen-users] How to share data between guest domains
On Saturday 18 March 2006 10:00 pm, Todd D. Esposito wrote:> That sounds an awfully bad idea. Most (all?) non-networked filesystemssure, but here we''re talking about cluster-aware filesystems (OCFS, GFS, and others).> However, on that note, I wonder if you could mount the same file system, > say something like /usr, into multiple domU''s READ ONLY. That could > certainly save some trouble when updating applications, etc. Anyone tried > it?this could certainly be usefull, ''classical'' UNIX servers used to have most of the system mounted read-only, and just some r/w partitions for /var, /tmp, /home, and such. the r/o is sometimes (not very often) shared among several nodes in SAN clusters. obviously, for the vast majority of cases, using the same backing store for different domU''s is a bad idea and should be verified by xm. but there _are_ some cases where it''s usefull, and it would be nice if there was some way to say "Yes, i know it''s repeated. Trust me, i know what i''m doing" in the config file. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Gabor HALASZ
2006-Mar-19 11:44 UTC
Re: [Xen-users] How to share data between guest domains
Javier Guerra wrote:> On Saturday 18 March 2006 8:18 pm, Gabor HALASZ wrote: > >>Yes, ocfs2 over gnbd works for me. > > > would that work over a shared image/LVM partition?Yes, I exported an evms volume into domU and shared with the gnbd server. -- Szerintem. Gabor HALASZ <halasz.g@freemail.hu> _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Richard Jones
2006-Mar-19 11:59 UTC
Re: [Xen-users] How to share data between guest domains
On Sat, Mar 18, 2006 at 09:00:28PM -0600, Todd D. Esposito wrote:> However, on that note, I wonder if you could mount the same file system, > say something like /usr, into multiple domU''s READ ONLY. That could > certainly save some trouble when updating applications, etc. Anyone tried > it?This would only work if the disk is read-only *and* never changes while the machines are up. The reason for that is that the kernel keeps disk blocks cached in memory, and assumes that the disk won''t change randomly underneath. If they did, then it would lead to a situation where the kernel is randomly using a mix of stale (cached) and fresh blocks. In other words, if you wanted to upgrade parts of /usr, you''d need to shut down all your domUs, carry out the upgrade operation, and then boot them up again. Over here all our domUs have separate partitions. You only need max 3G for a complete Debian with plenty of packages, and how much does 3G of disk cost these days? The task then moved to automating sysadmin as far as possible -- see cfengine and apt-cron. Rich. -- Richard Jones, CTO Merjis Ltd. Merjis - web marketing and technology - http://merjis.com Team Notepad - intranets and extranets for business - http://team-notepad.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra
2006-Mar-19 13:55 UTC
Re: [Xen-users] How to share data between guest domains
On Sunday 19 March 2006 6:44 am, Gabor HALASZ wrote:> Javier Guerra wrote: > > On Saturday 18 March 2006 8:18 pm, Gabor HALASZ wrote: > >>Yes, ocfs2 over gnbd works for me. > > > > would that work over a shared image/LVM partition? > > Yes, I exported an evms volume into domU and shared with the gnbd server.I mean without gnbd; just specifying the same backing for more than one domU. And making sure it contains a cluster-aware filesystem, of course. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Todd D. Esposito
2006-Mar-19 14:26 UTC
Re: [Xen-users] How to share data between guest domains
On Sun, March 19, 2006 5:59, Richard Jones said:> On Sat, Mar 18, 2006 at 09:00:28PM -0600, Todd D. Esposito wrote: >> However, on that note, I wonder if you could mount the same file system, >> say something like /usr, into multiple domU''s READ ONLY. That could >> certainly save some trouble when updating applications, etc. Anyone >> tried >> it? > > This would only work if the disk is read-only *and* never changes > while the machines are up. The reason for that is that the kernel > keeps disk blocks cached in memory, and assumes that the disk won''t > change randomly underneath. If they did, then it would lead to a > situation where the kernel is randomly using a mix of stale (cached) > and fresh blocks. > > In other words, if you wanted to upgrade parts of /usr, you''d need to > shut down all your domUs, carry out the upgrade operation, and then > boot them up again.This is in line with my first instinct, but I''m one of those "try it anyway, it might work" kinda guys, so thanks for saving me some lab time. ;-)> Over here all our domUs have separate partitions. You only need max > 3G for a complete Debian with plenty of packages, and how much does 3G > of disk cost these days? The task then moved to automating sysadmin > as far as possible -- see cfengine and apt-cron.Sure, disk is cheap; my aim was really to reduce sysadmin time/overhead. I''ve glanced over cfgengine, and will likely go that way as my number of systems (and domU/system) grows. apt-cron won''t do it for me, as I''m on Gentoo, but there are no shortage of similar tools. Thanks for the recommendations. - Todd _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Joost Roeleveld
2006-Mar-19 15:23 UTC
Re: [Xen-users] How to share data between guest domains
On Saturday 18 March 2006 20:17, Rene Kogels wrote:> Hi Joost, > > Thanks for your reply.You''re welcome> When you use NFS, does this imply you are using your domain0 for the > NFS-daemon or do you use a dedicated guest domain?I use both, the 3 I mentioned in my previous email are exported by domain0, but I also have a guest domain sharing some files using NFS and Samba. Don''t forget to run ''portmap'' in the guest domains as well or mounting will take a long time and set it to use TCP for the connection rather then UDP, as mentioned by others.> Thanks, > Rene > > On Sat, 2006-03-18 at 20:04 +0000, Joost Roeleveld wrote: > > Hi, > > > > On Saturday 18 March 2006 19:33, Rene wrote: > > > Hi all, > > > > > > What do you think is the best way to share data between guest domains? > > > Would that be something like NFS? > > > > I have seen some mails about a XenFS-like filesystem to be in development > > allowing quick and efficient sharing of data. > > But till that comes along, something like NFS, SCP, FTP,... or anything > > else that you''d use to share data between different physical machines. > > > > A guest domain is in most respects comparable to a physical machine, > > except that it shares the actual physical resources with all the other > > domains running on the physical machine. > > > > > Would appreciate it if someone could give me some advice. > > > > I personally use NFS to share a few filesystems between domains: > > - backup (to store all the backups to so i can write them to DVD inside > > Domain-0) > > - portage + distfiles (to avoid having to download this multiple times, > > these are for the Gentoo package manager, portage) > > > > But any other Network Filesystem should work as well. > > > > -- > > Joost > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Brock Edward Palen
2006-Mar-19 19:14 UTC
Re: [Xen-users] How to share data between guest domains
Todd D. Esposito wrote:>On Sun, March 19, 2006 5:59, Richard Jones said: > > >>On Sat, Mar 18, 2006 at 09:00:28PM -0600, Todd D. Esposito wrote: >> >> >>>However, on that note, I wonder if you could mount the same file system, >>>say something like /usr, into multiple domU''s READ ONLY. That could >>>certainly save some trouble when updating applications, etc. Anyone >>>tried >>>it? >>> >>> >>This would only work if the disk is read-only *and* never changes >>while the machines are up. The reason for that is that the kernel >>keeps disk blocks cached in memory, and assumes that the disk won''t >>change randomly underneath. If they did, then it would lead to a >>situation where the kernel is randomly using a mix of stale (cached) >>and fresh blocks. >> >>In other words, if you wanted to upgrade parts of /usr, you''d need to >>shut down all your domUs, carry out the upgrade operation, and then >>boot them up again. >> >> > >This is in line with my first instinct, but I''m one of those "try it >anyway, it might work" kinda guys, so thanks for saving me some lab time. >;-) > > > >>Over here all our domUs have separate partitions. You only need max >>3G for a complete Debian with plenty of packages, and how much does 3G >>of disk cost these days? The task then moved to automating sysadmin >>as far as possible -- see cfengine and apt-cron. >> >> > >Sure, disk is cheap; my aim was really to reduce sysadmin time/overhead. >I''ve glanced over cfgengine, and will likely go that way as my number of >systems (and domU/system) grows. apt-cron won''t do it for me, as I''m on >Gentoo, but there are no shortage of similar tools. Thanks for the >recommendations. > >- Todd > >Todd, I use xen with gentoo also one thing I do and i know is documented in some gentoo doc''s is i only use one portage tree I NFS export /usr/portage read-only and /usr/portage/distfiles rw to all my dom''U''s from dom0. When i sync I sync dom0 like normal emerge --sync and the domU''s i emerge --metadata This only saves about 1G/domU but its nifty to only sync once from rsync.gentoo. I used to use a local rsync mirror Brock> >_______________________________________________ >Xen-users mailing list >Xen-users@lists.xensource.com >http://lists.xensource.com/xen-users > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Gabor HALASZ
2006-Mar-19 20:10 UTC
Re: [Xen-users] How to share data between guest domains
Javier Guerra wrote:> On Sunday 19 March 2006 6:44 am, Gabor HALASZ wrote: > >>Javier Guerra wrote: >> >>>On Saturday 18 March 2006 8:18 pm, Gabor HALASZ wrote: >>> >>>>Yes, ocfs2 over gnbd works for me. >>> >>>would that work over a shared image/LVM partition? >> >>Yes, I exported an evms volume into domU and shared with the gnbd server. > > > I mean without gnbd; just specifying the same backing for more than one domU.IMHO it isn''t possible. -- Szerintem. Gabor HALASZ <halasz.g@freemail.hu> _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2006-Mar-20 11:11 UTC
Re: [Xen-users] How to share data between guest domains
> That works for me. > > What doesn''t work is mounting that file/device READ/WRITE in one domU > to update the filesystem. For that, I have to take down *all* domUs. > Not good... > > (When I try I get a vbd: error saying "already in use".) > > (I know about caching and that I need eg. a cluster-aware filesystem > to do this.) > > I''ve spent a couple of hours hunting through various Xen source files. > There''s a lot of Python functions that are only 3-5 lines long and > which does little else than calling the next function, which makes it > very hard to figure out what''s going on :-/. > > Could one of you devel guys please let me know where I need to go to > remove this silly limitation? :-)IIRC, you can use "w!'' for ''gimme write sharing - i really mean it!'' ;-) Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Molle Bestefich
2006-Mar-20 12:45 UTC
Re: [Xen-users] How to share data between guest domains
Todd D. Esposito wrote:> However, on that note, I wonder if you could mount the same file system, > say something like /usr, into multiple domU''s READ ONLY.That works for me. What doesn''t work is mounting that file/device READ/WRITE in one domU to update the filesystem. For that, I have to take down *all* domUs. Not good... (When I try I get a vbd: error saying "already in use".) (I know about caching and that I need eg. a cluster-aware filesystem to do this.) I''ve spent a couple of hours hunting through various Xen source files. There''s a lot of Python functions that are only 3-5 lines long and which does little else than calling the next function, which makes it very hard to figure out what''s going on :-/. Could one of you devel guys please let me know where I need to go to remove this silly limitation? :-) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Yura Pismerov
2006-Mar-20 13:49 UTC
[Xen-devel] Re: [Xen-users] How to share data between guest domains
I found that using NFS for things like this makes much more sense. You can run the domU with NFS root (read-only) and map certain areas you need read/write to tmpfs by mounting them with "mount --bind" in Linux). For example, if I use NFS root and want my /etc be writable I can always write its content to a tmpfs mounted area and run "mount --bind /tmpfs/etc /etc". This also will solve problems with centralized package updates when not only /usr is being updated, but some other areas (eg. /etc, /var/lib). You want those areas be shared between domU''s as well. Molle Bestefich wrote:>Todd D. Esposito wrote: > > >>However, on that note, I wonder if you could mount the same file system, >>say something like /usr, into multiple domU''s READ ONLY. >> >> > >That works for me. > >What doesn''t work is mounting that file/device READ/WRITE in one domU >to update the filesystem. For that, I have to take down *all* domUs. >Not good... > >(When I try I get a vbd: error saying "already in use".) > >(I know about caching and that I need eg. a cluster-aware filesystem >to do this.) > >I''ve spent a couple of hours hunting through various Xen source files. > There''s a lot of Python functions that are only 3-5 lines long and >which does little else than calling the next function, which makes it >very hard to figure out what''s going on :-/. > >Could one of you devel guys please let me know where I need to go to >remove this silly limitation? :-) > >_______________________________________________ >Xen-users mailing list >Xen-users@lists.xensource.com >http://lists.xensource.com/xen-users > >-- Yuri Pismerov, System Administrator Armor Technologies (Canada) Inc. P: 905 305 1946 (x.3519) http://www.armorware.net Privacy Protection Guaranteed! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Xin Zhao
2006-Mar-20 17:24 UTC
Re: [Xen-devel] Re: [Xen-users] How to share data between guest domains
NFS combined with your solution is definitely a temporary solution. But our ongoing project "Virtual Librarian" should provide better support. VL is designed to allow mutliple VMs to share a base software environment. If a VM needs to modify a shared file, VL can do copy-on-write to create a private copy for this VM. All modifications are therefore visible only to this VM In addition, if multiple private copies are identical, VL can merge them back to a shared copy. The benefits of VL inlcude: 1. A VM can take advantage of the global disk cache and benefit from previous data accesses from other VMs. We will expect better performance. 2. VL allows finer granularity of sharing, instead of directory level sharing. 3. The shared file system is transparent to guest applications and should be easily adopted. 4. VL allows centralized software updates. These updates can take effect right after the files are updated. We will put a detailed description of VL soon, if someone is interested in that. :) Xin Yura Pismerov wrote:> I found that using NFS for things like this makes much more sense. > You can run the domU with NFS root (read-only) and map certain areas you > need read/write > to tmpfs by mounting them with "mount --bind" in Linux). For example, if > I use NFS root and want my /etc > be writable I can always write its content to a tmpfs mounted area and > run "mount --bind /tmpfs/etc /etc". > This also will solve problems with centralized package updates when not > only /usr is being updated, but some other areas (eg. /etc, /var/lib). > You want those areas be shared between domU''s as well. > > > Molle Bestefich wrote: > > >> Todd D. Esposito wrote: >> >> >> >>> However, on that note, I wonder if you could mount the same file system, >>> say something like /usr, into multiple domU''s READ ONLY. >>> >>> >>> >> That works for me. >> >> What doesn''t work is mounting that file/device READ/WRITE in one domU >> to update the filesystem. For that, I have to take down *all* domUs. >> Not good... >> >> (When I try I get a vbd: error saying "already in use".) >> >> (I know about caching and that I need eg. a cluster-aware filesystem >> to do this.) >> >> I''ve spent a couple of hours hunting through various Xen source files. >> There''s a lot of Python functions that are only 3-5 lines long and >> which does little else than calling the next function, which makes it >> very hard to figure out what''s going on :-/. >> >> Could one of you devel guys please let me know where I need to go to >> remove this silly limitation? :-) >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> >> >> > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yura Pismerov
2006-Mar-21 13:06 UTC
Re: [Xen-devel] Re: [Xen-users] How to share data between guest domains
Looks interesting. There is also LVM persistent snapshots that allow doing something similar to what you described. We did not want to use LVM due to complexity it would bring in our setup. IMHO, in setups with hundreds and thousands VMs using NFS with a few powerful NAS boxes makes much more sense. So, ideally, NFS copy-on-write is what we are looking for. I know there is an alpha release of it out there, but the problem is we need it now, not a few years from now when it reaches production quality. Xin Zhao wrote:> NFS combined with your solution is definitely a temporary solution. > But our ongoing project "Virtual Librarian" should > provide better support. VL is designed to allow mutliple VMs to share > a base software environment. If a VM needs to modify a shared file, > VL can do copy-on-write to create a private copy for this VM. All > modifications are therefore visible only to > this VM In addition, if multiple private copies are identical, VL can > merge them back to a shared copy. > > The benefits of VL inlcude: > 1. A VM can take advantage of the global disk cache and benefit from > previous data accesses from other VMs. We will expect better performance. > 2. VL allows finer granularity of sharing, instead of directory level > sharing. > 3. The shared file system is transparent to guest applications and > should be easily adopted. > 4. VL allows centralized software updates. These updates can take > effect right after the files are updated. > > We will put a detailed description of VL soon, if someone is > interested in that. :) > > Xin > > Yura Pismerov wrote: >> I found that using NFS for things like this makes much more sense. >> You can run the domU with NFS root (read-only) and map certain areas you >> need read/write >> to tmpfs by mounting them with "mount --bind" in Linux). For example, if >> I use NFS root and want my /etc >> be writable I can always write its content to a tmpfs mounted area and >> run "mount --bind /tmpfs/etc /etc". >> This also will solve problems with centralized package updates when not >> only /usr is being updated, but some other areas (eg. /etc, /var/lib). >> You want those areas be shared between domU''s as well. >> >> >> Molle Bestefich wrote: >> >> >>> Todd D. Esposito wrote: >>> >>> >>> >>>> However, on that note, I wonder if you could mount the same file >>>> system, >>>> say something like /usr, into multiple domU''s READ ONLY. >>>> >>>> >>> That works for me. >>> >>> What doesn''t work is mounting that file/device READ/WRITE in one domU >>> to update the filesystem. For that, I have to take down *all* >>> domUs. Not good... >>> >>> (When I try I get a vbd: error saying "already in use".) >>> >>> (I know about caching and that I need eg. a cluster-aware filesystem >>> to do this.) >>> >>> I''ve spent a couple of hours hunting through various Xen source files. >>> There''s a lot of Python functions that are only 3-5 lines long and >>> which does little else than calling the next function, which makes it >>> very hard to figure out what''s going on :-/. >>> >>> Could one of you devel guys please let me know where I need to go to >>> remove this silly limitation? :-) >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >>> >>> >> >> >> >-- Yuri Pismerov, System Administrator Armor Technologies (Canada) Inc. P: 905 305 1946 (x.3519) http://www.armorware.net Privacy Protection Guaranteed! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel