Im curious as to what shared filesystems people have been using for domU''s I know you can use NFS but i would hardly call that a robust shared filesystem, I come from HPC ware we like performacnce more than reliablity (PVFS2) What shared filesystems have you used and how have you liked them, or how much work was it to use them as the / drive on domU''s, (intersted in migration) Luster, GFS, GPFS, etc? __________________________________________________ www.palen.serveftp.net Center for Advanced Computing http://cac.engin.umich.edu brockp@umich.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Brock Palen wrote:> Im curious as to what shared filesystems people have been using for > domU''s I know you can use NFS but i would hardly call that a robust > shared filesystem, I come from HPC ware we like performacnce more than > reliablity (PVFS2) What shared filesystems have you used and how have > you liked them, or how much work was it to use them as the / drive on > domU''s, (intersted in migration) Luster, GFS, GPFS, etc?I use GFS on changeset 9029, but it craps out all of the time. I get a lot of soft lockups in the GFS and CMAN drivers, unsynchrnoized GFS partitions, etc. I''m hoping that all of the bugs fixed in 3.0.2 will make things more stable. (If not, it''s bug report time.) Even though I would like to use it as the root partition for all of the VMs, I don''t. I also use NFS on a Xen 2 setup. That one works fine, but the performance is awful. -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello Brock I did a lot of case studies with clustered filesystems 1,5 years ago and I found out then, that the only usable and free clustered filesystem that exists for linux is ocfs2. All the rest isn''t cost free or s... I''ve you have the monney, go for veritas. bye Philipp ==============================================Philipp Jäggi SNCT Sandweiler bp 23 L-5230 Sandweiler "Christopher G. Stach II" <cgs@ldsys.net> Sent by: xen-users-bounces@lists.xensource.com 04/09/2006 12:34 AM To Brock Palen <brockp@umich.edu> cc Xen-users@lists.xensource.com Subject Re: [Xen-users] cluster filesystems Brock Palen wrote:> Im curious as to what shared filesystems people have been using for > domU''s I know you can use NFS but i would hardly call that a robust > shared filesystem, I come from HPC ware we like performacnce more than > reliablity (PVFS2) What shared filesystems have you used and how have > you liked them, or how much work was it to use them as the / drive on > domU''s, (intersted in migration) Luster, GFS, GPFS, etc?I use GFS on changeset 9029, but it craps out all of the time. I get a lot of soft lockups in the GFS and CMAN drivers, unsynchrnoized GFS partitions, etc. I''m hoping that all of the bugs fixed in 3.0.2 will make things more stable. (If not, it''s bug report time.) Even though I would like to use it as the root partition for all of the VMs, I don''t. I also use NFS on a Xen 2 setup. That one works fine, but the performance is awful. -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Philipp Jäggi wrote:> > Hello Brock > > I did a lot of case studies with clustered filesystems 1,5 years ago > and I found out then, that the only usable and free clustered > filesystem that exists for linux is ocfs2. All the rest isn''t cost > free or s...The ocfs2 website says that ocfs2 was pushed into the main line 2.6.16 did this happen? and if so has anyone tried kernel.org ocfs2 with xen? This would make my upgrade to 3 much faster :) Brock> > I''ve you have the monney, go for veritas. > > > bye Philipp > > ==============================================> Philipp Jäggi > SNCT Sandweiler > bp 23 > L-5230 Sandweiler > > > > > > *"Christopher G. Stach II" <cgs@ldsys.net>* > Sent by: xen-users-bounces@lists.xensource.com > > 04/09/2006 12:34 AM > > > To > Brock Palen <brockp@umich.edu> > cc > Xen-users@lists.xensource.com > Subject > Re: [Xen-users] cluster filesystems > > > > > > > > > > Brock Palen wrote: > > Im curious as to what shared filesystems people have been using for > > domU''s I know you can use NFS but i would hardly call that a robust > > shared filesystem, I come from HPC ware we like performacnce more than > > reliablity (PVFS2) What shared filesystems have you used and how have > > you liked them, or how much work was it to use them as the / drive on > > domU''s, (intersted in migration) Luster, GFS, GPFS, etc? > > I use GFS on changeset 9029, but it craps out all of the time. I get a > lot of soft lockups in the GFS and CMAN drivers, unsynchrnoized GFS > partitions, etc. I''m hoping that all of the bugs fixed in 3.0.2 will > make things more stable. (If not, it''s bug report time.) Even though I > would like to use it as the root partition for all of the VMs, I don''t. > > I also use NFS on a Xen 2 setup. That one works fine, but the > performance is awful. > > -- > Christopher G. Stach II > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Quoting Brock Edward Palen <brockp@umich.edu>:> The ocfs2 website says that ocfs2 was pushed into the main line 2.6.16 > did this happen? and if so has anyone tried kernel.org ocfs2 with > xen? This would make my upgrade to 3 much faster :) >I''m running Xen 3.0.2 using the 2.6.16 kernel on Ubuntu Dapper (the upcoming release). Using a shared ocfs2 volume in multiple domU''s seem to work fine, although I have not extensively tested (performance?) it yet. Xen normally refuses to start a domU with a volume that is already in use (mounted) elsewhere, but you can force it with an exclamation point. Example: disk = [''phy:/dev/vg1/jetU,hda1,w'', ''phy:/dev/vg1/shared,hdb1,w!''] ^^^^^ One last thing, I did compile Xen Xen/Linux from source; I don''t know if the binary distribution includes ocfs2 support. Regards, Peter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Peter Fokkinga wrote:> Quoting Brock Edward Palen <brockp@umich.edu>: >> The ocfs2 website says that ocfs2 was pushed into the main line 2.6.16 >> did this happen? and if so has anyone tried kernel.org ocfs2 with >> xen? This would make my upgrade to 3 much faster :) >> > > I''m running Xen 3.0.2 using the 2.6.16 kernel on Ubuntu Dapper > (the upcoming release). Using a shared ocfs2 volume in multiple > domU''s seem to work fine, although I have not extensively tested > (performance?) it yet.Ive just been reading on ocfs2 and a lwn article http://lwn.net/Articles/137278/ and makes the comment that its ment to be used: "s meant to operate with a disk which is, itself, shared across the cluster (perhaps via some sort of storage-area network or multipath scheme)." So this requires some sort of NAS unit? or does it work like PVFS2 ware each machine has its own local disk holding only a portion of the data? Like a global raid ware the hosts are the block devices and not the disks directly?> > Xen normally refuses to start a domU with a volume that is already > in use (mounted) elsewhere, but you can force it with an exclamation > point. Example: > disk = [''phy:/dev/vg1/jetU,hda1,w'', ''phy:/dev/vg1/shared,hdb1,w!''] > ^^^^^Ex thankyou> > One last thing, I did compile Xen Xen/Linux from source; I don''t > know if the binary distribution includes ocfs2 support.No problem use gentoo right now, so im doing that anyway.... :-)> > Regards, Peter > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Quoting Brock Edward Palen <brockp@umich.edu>:> Ive just been reading on ocfs2 and a lwn article > http://lwn.net/Articles/137278/ > and makes the comment that its ment to be used: "s meant to operate > with a disk which is, itself, shared across the cluster > (perhaps via some sort of storage-area network or multipath scheme)." > So this requires some sort of NAS unit? or does it work like PVFS2 > ware each machine has its own local disk holding only a portion of > the data? >If you want to share a disk using ocfs2 to multiple _physical_ servers you will probably need a SAN (expensive...) I once read about sharing a disk between hosts using firewire, but that''s really a hobby solution ;-) In Xen however it''s really simple, you just export a volume (whether it''s a file, a partition or a LVM volume) from dom0 to multiple domU''s. It''s really not much different than exporting an ext3 volume. Well, you do need a view ocfs2 processes in each domU that talk to each other (tcp, on port 7777) but you definitely do not need a SAN. Peter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Peter Fokkinga wrote:> Quoting Brock Edward Palen <brockp@umich.edu>: >> Ive just been reading on ocfs2 and a lwn article >> http://lwn.net/Articles/137278/ >> and makes the comment that its ment to be used: "s meant to operate >> with a disk which is, itself, shared across the cluster >> (perhaps via some sort of storage-area network or multipath scheme)." >> So this requires some sort of NAS unit? or does it work like PVFS2 >> ware each machine has its own local disk holding only a portion of >> the data? >> > > If you want to share a disk using ocfs2 to multiple _physical_ > servers you will probably need a SAN (expensive...) I once > read about sharing a disk between hosts using firewire, but > that''s really a hobby solution ;-) > > In Xen however it''s really simple, you just export a volume > (whether it''s a file, a partition or a LVM volume) from dom0 > to multiple domU''s. It''s really not much different than exporting > an ext3 volume. Well, you do need a view ocfs2 processes in each > domU that talk to each other (tcp, on port 7777) but you > definitely do not need a SAN.Peter, this may be a stupid question but what do you mean by "In Xen it''s really simple" as opposed to the SAN approach? OCFS2 is "just" a filesystem, it doesn''t seem to be a way of distributing blockdevices via ethernet like NFS does. So what is the point in using OCFS2 in conjunction with Xen without a SAN or iSCSI or multipath infrastructure? If I understood Brock correctly he is looking for a robust and probably high-performance (and cheap) alternative to NFS (I am too...). Thanks ahead for some "enlightening" thoughts. Cheers, Andrej _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Andrej Radonic wrote:> Peter, > this may be a stupid question but what do you mean by "In Xen it''s > really simple" as opposed to the SAN approach? > OCFS2 is "just" a filesystem, it doesn''t seem to be a way of > distributing blockdevices via ethernet like NFS does. > So what is the point in using OCFS2 in conjunction with Xen without a > SAN or iSCSI or multipath infrastructure? > If I understood Brock correctly he is looking for a robust and probably > high-performance (and cheap) alternative to NFS (I am too...). > Thanks ahead for some "enlightening" thoughts.The difference is between sharing between physical machines over a network vs. sharing between virtual machines in a single physical machine. -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
At 01:25 PM 4/10/2006, you wrote:>OCFS2 is "just" a filesystem, it doesn''t seem to be a way of >distributing blockdevices via ethernet like NFS does. >So what is the point in using OCFS2 in conjunction with Xen without >a SAN or iSCSI or multipath infrastructure? >If I understood Brock correctly he is looking for a robust and >probably high-performance (and cheap) alternative to NFS (I am too...).Have you considered drbd with gnbd? I''ve had lots of luck using gnbd with and without drbd, but I''m still working on making sure the secondary drbd becomes primary before gnbd starts running on the back-up server, in the case of a primary failure. Have you considered iSCSI Target Enterprise with or without drbd? While I only recently learned of ITE, this seems like a viable option as well. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Quoting "Christopher G. Stach II" <cgs@ldsys.net>:> Andrej Radonic wrote: >> Peter, >> this may be a stupid question but what do you mean by "In Xen it''s >> really simple" as opposed to the SAN approach? >> OCFS2 is "just" a filesystem, it doesn''t seem to be a way of >> distributing blockdevices via ethernet like NFS does. >> So what is the point in using OCFS2 in conjunction with Xen without >> a SAN or iSCSI or multipath infrastructure? >> If I understood Brock correctly he is looking for a robust and probably >> high-performance (and cheap) alternative to NFS (I am too...). >> Thanks ahead for some "enlightening" thoughts. > > The difference is between sharing between physical machines over a > network vs. sharing between virtual machines in a single physical > machine. >Yes, that''s precisely what I meant. Ocfs2 in my situation is a replacement for NFS for sharing files between domU''s on the same physical server. I basically virtualize the SAN, but this only works for the domU''s that exist on the same physical server. If you have different physical servers that need a shared storage (for performance or high availability reasons) then, as far as I know, you need to spend big money on a SAN, or iSCSI, etc. Peter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Peter Fokkinga schrieb:> > If you have different physical servers that need a shared > storage (for performance or high availability reasons) then, > as far as I know, you need to spend big money on a SAN, or > iSCSI, etc.Hello ! "Big money" is very relative. If you already habe Gigabit Ethernet Infrastructure for your servers at least you can use the software implementation of the Enterprise ISCSI Target (IET) iscsitarget.sourceforge.net. If you don''t want to configure and setup the software yourself you could buy http://www.open-e.com/. This is a Flash-DOM that fits in the IDE Port of a fileserver and turns this fileserver into an managable ISCSI Target (it is Linux based). (Ranges from about 300$ to 800$) -- __________________________________________________ Ralf Schenk fon (02 41) 9 91 21-0 fax (02 41) 9 91 21-59 rs@databay.de Databay AG Hüttenstraße 7 D-52068 Aachen www.databay.de Databay - einfach machen. _________________________________________________ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ralf Schenk wrote:> Peter Fokkinga schrieb: > >> If you have different physical servers that need a shared >> storage (for performance or high availability reasons) then, >> as far as I know, you need to spend big money on a SAN, or >> iSCSI, etc. >> > > Hello ! > > "Big money" is very relative. If you already habe Gigabit Ethernet > Infrastructure for your servers at least you can use the software > implementation of the Enterprise ISCSI Target (IET) > iscsitarget.sourceforge.net. >The best idea i have come up (though i wont be able to try it for some time) with with minimal patching is this: Use ocfs2 as file system, Use nbd+md to make a "networked raid" use iSCSI to export this network raid to Xen parent hosts, My question to those of you with better know how: Will this allow live migration (assume yes as now because of the "real" block device shown by ocfs2) Will domU''s living on the file system survive a file system node failure? (this is the who point of this thread) Brock> If you don''t want to configure and setup the software yourself you could > buy http://www.open-e.com/. This is a Flash-DOM that fits in the IDE > Port of a fileserver and turns this fileserver into an managable ISCSI > Target (it is Linux based). (Ranges from about 300$ to 800$) > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
At 11:35 AM 4/11/2006, you wrote:>The best idea i have come up (though i wont be able to try it for >some time) with with minimal patching is this: >Use ocfs2 as file system, >Use nbd+md to make a "networked raid" >use iSCSI to export this network raid to Xen parent hosts, >My question to those of you with better know how: > Will this allow live migration (assume yes as now because of the > "real" block device shown by ocfs2) > Will domU''s living on the file system survive a file system node > failure? (this is the who point of this thread) >BrockHow exactly will nbd+md failover? Does ocfs2 actually export "real" block devices, or are you just going to store disk images in a file on ocfs2? I thought ocfs2 was a filesystem. I don''t know enough about ocfs2 to answer your second question, but I think the answer lies in how nbd+md will failover and offer the ocfs2 filesystem. Obviously there''s some potential for data loss on the nbd. This sounds similar to what I''ve been trying to accomplish with drbd and gnbd. You can run drbd/gnbd without any patching, you just have to build modules against your kernel source, and then install the various packages to manage them. Depending on your time frame, drbd0.8 will make active/active configurations possible and you could run gnbd multipath, this would be a truly redundant setup with no failover trickery necessary. There is a relatively new pre-release of drbd0.8, but I''m not sure what the actual release time frame is. I''m also considering iscsi-target with drbd but it doesn''t currently offer multipath, so I''m not sure how cleanly it could failover. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Petersen wrote:> At 11:35 AM 4/11/2006, you wrote: >> The best idea i have come up (though i wont be able to try it for >> some time) with with minimal patching is this: >> Use ocfs2 as file system, >> Use nbd+md to make a "networked raid" >> use iSCSI to export this network raid to Xen parent hosts, >> My question to those of you with better know how: >> Will this allow live migration (assume yes as now because of the >> "real" block device shown by ocfs2) >> Will domU''s living on the file system survive a file system node >> failure? (this is the who point of this thread) >> Brock > > How exactly will nbd+md failover? Does ocfs2 actually export "real" > block devices, or are you just going to store disk images in a file on > ocfs2? I thought ocfs2 was a filesystem.Yeah ocfs2 is a file system, I realized the miss wording after i replied my apologies. Im thinking yes they would have to be files, which isnt ideal, but again im looking for migration with survivability in a block device (host) failing.> I don''t know enough about ocfs2 to answer your second question, but I > think the answer lies in how nbd+md will failover and offer the ocfs2 > filesystem. Obviously there''s some potential for data loss on the nbd. > > This sounds similar to what I''ve been trying to accomplish with drbd > and gnbd. You can run drbd/gnbd without any patching, you just have > to build modules against your kernel source, and then install the > various packages to manage them. Depending on your time frame, > drbd0.8 will make active/active configurations possible and you could > run gnbd multipath, this would be a truly redundant setup with no > failover trickery necessary. There is a relatively new pre-release of > drbd0.8, but I''m not sure what the actual release time frame is.all i can say is awesome, ive looked into drbd before i was using xen, it looked like dev was very slow beyond making it work. active active drbd would still require a cluster file system such as ocfs2 to keep things from tromping on each other. Thankyou for the heads up. Brock> > I''m also considering iscsi-target with drbd but it doesn''t currently > offer multipath, so I''m not sure how cleanly it could failover. > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Yeah ocfs2 is a file system, I realized the miss wording after i replied > my apologies. Im thinking yes they would have to be files, which isnt > ideal, but again im looking for migration with survivability in a block > device (host) failing.Anything that makes the storage available across the network, then, even NFS-root, would allow migration of the host. Host failure? Use HA-NFS. Don''t like NFS? Then as mentioned before, iscsi, nbd, etc., are your best options if you want to go cheap.> all i can say is awesome, ive looked into drbd before i was using xen, > it looked like dev was very slow beyond making it work. active active > drbd would still require a cluster file system such as ocfs2 to keep > things from tromping on each other.Make sure ocfs2 is actually supported on the block device of choice. It should work on nbd (but who knows about performance), but maybe nbd comes with some caveats of its own, like only accepting connections from a single host.... ocfs2 is designed for shared SCSI and the SAN, not the LAN -- I wouldn''t expect superb results from other environments without a whole lotta testing. John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
At 04:07 PM 4/11/2006, you wrote:>eah ocfs2 is a file system, I realized the miss wording after i >replied my apologies. Im thinking yes they would have to be files, >which isnt ideal, but again im looking for migration with >survivability in a block device (host) failing.Which is why I think GNBD is a good solution, especially if you want block devices instead of a network/cluster filesystem. I understand, as I''m trying to get the same thing going... You would have 2 servers running drbd0.8 (active-active) and GNBD multipath. I think this might be the only truly fault tolerant solution that doesn''t use a network/cluster filesystem. You can accomplish the same thing with drbd0.7, but it takes more fiddling and you loose multipath.>active active drbd would still require a cluster file system such as ocfs2Or GNBD, which is a cluster aware way of exporting block devices. Multiple dom0''s can then access all of the block devices, allowing for live migration. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Peter Fokkinga wrote:> >> >> The difference is between sharing between physical machines over a >> network vs. sharing between virtual machines in a single physical >> machine. >> > > Yes, that''s precisely what I meant. Ocfs2 in my situation is > a replacement for NFS for sharing files between domU''s on > the same physical server. I basically virtualize the SAN, > but this only works for the domU''s that exist on the same > physical server. > > If you have different physical servers that need a shared > storage (for performance or high availability reasons) then, > as far as I know, you need to spend big money on a SAN, or > iSCSI, etc.We are implementing some of the first Panasas cluster storage devices in the UK, for a govt. client and for the North West Grid. I should imagine the direct-flow client which Panasas provide would run well under Xen, and it would be fun to try. Drop me an email if you are interested in seeing how this performs on clusters. www.panasas.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users