Hi, I am setting up a xen 3 enviroment that has a file backend and 2 application servers with live emegration between the 2 application servers. --------- --------- | app 1 | | app 2 | --------- --------- \ / \ / \ / ---------------- | file backend | ---------------- I am planing on using lustre clustre file system on the file backend. Are there any out there that have tried this set up with success ? Have you got any thoughts on this ? Regards, Karsten _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher G. Stach II
2006-May-18 17:54 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Karsten Nielsen wrote:> Hi, > > I am setting up a xen 3 enviroment that has a file backend and 2 > application servers with live emegration between the 2 application servers. > > --------- --------- > | app 1 | | app 2 | > --------- --------- > \ / > \ / > \ / > ---------------- > | file backend | > ---------------- > > I am planing on using lustre clustre file system on the file backend. > > Are there any out there that have tried this set up with success ? > Have you got any thoughts on this ?I haven''t tried that exact setup, but it sounds like a very bad idea. Use LVM2 or a raw partition. -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Karsten Nielsen
2006-May-18 18:04 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Christopher G. Stach II wrote:> Karsten Nielsen wrote: >> Hi, >> >> I am setting up a xen 3 enviroment that has a file backend and 2 >> application servers with live emegration between the 2 application servers. >> >> --------- --------- >> | app 1 | | app 2 | >> --------- --------- >> \ / >> \ / >> \ / >> ---------------- >> | file backend | >> ---------------- >> >> I am planing on using lustre clustre file system on the file backend. >> >> Are there any out there that have tried this set up with success ? >> Have you got any thoughts on this ? > > I haven''t tried that exact setup, but it sounds like a very bad idea. > Use LVM2 or a raw partition. >How will you make the LVM2 or raw partitions avalible to the application servers ? - i have 2 physical application servers and 1 file backend server. That means that i have 3 servers. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher G. Stach II
2006-May-18 18:09 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Karsten Nielsen wrote:> > How will you make the LVM2 or raw partitions avalible to the application > servers ? - i have 2 physical application servers and 1 file backend > server. That means that i have 3 servers. >How would you make a file backend available to multiple physical servers? That''s a rhetorical question, but to answer yours, probably NBD. -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Karsten Nielsen
2006-May-18 18:25 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Mayby I frased my question wrong. I have read a lot on the mailing list about pros and cons of different ways to make the file backend avalible to multiple physical servers. But it seems that there are no real good answer to that question as fare as I have read. There are pros and cons to every solution. What I was looking for is a file backend that performs very well and is relayable. If I want to use ocfs2 I cannot resize the file system. (http://www.mail-archive.com/ocfs2-users@oss.oracle.com/msg00059.html) If I want to use GFS it''s performence is not that great (http://guialivre.governoeletronico.gov.br/mediawiki/index.php/TesteGFSGraficoRaid10_ext3vsgfs and http://guialivre.governoeletronico.gov.br/mediawiki/index.php/TestesGFS ) Mayby I am making this to complicated and should not worry about the lock system of clustre file systems what I am looking for is realy performance and relyability. Any hints ? And why do you think that Lustre is at bad idea ? Christopher G. Stach II wrote:> Karsten Nielsen wrote: >> How will you make the LVM2 or raw partitions avalible to the application >> servers ? - i have 2 physical application servers and 1 file backend >> server. That means that i have 3 servers. >> > > How would you make a file backend available to multiple physical > servers? That''s a rhetorical question, but to answer yours, probably NBD. >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher G. Stach II
2006-May-18 18:32 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Karsten Nielsen wrote:> Mayby I frased my question wrong. I have read a lot on the mailing list > about pros and cons of different ways to make the file backend avalible > to multiple physical servers. > > But it seems that there are no real good answer to that question as fare > as I have read. There are pros and cons to every solution. > > What I was looking for is a file backend that performs very well and is > relayable. > > If I want to use ocfs2 I cannot resize the file system. > (http://www.mail-archive.com/ocfs2-users@oss.oracle.com/msg00059.html) > If I want to use GFS it''s performence is not that great > (http://guialivre.governoeletronico.gov.br/mediawiki/index.php/TesteGFSGraficoRaid10_ext3vsgfs > and > http://guialivre.governoeletronico.gov.br/mediawiki/index.php/TestesGFS ) > > Mayby I am making this to complicated and should not worry about the > lock system of clustre file systems what I am looking for is realy > performance and relyability. > > Any hints ? > And why do you think that Lustre is at bad idea ?I think Lustre is fine, although it''s most likely overkill for just two app servers. DRBD is probably closer to what you want. I think the file backend portion is a bad idea. It''s a bad idea to unnecessarily involve a filesystem cache. Unless I had a very large cluster, I would still probably use GFS. -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thursday 18 May 2006 1:25 pm, Karsten Nielsen wrote:> Mayby I frased my question wrong. I have read a lot on the mailing list > about pros and cons of different ways to make the file backend avalible > to multiple physical servers.i think the main objection for most of us is the use of file backed domUs. using files for the domU''s storage is fine for testing on a single machine, but for performance it''s much better to use block devices. if you want to do live migration, then you have to make those block devices available to both physical boxes; therefore you should use a shared block device (NDB, DRBD, iSCSI, AoE, FC, etc). some people have used NFS (or samba!) file servers to hold the files holding the domU''s storage, but performance suffers a lot, and all involved resources are much more heavily taxed (dom0 RAM, LAN bandwitdh, NFS box RAM). another ''mixed'' usage is to use a shared block device, but instead of splitting it with LVM2 or EVMS, you could use a cluster file system (usually GFS or OCFS2) to hold the files. it''s somewhat ''lighter'' than using a fileserver, but still the filesystem is a significant overhead compared to going straight to the block device. also, in both cases, the VBD implementation is much more streamlined when you use a block device instead of a file. another way to see it is that the guest OSs want to have some disks. disks are block devices. those virtual disks won''t be ''real'' physical disks, but still want to do block-like IO. therefore, it''s more efficient if the whole chain of data between the real disks and the guest OS is done with block device protocols, without any filesystem in between. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Karsten Nielsen
2006-May-18 18:50 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Christopher G. Stach II wrote:> Karsten Nielsen wrote: >> Mayby I frased my question wrong. I have read a lot on the mailing list >> about pros and cons of different ways to make the file backend avalible >> to multiple physical servers. >> >> But it seems that there are no real good answer to that question as fare >> as I have read. There are pros and cons to every solution. >> >> What I was looking for is a file backend that performs very well and is >> relayable. >> >> If I want to use ocfs2 I cannot resize the file system. >> (http://www.mail-archive.com/ocfs2-users@oss.oracle.com/msg00059.html) >> If I want to use GFS it''s performence is not that great >> (http://guialivre.governoeletronico.gov.br/mediawiki/index.php/TesteGFSGraficoRaid10_ext3vsgfs >> and >> http://guialivre.governoeletronico.gov.br/mediawiki/index.php/TestesGFS ) >> >> Mayby I am making this to complicated and should not worry about the >> lock system of clustre file systems what I am looking for is realy >> performance and relyability. >> >> Any hints ? >> And why do you think that Lustre is at bad idea ? > > I think Lustre is fine, although it''s most likely overkill for just two > app servers. DRBD is probably closer to what you want. I think the > file backend portion is a bad idea. It''s a bad idea to unnecessarily > involve a filesystem cache. Unless I had a very large cluster, I would > still probably use GFS. >The reason fore having a file backend is that I will be able to live imigrate domU''s from one server to the next in case of hardware failor or extremly high load on one of the physical servers. If I understand you correct you suggest that I make a lot of storage avaliable on both application servers and syncronize data with DRBD ? But if I add a 3. application server that setup does not work unless i add storage space on all three application servers and implement that in the DRBD ? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Luster is built for large super clusters, (compute clusters) Its built to be fast (over a few gigs/s throughput) but sucks for small files. We will be evaluating luster/pvfs2/gpfs for our compute clusters (about 800 nodes) im not sure how luster does failure when a storage node dies, if you find out let me know. But with luster you would have to store system images as files not devices. Others can chime in but in most the xen docs ive read file backed VM''s were not the preferred. A database cluster file system like ocfs or somthing else would prob work better for reliability. Brock On May 18, 2006, at 2:25 PM, Karsten Nielsen wrote:> Mayby I frased my question wrong. I have read a lot on the mailing > list about pros and cons of different ways to make the file backend > avalible to multiple physical servers. > > But it seems that there are no real good answer to that question as > fare as I have read. There are pros and cons to every solution. > > What I was looking for is a file backend that performs very well > and is relayable. > > If I want to use ocfs2 I cannot resize the file system. (http:// > www.mail-archive.com/ocfs2-users@oss.oracle.com/msg00059.html) > If I want to use GFS it''s performence is not that great (http:// > guialivre.governoeletronico.gov.br/mediawiki/index.php/ > TesteGFSGraficoRaid10_ext3vsgfs and http:// > guialivre.governoeletronico.gov.br/mediawiki/index.php/TestesGFS ) > > Mayby I am making this to complicated and should not worry about > the lock system of clustre file systems what I am looking for is > realy performance and relyability. > > Any hints ? > And why do you think that Lustre is at bad idea ? > > Christopher G. Stach II wrote: >> Karsten Nielsen wrote: >>> How will you make the LVM2 or raw partitions avalible to the >>> application >>> servers ? - i have 2 physical application servers and 1 file backend >>> server. That means that i have 3 servers. >>> >> How would you make a file backend available to multiple physical >> servers? That''s a rhetorical question, but to answer yours, >> probably NBD. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thursday 18 May 2006 1:50 pm, Karsten Nielsen wrote:> The reason fore having a file backend is that I will be able to live > imigrate domU''s from one server to the next in case of hardware failor > or extremly high load on one of the physical servers.live migration is better done with shared block devices. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher G. Stach II
2006-May-18 19:54 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Karsten Nielsen wrote:> The reason fore having a file backend is that I will be able to live > imigrate domU''s from one server to the next in case of hardware failor > or extremly high load on one of the physical servers.That''s not a valid reason. :) There''s no requirement to use files for that. Use NBD and GFS or DRBD.> If I understand you correct you suggest that I make a lot of storage > avaliable on both application servers and syncronize data with DRBD ? > > But if I add a 3. application server that setup does not work unless i > add storage space on all three application servers and implement that in > the DRBD ?DRBD is better for HA. Do you want to load balance or just failover? -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Karsten Nielsen
2006-May-18 20:05 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Christopher G. Stach II wrote:> Karsten Nielsen wrote: >> The reason fore having a file backend is that I will be able to live >> imigrate domU''s from one server to the next in case of hardware failor >> or extremly high load on one of the physical servers. > > That''s not a valid reason. :) There''s no requirement to use files for > that. Use NBD and GFS or DRBD.Sorry my bad. I ment shared storage for live imigration. Not a file backend. I see the mix up. And my bad english.... :( My picture should have look like this. Sorry! --------- --------- | app 1 | | app 2 | --------- --------- \ / \ / \ / ---------------- | shared storage | ---------------- I would like to have a very flexible setup. Where I can add application servers as well as more shared storage ( my storage i a server with sata raid 5 up to 4TB) should I share my storage DRBD and that way be able to add an other shared storage device at later point for HA and on my application servers use GFS or OCFS2 ? I am testing with Lustre on my shared storage at the moment.> >> If I understand you correct you suggest that I make a lot of storage >> avaliable on both application servers and syncronize data with DRBD ? >> >> But if I add a 3. application server that setup does not work unless i >> add storage space on all three application servers and implement that in >> the DRBD ? > > DRBD is better for HA. Do you want to load balance or just failover? >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher G. Stach II
2006-May-18 20:13 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Karsten Nielsen wrote:> should I share my storage DRBD and that way be able to add an other > shared storage device at later point for HA and on my application > servers use GFS or OCFS2 ?It sounds like you need something more like GFS, then. I''m not confident with OCFS2 and its limitations to even consider right now. I would export the storage device from the disk server with NBD, make an CLVM2 volume on it, and put GFS on it. That way you can resize the share if you need to and you get the consistent naming that you need for migration. -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Karsten Nielsen
2006-May-18 20:23 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Christopher G. Stach II wrote:> Karsten Nielsen wrote: >> should I share my storage DRBD and that way be able to add an other >> shared storage device at later point for HA and on my application >> servers use GFS or OCFS2 ? > > It sounds like you need something more like GFS, then. I''m not > confident with OCFS2 and its limitations to even consider right now. > > I would export the storage device from the disk server with NBD, make an > CLVM2 volume on it, and put GFS on it. That way you can resize the > share if you need to and you get the consistent naming that you need for > migration.Thank you for your thoughts on this subject! It sounds like a plan but how about the performance in GFS I have read that it is not very good have you god any experince with that ? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher G. Stach II
2006-May-18 20:46 UTC
Re: [Xen-users] lustre clustre file system and xen 3
Karsten Nielsen wrote:> Thank you for your thoughts on this subject! > > It sounds like a plan but how about the performance in GFS I have read > that it is not very good have you god any experince with that ?You''re certainly not going to get full disk speed with it, but the overhead without a network involved is negligible. It makes NFS cry. Of course, YMMV. The only reliable way I''ve found to compare filesystems is real world benchmarking in that given domain. If you do go ahead and benchmark some of them, I''m sure that many people on the list would be interested in the numbers. I''d like to see OCFS2 vs GFS for multiple use cases. -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Karsten Nielsen
2006-May-24 10:47 UTC
Re: [Xen-users] lustre clustre file system and xen 3
OCFS2 is not of intrest any more because OCFS2 do not support mmap which is used by apt-get in debian. Christopher G. Stach II wrote:> Karsten Nielsen wrote: >> Thank you for your thoughts on this subject! >> >> It sounds like a plan but how about the performance in GFS I have read >> that it is not very good have you god any experince with that ? > > You''re certainly not going to get full disk speed with it, but the > overhead without a network involved is negligible. It makes NFS cry. > Of course, YMMV. > > The only reliable way I''ve found to compare filesystems is real world > benchmarking in that given domain. If you do go ahead and benchmark > some of them, I''m sure that many people on the list would be interested > in the numbers. I''d like to see OCFS2 vs GFS for multiple use cases. >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi All, I am trying to find a way that my domains can check their own network usage. Usually iptraf will do it but in this case iptraf check eth0 and collects traffic stats for all traffic on that box rather than just the traffic from that domain. I assume I either have the domains setup wrong in that they should have a virtual interface each or there is another package I should use to stat. Anyone have any idea''s? Cheers, Dee -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.394 / Virus Database: 268.8.0/353 - Release Date: 5/31/2006 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Apparently Analagous Threads
- best practice for lustre clustre startup
- gluster-volgen - syntax for mirroring/distributing across 6 nodes
- Posix warning : Access to ... is crossing device
- Input/output error when running `ls` and `cd` on directories
- 3.1.2 Debian - client_rpc_notify "failed to get the port number for remote subvolume"