Hi all; Is there any experiance about exporting Lustre via NFS to outside world? Any whitepapers, how to guides, schemas etc.. Any info welcome regards <http://www.sun.com/> http://www.sun.com/emrkt/sigs/6g_top.gif Mertol Ozyoney Storage Practice - Sales Manager Sun Microsystems, TR Istanbul TR Phone +902123352200 Mobile +905339310752 Fax +902123352222 Email <mailto:Ayca.Yalcin at Sun.COM> mertol.ozyoney at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071214/05ee7edc/attachment-0002.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 1257 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071214/05ee7edc/attachment-0002.gif
We have been discussing among ourselves. Oleg Drokin is going to respond ... Oleg? Mertol Ozyoney wrote:> > Hi all; > > > > Is there any experiance about exporting Lustre via NFS to outside > world? Any whitepapers, how to guides, schemas etc.. > > > > Any info welcome > > > > regards > > > > > > http://www.sun.com/emrkt/sigs/6g_top.gif <http://www.sun.com/> > > > > *Mertol Ozyoney * > Storage Practice - Sales Manager > > *Sun Microsystems, TR* > Istanbul TR > Phone +902123352200 > Mobile +905339310752 > Fax +902123352222 > Email mertol.ozyoney at Sun.COM <mailto:Ayca.Yalcin at Sun.COM> > > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071213/eb7eafa2/attachment-0002.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1257 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071213/eb7eafa2/attachment-0002.gif
Hello! On Dec 13, 2007, at 7:50 PM, Mertol Ozyoney wrote:> Is there any experiance about exporting Lustre via NFS to outside > world? Any whitepapers, how to guides, schemas etc..There are some people who do this, also HP support this with their Lustre spin-off and I think they are one of the biggest users at the moment. Their latest tests indicate that lustre reexported with NFS achieves around 90% of the lustre link speed (this was tested with gigabit connections, but I see nothing that would prevent us from this on higher connection speeds provided there is enough CPU and bus bandwidth). These results were achieved with 2.6 kernel on (2.4 kernels perform significantly worse, like 2x worse due to very small i/o submitted to lustre). To obtain this sort of speed special patch (bug 13371 in our bugzilla) was used on lustre 1.4.11, that is not part of released lustre yet, though it will likely be included into 1.4.12. Port of the patch is also underway for 1.6 lustre releases. Metadata speed is mostly around 1/2 or 2/3 compared to NFS exported off ext3 if lustre transport is tcp. For lower-latency network (like infiniband) we do better, but I do not have any numbers at hand. Exporting lustre via NFS is not unlike any other filesystem, you just write /etc/exports entry for fs you want to export and you are mostly done (for some older releases you also need to specify fsid parameter). Some people (HP) also export one lustre filesystem via several NFS servers (to many clients) that allows to achieve better speed when lustre link is significantly faster than NFS side link (also for load-balancing?), for this case normally you need to have identical fsid= setting in your /etc/exports, though current 1.6 release eliminate this requirement. Also in this case you cannot have consistent posix locking unless you use 2.6.22 kernel or above. (also sles9 kernels have special patch that allows this. The patch was dropped in sles10, though). Bye, Oleg
On Thu, 13 Dec 2007, Oleg Drokin wrote:> Hello! > > On Dec 13, 2007, at 7:50 PM, Mertol Ozyoney wrote: > >> Is there any experiance about exporting Lustre via NFS to outside >> world? Any whitepapers, how to guides, schemas etc.. > > There are some people who do this, also HP support this with their > Lustre spin-off and I think they are one of the biggest users at the > moment.We have HP''s currently available Lustre product (SFS 2.2-0 based on Lustre 1.4.6) which only supports NFS re-export with 2.4 kernels. There was some interesting discussion about the re-export with this version for the 2.6 kernels: https://bugzilla.lustre.org/show_bug.cgi?id=10420 We are told that a newer SFS release should be available sometime in January 08 and will support NFS re-export with 2.6 kernels, though we were also told this prior to the previous release. And according to https://bugzilla.lustre.org/show_bug.cgi?id=13371 there still seems to be some issues that will be hopefully sorted out by then. -k
Hello! On Dec 14, 2007, at 4:37 PM, Kaizaad Bilimorya wrote:> We have HP''s currently available Lustre product (SFS 2.2-0 based on > Lustre > 1.4.6) which only supports NFS re-export with 2.4 kernels. There was > some > interesting discussion about the re-export with this version for the > 2.6 > kernels: > https://bugzilla.lustre.org/show_bug.cgi?id=10420This one is old and all the code long landed.> We are told that a newer SFS release should be available sometime in > January 08 and will support NFS re-export with 2.6 kernels, though > we were > also told this prior to the previous release. And according to > https://bugzilla.lustre.org/show_bug.cgi?id=13371 > there still seems to be some issues that will be hopefully sorted > out by > then.13371 has patches that HP tried and decided they like new performance, so those patches would be in their new release of some sort. Bye, Oleg
Oleg thanks for your excellent answers. To my understanding (My knowledge on Linux is not broad) We will have stable NFS export capabilities both for 1.4x and 1.6.x with in weeks? (Project will be istalled in februrary and NFS Gateways might wait till the end of fabruary) Basically we will attach a Sun 25k to Lustre partition. I plan to use 2x Gateway server each with IB to Lustre and 10 Gbit ethernet to 25k . PS: BTW do HP support IB loadbalancing with their version of Lustre? regards Mertol Ozyoney Storage Practice - Sales Manager Sun Microsystems, TR Istanbul TR Phone +902123352200 Mobile +905339310752 Fax +902123352222 Email mertol.ozyoney at Sun.COM -----Original Message----- From: lustre-discuss-bounces at clusterfs.com [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of Oleg Drokin Sent: Friday, December 14, 2007 5:18 AM To: Mertol.Ozyoney at Sun.COM Cc: e.poda at remivac.com; lustre-solutions at sun.com; lustre-discuss at clusterfs.com; i.apps at remivac.com Subject: Re: [Lustre-discuss] Lustre NFs expport Hello! On Dec 13, 2007, at 7:50 PM, Mertol Ozyoney wrote:> Is there any experiance about exporting Lustre via NFS to outside > world? Any whitepapers, how to guides, schemas etc..There are some people who do this, also HP support this with their Lustre spin-off and I think they are one of the biggest users at the moment. Their latest tests indicate that lustre reexported with NFS achieves around 90% of the lustre link speed (this was tested with gigabit connections, but I see nothing that would prevent us from this on higher connection speeds provided there is enough CPU and bus bandwidth). These results were achieved with 2.6 kernel on (2.4 kernels perform significantly worse, like 2x worse due to very small i/o submitted to lustre). To obtain this sort of speed special patch (bug 13371 in our bugzilla) was used on lustre 1.4.11, that is not part of released lustre yet, though it will likely be included into 1.4.12. Port of the patch is also underway for 1.6 lustre releases. Metadata speed is mostly around 1/2 or 2/3 compared to NFS exported off ext3 if lustre transport is tcp. For lower-latency network (like infiniband) we do better, but I do not have any numbers at hand. Exporting lustre via NFS is not unlike any other filesystem, you just write /etc/exports entry for fs you want to export and you are mostly done (for some older releases you also need to specify fsid parameter). Some people (HP) also export one lustre filesystem via several NFS servers (to many clients) that allows to achieve better speed when lustre link is significantly faster than NFS side link (also for load-balancing?), for this case normally you need to have identical fsid= setting in your /etc/exports, though current 1.6 release eliminate this requirement. Also in this case you cannot have consistent posix locking unless you use 2.6.22 kernel or above. (also sles9 kernels have special patch that allows this. The patch was dropped in sles10, though). Bye, Oleg _______________________________________________ Lustre-discuss mailing list Lustre-discuss at clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Hello! On Dec 17, 2007, at 5:53 AM, Mertol Ozyoney wrote:> To my understanding (My knowledge on Linux is not broad) > We will have stable NFS export capabilities both for 1.4x and 1.6.x > with in > weeks? (Project will be istalled in februrary and NFS Gateways might > wait > till the end of fabruary)Yes, I think so. In fact some people use 1.4.x NFS servers right now and the most annoying problem so far is speed issue tracked in bug 13371.> Basically we will attach a Sun 25k to Lustre partition. > I plan to use 2x Gateway server each with IB to Lustre and 10 Gbit > ethernet > to 25k .Make sure you have enough bus bandwidth on the gateway for both connections. If you plan to run two separate instances of NFS servers to same lustre FS on same gateways, you need to come up with some way for sun25k (I presume it is single machine with many CPUs?) to use both NFS servers for same export.> PS: BTW do HP support IB loadbalancing with their version of Lustre?I do not know. I''m not even sure what IB loadbalancing is. Bye, Oleg
> some people use 1.4.x NFS servers right nowYes, for example the current version of HP''s SFS product uses Lustre 1.4.6.4 (with patches) and supports NFS export *from 2.4 kernel nodes only* (because support for NFS export on 2.6 kernel requires later Lustre 1.4x stream code). -----Original Message----- From: Oleg Drokin [mailto:Oleg.Drokin at Sun.COM] Sent: 19 December 2007 03:12 To: Mertol.Ozyoney at Sun.COM Cc: e.poda at remivac.com; lustre-solutions at Sun.COM; lustre-discuss at clusterfs.com; i.apps at remivac.com Subject: Re: [Lustre-discuss] Lustre NFs expport Hello! On Dec 17, 2007, at 5:53 AM, Mertol Ozyoney wrote:> To my understanding (My knowledge on Linux is not broad) > We will have stable NFS export capabilities both for 1.4x and 1.6.x > with in > weeks? (Project will be istalled in februrary and NFS Gateways might > wait > till the end of fabruary)Yes, I think so. In fact some people use 1.4.x NFS servers right now and the most annoying problem so far is speed issue tracked in bug 13371.> Basically we will attach a Sun 25k to Lustre partition. > I plan to use 2x Gateway server each with IB to Lustre and 10 Gbit > ethernet > to 25k .Make sure you have enough bus bandwidth on the gateway for both connections. If you plan to run two separate instances of NFS servers to same lustre FS on same gateways, you need to come up with some way for sun25k (I presume it is single machine with many CPUs?) to use both NFS servers for same export.> PS: BTW do HP support IB loadbalancing with their version of Lustre?I do not know. I''m not even sure what IB loadbalancing is. Bye, Oleg