Hi Nathan-- Thanks for sharing these results! That is very encouraging. Did you try using the same NFS client to talk to a normal Linux kernel NFS server (serving an ext3-based file system)? It may help determine whether the 2 GB problem is on the client or server. Thanks-- -Phil On 9/14/2005 15:52, Nathan Dauchy wrote:> Oleg Drokin wrote: >> On Fri, Sep 09, 2005 at 10:44:45AM -0600, Nathan Dauchy wrote: >>>> Because Samba is a userspace >>>> server, we have many many fewer problems than we do with the kernel NFS >>>> server, which uses the underlying file system in unique ways. >>> Has anyone tried using a user-space NFS server such as this one: >>> http://unfs3.sourceforge.net/ >> >> Not that we know of, if you would try, please tell us your experiences. >> Also userspace nfsd for linux was last changed in 2002 or so. >> Userspace nfsd should be much faster with Lustre compared to kernel nfsd. >> > > I tested both the kernel and user-space NFS servers packaged with SuSE > 9.1. The kernel server did not work, but the user-space implementation > works and gets about 85% of the performance we could possibly expect out > of the single ethernet link used to serve NFS. HOWEVER, files greater > than 2GB can''t be read or written. The client can read/write >2G files > to the local disk just fine. > > It is going to take me a long while to dig into the NFS server code to > see where that 2GB limit is coming from. If anyone has pointers of > where or what to look for it would help a lot! > > Thanks, > Nathan >
Hello! On Fri, Sep 09, 2005 at 10:44:45AM -0600, Nathan Dauchy wrote:> I understand that it is possible to export the Lustre filesystem with > NFS or CIFS, according to the FAQ and mailing list threads: > http://www.clusterfs.com/faq.html#install-2 > https://lists.clusterfs.com/pipermail/lustre-discuss/2004-December/000537.html > > Are there any "gotchas" in terms of security or performance when > configuring Samba? Does anyone have a smb.conf file they can share thatNo, Samba would behave like if you were exporting any other local FS.> >Because Samba is a userspace > >server, we have many many fewer problems than we do with the kernel NFS > >server, which uses the underlying file system in unique ways. > Has anyone tried using a user-space NFS server such as this one: > http://unfs3.sourceforge.net/Not that we know of, if you would try, please tell us your experiences. Also userspace nfsd for linux was last changed in 2002 or so. Userspace nfsd should be much faster with Lustre compared to kernel nfsd.> Is there a known reason why NFS export performance is slow, or is it > related to the kernel NFS server as well?Yes, the reason is known and it lies in the way kernel nfsd uses filesystems. Userspace nfsd should be free of these problems (it might have its own ones, though. Like it cannot usually tell mountpoints and traverses them unknowingly, and with that, nobody guarantees you that inode number spaces are non intersecting on different fulesystems). Bye, Oleg
Oleg Drokin wrote:> On Fri, Sep 09, 2005 at 10:44:45AM -0600, Nathan Dauchy wrote: >>>Because Samba is a userspace >>>server, we have many many fewer problems than we do with the kernel NFS >>>server, which uses the underlying file system in unique ways. >>Has anyone tried using a user-space NFS server such as this one: >>http://unfs3.sourceforge.net/ > > Not that we know of, if you would try, please tell us your experiences. > Also userspace nfsd for linux was last changed in 2002 or so. > Userspace nfsd should be much faster with Lustre compared to kernel nfsd. >I tested both the kernel and user-space NFS servers packaged with SuSE 9.1. The kernel server did not work, but the user-space implementation works and gets about 85% of the performance we could possibly expect out of the single ethernet link used to serve NFS. HOWEVER, files greater than 2GB can''t be read or written. The client can read/write >2G files to the local disk just fine. It is going to take me a long while to dig into the NFS server code to see where that 2GB limit is coming from. If anyone has pointers of where or what to look for it would help a lot! Thanks, Nathan
On Sep 14, 2005 15:57 -0400, Phil Schwan wrote:> Did you try using the same NFS client to talk to a normal Linux kernel NFS > server (serving an ext3-based file system)? It may help determine whether > the 2 GB problem is on the client or server.I think (though not positive) that NFS v2 does not support > 2GB files, while NFS v3 does.> On 9/14/2005 15:52, Nathan Dauchy wrote: > > I tested both the kernel and user-space NFS servers packaged with SuSE > > 9.1. The kernel server did not work, but the user-space implementation > > works and gets about 85% of the performance we could possibly expect out > > of the single ethernet link used to serve NFS. HOWEVER, files greater > > than 2GB can''t be read or written. The client can read/write >2G files > > to the local disk just fine. > > > > It is going to take me a long while to dig into the NFS server code to > > see where that 2GB limit is coming from. If anyone has pointers of > > where or what to look for it would help a lot!Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Andreas Dilger wrote:> On Sep 14, 2005 15:57 -0400, Phil Schwan wrote: >>Did you try using the same NFS client to talk to a normal Linux kernel NFS >>server (serving an ext3-based file system)? It may help determine whether >>the 2 GB problem is on the client or server. > > I think (though not positive) that NFS v2 does not support > 2GB files, > while NFS v3 does. >OK, we will have to go back and check the mount options to see which version of the protocol was used. If it turns out that the user-space NFS server implementation shipped with SuSE only supports v2, then I''ll take a whack at http://unfs3.sourceforge.net/. Thanks to all for the pointers! -Nathan
Greetings all, I understand that it is possible to export the Lustre filesystem with NFS or CIFS, according to the FAQ and mailing list threads: http://www.clusterfs.com/faq.html#install-2 https://lists.clusterfs.com/pipermail/lustre-discuss/2004-December/000537.html Are there any "gotchas" in terms of security or performance when configuring Samba? Does anyone have a smb.conf file they can share that allows root to mount the lustre filesystem and controls permissions as if the Lustre FS was mounted natively? (It might be helpful to add this to the FAQ.) Also, it seems like CIFS is recommended over NFS for both stability and performance reasons. Quoting Phil Schwan:> Because Samba is a userspace > server, we have many many fewer problems than we do with the kernel NFS > server, which uses the underlying file system in unique ways.Has anyone tried using a user-space NFS server such as this one: http://unfs3.sourceforge.net/ SuSE also comes with two NFS server packages, so it shouldn''t be too hard for me to try them both: nfs-utils-1.0.6-91.x86_64.rpm (kernel) nfs-server-2.2beta51-195.x86_64.rpm (user-space) Is there a known reason why NFS export performance is slow, or is it related to the kernel NFS server as well? Thanks, Nathan