Thibault Godouet
2015-Aug-10 08:19 UTC
[Gluster-users] NFS file locking with gluster 3.7.3
Thanks Niel for your helpful answer. Regarding the locking, indeed that solves my issue. Now I'm wondering how to monitor this. The best I have so far is get the list of RPC binds and the TCP/UDP port in particular, and then run a lsof to find out if it is Gluster. Should work, but a bit indirect. If someone knows a better way I'd be interested to know. As for Ganesha, I saw articles explaining that it effectively removes layers, hence why I thought NFS v3 via Ganesha would be faster than native Gluster NFS. Given your answer I take it there are other moving parts / differences. Is there a general known guideline on which is best when? E.g. does one handle better small files than the other one or something like that? On 5 Aug 2015 7:06 pm, "Niels de Vos" <ndevos at redhat.com> wrote:> On Wed, Aug 05, 2015 at 04:11:47PM +0100, Thibault Godouet wrote: > > Looking around I get the impression that file locking (NLM) may simply > not > > be supported in glusterfs's built-in NFS server. > > This is actually supported. But note that you can not run a userspace > NLM implementation provided by a NFS-server (Gluster/NFS or NFS-Ganesha) > on a system that acts as an NFS-client. The Linux kernel NFS-client uses > the lockd kernel module, and there can be only one NLM implementation be > registered at rpcbind. Whichever service (nfs-client or nfs-server) > starts first, will be able to register itself, the 2nd one will (mostly > silently) fail. > > > I get the impression that Ganesha is aimed at supporting NFS better, and > > presumably supports locking well, so I should give it a try (If I > > understand well the performance is also likely to be higher, which is a > > nice bonus!) > > NFS-Ganesha offers more features than Gluster/NFS. The performance is > highly dependent on the workload, Gluster/NFS can be faster for many of > them. > > Cheers, > Niels > > > > > > If someone could confirm this that would be useful to make sure I'm going > > in the right direction. > > > > Thanks, > > Thibault. > > On 4 Aug 2015 1:23 pm, "Thibault Godouet" <tibo92 at godouet.net> wrote: > > > > > Hi, > > > > > > > > > > > > I have a cluster of 2 servers running 3.7.3 with replication, and > standard > > > NFS (no ganesha). This in on CentOS 6. > > > > > > I use CTDB with 2 virtual IPs (one for each server in a normal > > > situation) to share the volume over NFS and CIFS (samba). > > > > > > > > > > > > fnctl() file locking doesn't seem to work when the volume is mounted > over > > > NFS. > > > > > > This is apparent with a 'svn info' (svn 1.8 if it made any difference) > in > > > a local working copy: > > > > > > > > > > > > $ svn info > > > svn: E200033: Another process is blocking the working copy database, or > > > the underlying filesystem does not support file locking; if the working > > > copy is on a network filesystem, make sure file locking has been > enabled on > > > the file server > > > svn: E200033: sqlite[S5]: database is locked, executing statement > 'PRAGMA > > > synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA > foreign_keys=OFF;PRAGMA > > > locking_mode = NORMAL;' > > > > > > > > > > > > a strace shows: > > > > > > > > > > > > $ svn info > > > svn: E200033: Another process is blocking the working copy database, or > > > the underlying filesystem does not support file locking; if the working > > > copy is on a network filesystem, make sure file locking has been > enabled on > > > the file server > > > svn: E200033: sqlite[S5]: database is locked, executing statement > 'PRAGMA > > > synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA > foreign_keys=OFF;PRAGMA > > > locking_mode = NORMAL;' > > > > > > > > > > > > Everything seems to work fine on native Gluster (FUSE) mounts: the same > > > 'svn info' works nicely. > > > > > > I can't really use native mounts due to the performance hit (many small > > > files) and the fact I would need to install the gluster client > software on > > > every server. > > > > > > > > > > > > Is fnctl() file locking supported in Gluster NFS mounts? If so, any > idea > > > why it doesn't work for me? > > > > > > > > > > > > Thanks, > > > > > > Thibault. > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150810/10bff37b/attachment.html>
On Mon, Aug 10, 2015 at 09:19:25AM +0100, Thibault Godouet wrote:> Thanks Niel for your helpful answer. > > Regarding the locking, indeed that solves my issue. Now I'm wondering how > to monitor this. The best I have so far is get the list of RPC binds and > the TCP/UDP port in particular, and then run a lsof to find out if it is > Gluster. Should work, but a bit indirect. If someone knows a better way > I'd be interested to know.That is almost how I do it as well. Instead of 'lsof' I use 'netstat' or 'ss', depending on the Linux distribution.> As for Ganesha, I saw articles explaining that it effectively removes > layers, hence why I thought NFS v3 via Ganesha would be faster than native > Gluster NFS. Given your answer I take it there are other moving parts / > differences. Is there a general known guideline on which is best when? > E.g. does one handle better small files than the other one or something > like that?I am not aware of any guidelines for this. The difference in performance is highly dependent on the workload and use-case. There is little difference in the layers between Gluster/NFS and NFS-Ganesha, both are userspace nfs-server implementations (neither has context switches for the Linux-VFS like fuse mounts have). If you need the best performance, you should problaby just try both configurations, and run your intended workload against the servers. Artificial/standard tests most often do not emulate a real workload. HTH, Niels> On 5 Aug 2015 7:06 pm, "Niels de Vos" <ndevos at redhat.com> wrote: > > > On Wed, Aug 05, 2015 at 04:11:47PM +0100, Thibault Godouet wrote: > > > Looking around I get the impression that file locking (NLM) may simply > > not > > > be supported in glusterfs's built-in NFS server. > > > > This is actually supported. But note that you can not run a userspace > > NLM implementation provided by a NFS-server (Gluster/NFS or NFS-Ganesha) > > on a system that acts as an NFS-client. The Linux kernel NFS-client uses > > the lockd kernel module, and there can be only one NLM implementation be > > registered at rpcbind. Whichever service (nfs-client or nfs-server) > > starts first, will be able to register itself, the 2nd one will (mostly > > silently) fail. > > > > > I get the impression that Ganesha is aimed at supporting NFS better, and > > > presumably supports locking well, so I should give it a try (If I > > > understand well the performance is also likely to be higher, which is a > > > nice bonus!) > > > > NFS-Ganesha offers more features than Gluster/NFS. The performance is > > highly dependent on the workload, Gluster/NFS can be faster for many of > > them. > > > > Cheers, > > Niels > > > > > > > > > > If someone could confirm this that would be useful to make sure I'm going > > > in the right direction. > > > > > > Thanks, > > > Thibault. > > > On 4 Aug 2015 1:23 pm, "Thibault Godouet" <tibo92 at godouet.net> wrote: > > > > > > > Hi, > > > > > > > > > > > > > > > > I have a cluster of 2 servers running 3.7.3 with replication, and > > standard > > > > NFS (no ganesha). This in on CentOS 6. > > > > > > > > I use CTDB with 2 virtual IPs (one for each server in a normal > > > > situation) to share the volume over NFS and CIFS (samba). > > > > > > > > > > > > > > > > fnctl() file locking doesn't seem to work when the volume is mounted > > over > > > > NFS. > > > > > > > > This is apparent with a 'svn info' (svn 1.8 if it made any difference) > > in > > > > a local working copy: > > > > > > > > > > > > > > > > $ svn info > > > > svn: E200033: Another process is blocking the working copy database, or > > > > the underlying filesystem does not support file locking; if the working > > > > copy is on a network filesystem, make sure file locking has been > > enabled on > > > > the file server > > > > svn: E200033: sqlite[S5]: database is locked, executing statement > > 'PRAGMA > > > > synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA > > foreign_keys=OFF;PRAGMA > > > > locking_mode = NORMAL;' > > > > > > > > > > > > > > > > a strace shows: > > > > > > > > > > > > > > > > $ svn info > > > > svn: E200033: Another process is blocking the working copy database, or > > > > the underlying filesystem does not support file locking; if the working > > > > copy is on a network filesystem, make sure file locking has been > > enabled on > > > > the file server > > > > svn: E200033: sqlite[S5]: database is locked, executing statement > > 'PRAGMA > > > > synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA > > foreign_keys=OFF;PRAGMA > > > > locking_mode = NORMAL;' > > > > > > > > > > > > > > > > Everything seems to work fine on native Gluster (FUSE) mounts: the same > > > > 'svn info' works nicely. > > > > > > > > I can't really use native mounts due to the performance hit (many small > > > > files) and the fact I would need to install the gluster client > > software on > > > > every server. > > > > > > > > > > > > > > > > Is fnctl() file locking supported in Gluster NFS mounts? If so, any > > idea > > > > why it doesn't work for me? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Thibault. > > > > > > > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > http://www.gluster.org/mailman/listinfo/gluster-users > > > >-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150813/acdfba86/attachment.sig>