thr3ads.net - Gluster users - [Gluster-users] NFS file locks [Apr 2010]

If this information is useful, please help other people find it:
Share via:

Steinmetz, Ian

2010-Apr-29 14:46 UTC

[Gluster-users] NFS file locks

I'm seeing an issue where I can't lock files on a NFS exported
GlusterFS mount.  I have two servers connected to each other doing AFR
to provide a high available NFS server (mirror the content, one VIP for
NFS mounts to clients).  Both of the servers have mounted
"/mnt/glusterfs" using GlusterFS with the client pointing to both
servers.  I then export the filesyste with NFS.  I grabbed a quick perl
program that tries to lock a file for testing, which fails only on the
glusterfs.  When I export a normal directory "/mnt/test" the locking
works.
	Any ideas appreciated.  I have a feeling I've implemented the
posix/locks option incorrectly.

Both servers are running Ubuntu with identical setups, below are
relevant configs.
root at gluster01:/mnt/glusterfs# uname -a
Linux gluster01 2.6.31-20-generic-pae #58-Ubuntu SMP Fri Mar 12 06:25:51
UTC 2010 i686 GNU/Linux

root at gluster01:/mnt/glusterfs# cat /etc/exports
/mnt/glusterfs  <ip removed for
security>/25(rw,no_root_squash,no_all_squash,no_subtree_check,sync,insec
ure,fsid=10)
/mnt/test       <ip removed for
security>/25(rw,no_root_squash,no_all_squash,no_subtree_check,sync,insec
ure,fsid=11)

* I've tried async, rsync, removing all options except FSID.

root at gluster02:/etc/glusterfs# cat glusterfs.vol 
volume brick1
 type protocol/client
 option transport-type tcp/client
 option remote-host <ip removed for security> # IP address of the remote
brick
 option remote-subvolume brick        # name of the remote volume
end-volume
 
volume brick2
 type protocol/client
 option transport-type tcp/client
 option remote-host <ip removed for security>      # IP address of the
remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume
 
volume afr1
 type cluster/afr
 subvolumes brick1 brick2
end-volume
 
volume writebehind
  type performance/write-behind
  option window-size 4MB
  subvolumes afr1
end-volume

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind
end-volume

volume readahead
   type performance/read-ahead
   option page-size 128KB     # unit in bytes
   subvolumes cache
end-volume

volume iothreads
   type performance/io-threads
   option thread-count 4
   option cache-size 64MB
   subvolumes readahead
end-volume

root at gluster02:/etc/glusterfs# cat glusterfsd.vol 
volume posix
  type storage/posix
  option directory /data/export
end-volume
 
volume locks
  type features/posix-locks
  option manditory on  # tried with and without this, found in a search
of earlier post
  subvolumes posix
end-volume
 
volume brick
  type performance/io-threads
  option thread-count 8
  subvolumes locks
end-volume
 
volume server
  type protocol/server
  option transport-type tcp
  option auth.addr.brick.allow *
  option auth.addr.brick-ns.allow *
  option transport.socket.nodelay on
  option auth.ip.locks.allow *
  subvolumes brick
end-volume


* file to test locking...
root at gluster02:/mnt/glusterfs# cat locktest.pl 
#!/usr/bin/perl

use Fcntl qw(:flock);

my $lock_file = 'lockfile';

open(LOCKFILE,">>$lock_file") or die "Cannot open
$lock_file: $!\n";
print "Opened file $lock_file\n";
flock(LOCKFILE, LOCK_SH) or die "Can't get shared lock on $lock_file:
$!\n";
print "Got shared lock on file $lock_file\n";
sleep 2;
close LOCKFILE;
print "Closed file $lock_file\n";

exit;

*Test run from gluster02 using normal NFS mount:
root at gluster02:/# mount <ip removed for security>:/mnt/test /mnt/test
root at gluster02:/# cd /mnt/test
root at gluster02:/mnt/test# ./locktest.pl 
Opened file lockfile
Got shared lock on file lockfile
Closed file lockfile

*Test run from gluster02 using gluster exported NFS mount:
root at gluster02:/# mount 74.81.128.17:/mnt/glusterfs /mnt/test
root at gluster02:/# cd /mnt/test
root at gluster02:/mnt/test# ./locktest.pl 
Opened file lockfile
Can't get shared lock on lockfile:
No locks available

--
Ian Steinmetz
Comcast Engineering - Houston
713-375-7866

Steinmetz, Ian

2010-Apr-29 15:47 UTC

head link

[Gluster-users] NFS file locks

Forgot to mention gluster version, which I compiled from source:

root at gluster02:~# glusterfs -V
glusterfs 3.0.4 built on Apr 23 2010 12:53:11
Repository revision: v3.0.4
Copyright (c) 2006-2009 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU
General Public License.

--
Ian Steinmetz
Comcast Engineering - Houston
713-375-7866

Shehjar Tikoo

2010-May-03 04:21 UTC

head link

[Gluster-users] NFS file locks

There are a couple of things we can do:

-> Mail us the Glusterfs log files from the NFS server and the glusterfs 
servers when the lock script fails. Do file a bug if you can.

-> On the NFS client machine, before you run the mount command, make 
sure you run the following command.

	$ rpc.statd

-> Run the same perl script but this time at the nfs server over the 
glusterfs mount point, not at the NFS client. If it runs fine, it is 
probably related to locking over NFS and we'll look at other places to 
figure it out.

-Shehjar



Steinmetz, Ian wrote:> 	I'm seeing an issue where I can't lock files on a NFS exported
> GlusterFS mount.  I have two servers connected to each other doing AFR
> to provide a high available NFS server (mirror the content, one VIP for
> NFS mounts to clients).  Both of the servers have mounted
> "/mnt/glusterfs" using GlusterFS with the client pointing to both
> servers.  I then export the filesyste with NFS.  I grabbed a quick perl
> program that tries to lock a file for testing, which fails only on the
> glusterfs.  When I export a normal directory "/mnt/test" the
locking
> works.
> 	Any ideas appreciated.  I have a feeling I've implemented the
> posix/locks option incorrectly.
> 
> Both servers are running Ubuntu with identical setups, below are
> relevant configs.
> root at gluster01:/mnt/glusterfs# uname -a
> Linux gluster01 2.6.31-20-generic-pae #58-Ubuntu SMP Fri Mar 12 06:25:51
> UTC 2010 i686 GNU/Linux
> 
> root at gluster01:/mnt/glusterfs# cat /etc/exports
> /mnt/glusterfs  <ip removed for
> security>/25(rw,no_root_squash,no_all_squash,no_subtree_check,sync,insec
> ure,fsid=10)
> /mnt/test       <ip removed for
> security>/25(rw,no_root_squash,no_all_squash,no_subtree_check,sync,insec
> ure,fsid=11)
> 
> * I've tried async, rsync, removing all options except FSID.
> 
> root at gluster02:/etc/glusterfs# cat glusterfs.vol 
> volume brick1
>  type protocol/client
>  option transport-type tcp/client
>  option remote-host <ip removed for security> # IP address of the
remote
> brick
>  option remote-subvolume brick        # name of the remote volume
> end-volume
>  
> volume brick2
>  type protocol/client
>  option transport-type tcp/client
>  option remote-host <ip removed for security>      # IP address of
the
> remote brick
>  option remote-subvolume brick        # name of the remote volume
> end-volume
>  
> volume afr1
>  type cluster/afr
>  subvolumes brick1 brick2
> end-volume
>  
> volume writebehind
>   type performance/write-behind
>   option window-size 4MB
>   subvolumes afr1
> end-volume
> 
> volume cache
>   type performance/io-cache
>   option cache-size 512MB
>   subvolumes writebehind
> end-volume
> 
> volume readahead
>    type performance/read-ahead
>    option page-size 128KB     # unit in bytes
>    subvolumes cache
> end-volume
> 
> volume iothreads
>    type performance/io-threads
>    option thread-count 4
>    option cache-size 64MB
>    subvolumes readahead
> end-volume
> 
> root at gluster02:/etc/glusterfs# cat glusterfsd.vol 
> volume posix
>   type storage/posix
>   option directory /data/export
> end-volume
>  
> volume locks
>   type features/posix-locks
>   option manditory on  # tried with and without this, found in a search
> of earlier post
>   subvolumes posix
> end-volume
>  
> volume brick
>   type performance/io-threads
>   option thread-count 8
>   subvolumes locks
> end-volume
>  
> volume server
>   type protocol/server
>   option transport-type tcp
>   option auth.addr.brick.allow *
>   option auth.addr.brick-ns.allow *
>   option transport.socket.nodelay on
>   option auth.ip.locks.allow *
>   subvolumes brick
> end-volume
> 
> 
> * file to test locking...
> root at gluster02:/mnt/glusterfs# cat locktest.pl 
> #!/usr/bin/perl
> 
> use Fcntl qw(:flock);
> 
> my $lock_file = 'lockfile';
> 
> open(LOCKFILE,">>$lock_file") or die "Cannot open
$lock_file: $!\n";
> print "Opened file $lock_file\n";
> flock(LOCKFILE, LOCK_SH) or die "Can't get shared lock on
$lock_file:
> $!\n";
> print "Got shared lock on file $lock_file\n";
> sleep 2;
> close LOCKFILE;
> print "Closed file $lock_file\n";
> 
> exit;
> 
> *Test run from gluster02 using normal NFS mount:
> root at gluster02:/# mount <ip removed for security>:/mnt/test
/mnt/test
> root at gluster02:/# cd /mnt/test
> root at gluster02:/mnt/test# ./locktest.pl 
> Opened file lockfile
> Got shared lock on file lockfile
> Closed file lockfile
> 
> *Test run from gluster02 using gluster exported NFS mount:
> root at gluster02:/# mount 74.81.128.17:/mnt/glusterfs /mnt/test
> root at gluster02:/# cd /mnt/test
> root at gluster02:/mnt/test# ./locktest.pl 
> Opened file lockfile
> Can't get shared lock on lockfile:
> No locks available
> 
> --
> Ian Steinmetz
> Comcast Engineering - Houston
> 713-375-7866
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Pavan Sondur

2010-May-13 10:15 UTC

head link

[Gluster-users] NFS file locks

Ian,
Can you send me the whole log file? The trace output captures all locks
granted/blocked etc. Just wanted to analyze the log from the start of the lock
testing session (when you execute locktest.pl) to when the problem is seen.

Pavan
----- Original Message -----
From: "Ian Steinmetz" <Ian_Steinmetz at cable.comcast.com>
To: "Pavan Sondur" <pavan at gluster.com>
Cc: gluster-users at gluster.org, "Shehjar Tikoo" <shehjart at
gluster.com>
Sent: Wednesday, May 12, 2010 10:57:14 PM
Subject: RE: [Gluster-users] NFS file locks

Pavan,

I turned on logging, and ran the perl lockd script again.  Below is the output
during the file locking test.  If I remount the share it reproduces the 1st two
errors over again exactly the same but with different timestamps.

[2010-05-12 12:23:11] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=1139, fd=0xb5500770, path=/locktest.pl}
[2010-05-12 12:23:12] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:23:17] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:23:22] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:23:27] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:23:32] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:23:37] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:23:42] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:23:47] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:23:52] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:26:39] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=1139, fd=0xb5500770, path=/locktest.pl}
[2010-05-12 12:26:39] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}
[2010-05-12 12:27:21] N [common.c:375:pl_trace_release] locks: [RELEASE] Lockee
= {ino=76, fd=0xb5500770, path=/lockfile}

--
Ian Steinmetz

-----Original Message-----
From: Pavan Sondur [mailto:pavan at gluster.com] 
Sent: Wednesday, May 12, 2010 1:53 AM
To: Steinmetz, Ian
Cc: gluster-users at gluster.org; Shehjar Tikoo
Subject: Re: [Gluster-users] NFS file locks

Ian,
Mandatory locking is not supported in 3.0.x releases. Even if you turn this
option on, it does not
in any way change any locking behavior in GlusterFS. You can perhaps try this,
specifically for locking:

 Turn on an option in the locks translator - 'option trace on'. It
prints debug information about locks which is useful for debugging.

Send us the output and we'll see if a bug needs to be filed. Nevertheless,
we'll try and reproduce it in-house.

Pavan

----- Original Message -----
From: "Ian Steinmetz" <Ian_Steinmetz at cable.comcast.com>
To: "Shehjar Tikoo" <shehjart at gluster.com>
Cc: gluster-users at gluster.org
Sent: Wednesday, May 5, 2010 12:04:52 AM
Subject: Re: [Gluster-users] NFS file locks

Shehjar,

Good catch on the typo, I fixed it an restarted.  Still had the same
issue.  I also was able to get samba working, and file locking seems to
work fine; however, with samba the file owner/group and priv aren't
transferred to clients.  This at least points to an issue with how
NFS/FUSE/GLUSTER are working together.

I'm also searching to see if I can find any other FUSE file systems that
have had similar issues.

--
Ian Steinmetz

-----Original Message-----
From: Shehjar Tikoo [mailto:shehjart at gluster.com] 
Sent: Tuesday, May 04, 2010 1:40 AM
To: Steinmetz, Ian
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] NFS file locks

Steinmetz, Ian wrote:> I've turned on debug logging for the server and client of GlusterFS
and> amended them below.   I've replaced the IP addresses with
"x.x.x" and
> left the last octet for security.  It appears I'm able to lock the
file> when I run the program directly on the gluster mount point, just not
> when it's mounted via NFS.  I checked and rpc.statd is running.  One
odd> thing, when I run the perl locking program  directly on the mnt point,
> it appears to work but spits out the following log message:
> 
> [2010-05-03 09:16:39] D [read-ahead.c:468:ra_readv] readahead:
> unexpected offset (4096 != 362) resetting
This is nothing to worry about..just debugging output.

.....
.....
.....
> 
> [glusterfsd server logfile during above testing]
> 
> root at gluster02:/var/log/glusterfs# /usr/sbin/glusterfsd -p
> /var/run/glusterfsd.pid -f /etc/glusterfs/glusterfsd.vol --log-file
> /var/log/glusterfsd.log --debug -N
.....
.....
.....
> Given volfile:
>
+-----------------------------------------------------------------------> -------+
>   1: volume posix
>   2:   type storage/posix
>   3:   option directory /data/export
>   4: end-volume
>   5:  
>   6: volume locks
>   7:   type features/locks
>   8:   option manditory on	
I'll continue looking into this but in the mean time you could test over

NFS again with "manditory" changed to "mandatory". Let me
know if that
works.

Thanks
-Shehjar
>   9:   subvolumes posix
>  10: end-volume_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Gluster users - Apr 2010 - NFS file locks

[Gluster-users] NFS file locks

[Gluster-users] NFS file locks

[Gluster-users] NFS file locks

[Gluster-users] NFS file locks