Gluster users - May 2013 - State of Gluster 3.4 alpha 3 NFS ACL support

Wonder if someone can point me in the right direction here - just
built some new bricks running gluster 3.4 alpha 3 (from
http://download.gluster.org) with the objective of testing them to
replace some existing 3.2 systems in production.

The main thing I was after was NFS ACL support, however on getting
everything setup, I still don't seem to be able to see POSIX extended
ACLs on the filesystem when mounted via NFS, they as ever show up fine
when mounted via the gluster fuse client.

I know NFS ACL support is on the feature list for 3.4, I'm wondering
if it's not yet included in the alphas?

Thanks!

James

Basic test output below:

[jim at puppet testdir]$ sudo mount -t nfs -o nfsvers=3,acl
webcontent:/ssl /tmp/nfs/
[jim at puppet testdir]$ sudo mount -t glusterfs -o acl webcontent:/ssl
/tmp/gluster/

[jim at puppet testdir]$ ls -la /tmp/nfs/testdir/
total 8
drwxrwxr-x  2 root root 4096 May  4 00:50 .
drwxr-xr-x 10 root root 4096 May  4 00:49 ..
-rw-rwxr--  1 root root    0 May  4 00:50 file1
-rw-r--r--  1 root root    0 May  4 00:50 file2
-rw-r--r--  1 root root    0 May  4 00:50 file3

[jim at puppet testdir]$ ls -la /tmp/gluster/testdir/
total 8
drwxrwxr-x+  2 root root 4096 May  4 00:50 .
drwxr-xr-x  10 root root 4096 May  4 00:49 ..
-rw-rwxr--+  1 root root    0 May  4 00:50 file1
-rw-r--r--   1 root root    0 May  4 00:50 file2
-rw-r--r--   1 root root    0 May  4 00:50 file3

[jim at puppet testdir]$ getfacl /tmp/nfs/testdir/file1
getfacl: /tmp/nfs/testdir/file1: Invalid argument

[jim at puppet testdir]$ getfacl /tmp/gluster/testdir/file1
getfacl: Removing leading '/' from absolute path names
# file: tmp/gluster/testdir/file1
# owner: root
# group: root
user::rw-
group::r--
group:apache:rwx
mask::rwx
other::r--

On Sat, May 4, 2013 at 12:54 AM, James Yale <james.yale at pun.net>
wrote:
> Wonder if someone can point me in the right direction here - just
> built some new bricks running gluster 3.4 alpha 3 (from
> http://download.gluster.org) with the objective of testing them to
> replace some existing 3.2 systems in production.
>
> The main thing I was after was NFS ACL support, however on getting
> everything setup, I still don't seem to be able to see POSIX extended
> ACLs on the filesystem when mounted via NFS, they as ever show up fine
> when mounted via the gluster fuse client.
>
> I know NFS ACL support is on the feature list for 3.4, I'm wondering
> if it's not yet included in the alphas?
>
> Thanks!
>
> James
>
> Basic test output below:
>
> [jim at puppet testdir]$ sudo mount -t nfs -o nfsvers=3,acl
> webcontent:/ssl /tmp/nfs/
> [jim at puppet testdir]$ sudo mount -t glusterfs -o acl webcontent:/ssl
> /tmp/gluster/
>
> [jim at puppet testdir]$ ls -la /tmp/nfs/testdir/
> total 8
> drwxrwxr-x  2 root root 4096 May  4 00:50 .
> drwxr-xr-x 10 root root 4096 May  4 00:49 ..
> -rw-rwxr--  1 root root    0 May  4 00:50 file1
> -rw-r--r--  1 root root    0 May  4 00:50 file2
> -rw-r--r--  1 root root    0 May  4 00:50 file3
>
> [jim at puppet testdir]$ ls -la /tmp/gluster/testdir/
> total 8
> drwxrwxr-x+  2 root root 4096 May  4 00:50 .
> drwxr-xr-x  10 root root 4096 May  4 00:49 ..
> -rw-rwxr--+  1 root root    0 May  4 00:50 file1
> -rw-r--r--   1 root root    0 May  4 00:50 file2
> -rw-r--r--   1 root root    0 May  4 00:50 file3
>
> [jim at puppet testdir]$ getfacl /tmp/nfs/testdir/file1
> getfacl: /tmp/nfs/testdir/file1: Invalid argument
>
> [jim at puppet testdir]$ getfacl /tmp/gluster/testdir/file1
> getfacl: Removing leading '/' from absolute path names
> # file: tmp/gluster/testdir/file1
> # owner: root
> # group: root
> user::rw-
> group::r--
> group:apache:rwx
> mask::rwx
> other::r--
To expand on this, I've done some further testing - quite strange results.

I brought up a new pair of bricks running 3.4 alpha 3, peer probed,
created (replica) volumes and rsync'd my data on - I was worried that
last time I might have accdentially rsync'd the .glusterfs directory
so made sure to avoid it this time.

I'm getting the same error as before, but I've noticed a further
oddity, existing directories produce the invalid argument error when
access with getfacl:

[jim at web02 ~]$ getfacl /mnt/webcontent/sites/
getfacl: /mnt/webcontent/sites/: Invalid argument

However creating a new directory allows getfacl and setfacl to be used
to view and set ACLs, but the changes then are only visible to the nfs
client, and aren't visible on the server, or to clients mounting the
file system over the fuse client. Please see below for details:

[jim at web02 ~]$ sudo mkdir  /mnt/webcontent/test

[jim at web02 ~]$ ls -la /mnt/webcontent/
total 24
drwxr-xr-x  7 root root 4096 May  7 17:18 .
drwxr-xr-x. 5 root root 4096 Sep  3  2012 ..
drwxrwx---  3 jim  jim  4096 Feb 22  2012 adminserver
drwxrwx---  7 root root 4096 Mar  7 14:02 sites
drwxr-xr-x  2 root root 4096 May  7 17:18 test
drwxrwx---  4 jim  jim  4096 Feb 28  2012 webserver

[jim at web02 ~]$ getfacl /mnt/webcontent/test/
getfacl: Removing leading '/' from absolute path names
# file: mnt/webcontent/test/
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

[jim at web02 ~]$ sudo setfacl -R -m g:apache:rwx /mnt/webcontent/test/

[jim at web02 ~]$ getfacl /mnt/webcontent/test/
getfacl: Removing leading '/' from absolute path names
# file: mnt/webcontent/test/
# owner: root
# group: root
user::rwx
group::r-x
group:apache:rwx
mask::rwx
other::r-x

[jim at web02 ~]$ ls -la /mnt/webcontent/
total 24
drwxr-xr-x  7 root root 4096 May  7 17:18 .
drwxr-xr-x. 5 root root 4096 Sep  3  2012 ..
drwxrwx---  3 jim  jim  4096 Feb 22  2012 adminserver
drwxrwx---  7 root root 4096 Mar  7 14:02 sites
drwxr-xr-x+ 2 root root 4096 May  7 17:18 test
drwxrwx---  4 jim  jim  4096 Feb 28  2012 webserver

Meanwhile on the server:

[jim at webcontent08 ~]$ ls -la /exports/webcontent/
total 44
drwxr-xr-x    7 root root 4096 May  7 17:18 .
drwxr-xr-x    6 root root 4096 May  7 16:37 ..
drwxrwx---+   3 jim  jim  4096 Feb 22  2012 adminserver
drw-------  260 root root 4096 May  7 16:43 .glusterfs
drwxrwx---+   7 root root 4096 Mar  7 14:02 sites
drwxr-xr-x    2 root root 4096 May  7 17:18 test
drwxrwx---+   4 jim  jim  4096 Feb 28  2012 webserver

And a gluster FUSE client:

[jim at webcontent07 tmp]$ ls -la webcontent/
total 24
drwxr-xr-x  7 root root 4096 May  7 17:18 .
drwxrwxrwt. 8 root root 4096 May  7 17:21 ..
drwxrwx---+ 3 jim  jim  4096 Feb 22  2012 adminserver
drwxrwx---+ 7 root root 4096 Mar  7 14:02 sites
drwxr-xr-x  2 root root 4096 May  7 17:18 test
drwxrwx---+ 4 jim  jim  4096 Feb 28  2012 webserver


Any suggestions on what to try next would be appreciated, thanks!