The issue here is the way we fetch ACLs on a directory.
If 'directory', we first fetch DEFAULT_ACL and then fetch ACCESS_ACL.
But there seems like a bug in the code. If there is no default ACL set
on a directory, we seem to be bailing out and skip fetching ACCESS_ACL.
Here is the code snippet -->
324 /* acl3_default_getacl_cbk: fetch and decode the ACL set in the
325 * POSIX_ACL_DEFAULT_XATTR xattr.
326 *
327 * The POSIX_ACL_DEFAULT_XATTR xattr is only set on directories, not
on files.
328 *
329 * When done with POSIX_ACL_DEFAULT_XATTR, we also need to get and
decode the
330 * ACL that can be set in POSIX_ACL_DEFAULT_XATTR.
331 */
332 int
333 acl3_default_getacl_cbk (call_frame_t *frame, void *cookie, xlator_t
*this,
334 int32_t op_ret, int32_t op_errno, dict_t *dict,
335 dict_t *xdata)
336 {
........
.......
........
353 if ((op_ret < 0) && (op_errno != ENODATA &&
op_errno !=
ENOATTR)) {
354 stat = nfs3_cbk_errno_status (op_ret, op_errno);
355 goto err;
356 } else if (!dict) {
357 /* no ACL has been set */
358 stat = NFS3_OK;
359 goto err;
360 }
......
......
......
385 ret = nfs_getxattr (cs->nfsx, cs->vol, &nfu,
&cs->resolvedloc,
386 POSIX_ACL_ACCESS_XATTR, NULL,
acl3_getacl_cbk, cs);
387 if (ret < 0) {
388 stat = nfs3_errno_to_nfsstat3 (-ret);
389 goto err;
390 }
391
392 return 0;
393
394 err:
395 if (getaclreply)
396 getaclreply->status = stat;
397 acl3_getacl_reply (cs->req, getaclreply);
398 nfs3_call_state_wipe (cs);
399 return 0;
400 }
<----
To verify this, along with ACCESS ACL, please set an inherit/default ACL
on that directory and check getfacl output.
Thanks,
Soumya
On 07/31/2015 05:51 PM, Soumya Koduri wrote:>
>
> On 07/31/2015 05:33 PM, J?ri Palis wrote:
>> Playing around with my GlusterFS test setup I discovered following
>> anomaly
>>
>> On volume with low access traffic, ACLs on directories (managed over
>> NFS) sort of work, I can add new ACL entry to directory and when I
>> execute getfacl right after setting ACL, it displays correct settings.
>> However this data is never replicated to another GlusterFS node
>> hosting this particular volume and ACL disappears few minutes after
>> setting it.
>>
>> Same operation when performed on file works correctly, ACL entry set
>> to particular file on one GlusterFS NFS server is replicated to
>> participating node almost immediately.
>>
>> So, it would be interesting to see if someone here can replicate this
>> anomaly.
>>
>> J.
>
> I could reproduce this similar issue - After remounting the volume, the
> directory ACLs are not displayed. I shall look further into this and
> update the findings.
>
> # getfacl /mnt/dir5
> getfacl: Removing leading '/' from absolute path names
> # file: mnt/dir5
> # owner: root
> # group: root
> user::rwx
> group::r-x
> group:tmpgroup:r-x
> mask::r-x
> other::r-x
> #
> # umount /mnt
> # mount -t nfs 10.70.xx.xx:/vol0 /mnt
> # getfacl /mnt/dir5
> getfacl: Removing leading '/' from absolute path names
> # file: mnt/dir5
> # owner: root
> # group: root
> user::rwx
> group::r-x
> other::r-x
> #
>
> Though these ACLs are displayed when done getfacl using brick-path
> directly.
>
> Thanks,
> Soumya
>>
>> On 31 Jul 2015, at 09:35, Soumya Koduri <skoduri at redhat.com>
wrote:
>>
>>> I have tested it using the gluster-NFS server with GlusterFS
version
>>> 3.7.* running on a RHEL7 machine and RHEL 6.7 as NFS client. ACLs
>>> with named groups got properly set on the directory.
>>>
>>> Could you please provide us the packet trace (better taken on the
>>> server side so that we can check Gluster operations too) while
doing
>>> setfacl and getfacl ?
>>>
>>> Thanks,
>>> Soumya
>>>
>>> On 07/30/2015 07:38 PM, J?ri Palis wrote:
>>>> Hi,
>>>>
>>>> Mounted GlusterFS volume with native mount and ACL?s are
working as
>>>> expected, mounted same volume with nfs protocol and the result
is
>>>> exactly the same as I described below. ACL set to files work
and ACL
>>>> set
>>>> to directory do not work as expected. Ohh, I?m out of ideas :(
>>>>
>>>> J.
>>>> On 30 Jul 2015, at 16:38, J?ri Palis <jyri.palis at
gmail.com
>>>> <mailto:jyri.palis at gmail.com>> wrote:
>>>>
>>>>>
>>>>> [2015-07-30 13:16:01.002296] T
[rpcsvc.c:316:rpcsvc_program_actor]
>>>>> 0-rpc-service: Actor found: ACL3 - SETACL for 10.1.1.32:742
>>>>> [2015-07-30 13:16:01.002325] T [MSGID: 0]
[acl3.c:672:acl3svc_setacl]
>>>>> 0-nfs-ACL: FH to Volume: acltest
>>>>> [2015-07-30 13:16:01.004287] T
[rpcsvc.c:1319:rpcsvc_submit_generic]
>>>>> 0-rpc-service: submitted reply for rpc-message (XID:
0x16185ddc,
>>>>> Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport
>>>>> (socket.nfs-server)
>>>>> [2015-07-30 13:16:22.823894] T
[rpcsvc.c:316:rpcsvc_program_actor]
>>>>> 0-rpc-service: Actor found: ACL3 - GETACL for 10.1.1.32:742
>>>>> [2015-07-30 13:16:22.823900] T [MSGID: 0]
[acl3.c:532:acl3svc_getacl]
>>>>> 0-nfs-ACL: FH to Volume: acltest
>>>>> [2015-07-30 13:16:22.824218] D [MSGID: 0]
>>>>> [client-rpc-fops.c:1156:client3_3_getxattr_cbk]
0-acltest-client-1:
>>>>> remote operation failed: No data available. Path:
>>>>> <gfid:12f02b4f-a181-47d4-9b5b-69e889483570>
>>>>> (12f02b4f-a181-47d4-9b5b-69e889483570). Key:
system.posix_acl_default
>>>>> [2015-07-30 13:16:22.825675] D [MSGID: 0]
>>>>> [client-rpc-fops.c:1156:client3_3_getxattr_cbk]
0-acltest-client-0:
>>>>> remote operation failed: No data available. Path:
>>>>> <gfid:12f02b4f-a181-47d4-9b5b-69e889483570>
>>>>> (12f02b4f-a181-47d4-9b5b-69e889483570). Key:
system.posix_acl_default
>>>>> [2015-07-30 13:16:22.825713] T
[rpcsvc.c:1319:rpcsvc_submit_generic]
>>>>> 0-rpc-service: submitted reply for rpc-message (XID:
0x63815edc,
>>>>> Program: ACL3, ProgVers: 3, Proc: 1) to rpc-transport
>>>>> (socket.nfs-server)
>>>>> [2015-07-30 13:16:22.828243] T
[rpcsvc.c:316:rpcsvc_program_actor]
>>>>> 0-rpc-service: Actor found: ACL3 - SETACL for 10.1.1.32:742
>>>>> [2015-07-30 13:16:22.828266] T [MSGID: 0]
[acl3.c:672:acl3svc_setacl]
>>>>> 0-nfs-ACL: FH to Volume: acltest
>>>>> [2015-07-30 13:16:22.829931] T
[rpcsvc.c:1319:rpcsvc_submit_generic]
>>>>> 0-rpc-service: submitted reply for rpc-message (XID:
0x75815edc,
>>>>> Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport
>>>>> (socket.nfs-server)
>>>>>
>>>>> Enabled trace for few moments and tried to make any sense
of it by
>>>>> searching for lines containing ?acl? according to this
everything
>>>>> kind of works except lines which state that ?remote
operation failed?
>>>>> GlusterFS failed to replicate or commit acl changes?
>>>>>
>>>>>>
>>>>>>
>>>>>> On 07/30/2015 06:22 PM, J?ri Palis wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks Niels, your hints about those two options
did the trick
>>>>>>> although
>>>>>>> I had to enable both of them and I had to add nscd
(sssd provides
>>>>>>> user
>>>>>>> identities) to this mix as well.
>>>>>>>
>>>>>>> Now back to the problem with ACL?s. Is your test
setup something
>>>>>>> like
>>>>>>> this: GlusterFS 3.7.2 replicated volume on
Centos/RHEL 7 and
>>>>>>> client or
>>>>>>> clients accessing GlusterFS volumes by NFS
protocol, correct?
>>>>>>>
>>>>>> As Jiffin had suggested, did you try the same command
on GlusterFS
>>>>>> Native mount?
>>>>>>
>>>>>> Log levels can be increased to TRACE/DEBUG mode using
the command
>>>>>> 'gluster vol set <volname>
diagnostics.client-log-level
>>>>>> [TRACE,DEBUG]'
>>>>>>
>>>>>> Also please capture a packet trace on the server-side
using the
>>>>>> command - 'tcpdump -i any -s 0 -w
/var/tmp/nfs-acl.pcap tcp and not
>>>>>> port 22'
>>>>>>
>>>>>> Verify the packets sent by Gluster-NFS process to the
brick process
>>>>>> to set the ACL.
>>>>>>
>>>>>> Thanks,
>>>>>> Soumya
>>>>>>
>>>>>>> # gluster volume info acltest
>>>>>>> Volume Name: acltest
>>>>>>> Type: Replicate
>>>>>>> Volume ID: 9e0de3f5-45ba-4612-a4f1-16bc5d1eb985
>>>>>>> Status: Started
>>>>>>> Number of Bricks: 1 x 2 = 2
>>>>>>> Transport-type: tcp
>>>>>>> Bricks:
>>>>>>> Brick1: vfs-node-01:/data/gfs/acltest/brick0/brick
>>>>>>> Brick2: vfs-node-02:/data/gfs/acltest/brick0/brick
>>>>>>> Options Reconfigured:
>>>>>>> server.manage-gids: on
>>>>>>> nfs.server-aux-gids: on
>>>>>>> performance.readdir-ahead: on
>>>>>>> server.event-threads: 32
>>>>>>> performance.cache-size: 2GB
>>>>>>> storage.linux-aio: on
>>>>>>> nfs.disable: off
>>>>>>> performance.write-behind-window-size: 1GB
>>>>>>> performance.nfs.io-cache: on
>>>>>>> performance.nfs.write-behind-window-size: 250MB
>>>>>>> performance.nfs.stat-prefetch: on
>>>>>>> performance.nfs.read-ahead: on
>>>>>>> performance.nfs.io-threads: on
>>>>>>> cluster.readdir-optimize: on
>>>>>>> network.remote-dio: on
>>>>>>> auth.allow: 10.1.1.32,10.1.1.42
>>>>>>> diagnostics.latency-measurement: on
>>>>>>> diagnostics.count-fop-hits: on
>>>>>>> nfs.rpc-auth-allow: 10.1.1.32,10.1.1.42
>>>>>>> nfs.trusted-sync: on
>>>>>>>
>>>>>>> Maybe there is a way to increase verbosity of nfs
server which could
>>>>>>> help me to trace this problem. I did not find any
good hints for
>>>>>>> increasing verbosity of nfs server in
documentation.
>>>>>>>
>>>>>>> Regards,
>>>>>>> J.
>>>>>>>
>>>>>>> On 30 Jul 2015, at 10:09, Jiffin Tony Thottan
<jthottan at redhat.com
>>>>>>> <mailto:jthottan at redhat.com>
>>>>>>> <mailto:jthottan at redhat.com>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 29/07/15 20:14, Niels de Vos wrote:
>>>>>>>>> On Wed, Jul 29, 2015 at 05:22:31PM +0300,
J?ri Palis wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Another issue with NFS and sec=sys
mode. As we all know there
>>>>>>>>>> is a
>>>>>>>>>> limit of 15 security ids involved when
running NFS in sec=sys
>>>>>>>>>> mode.
>>>>>>>>>> This limit makes effective and granular
usage of ACL assigned
>>>>>>>>>> through
>>>>>>>>>> groups almost unusable. One way to
overcome this limit is to use
>>>>>>>>>> kerberised NFS but GlusterFS does not
natively support this
>>>>>>>>>> access
>>>>>>>>>> mode . Another option, at least
according to one email thread,
>>>>>>>>>> states
>>>>>>>>>> that GlusterFS has an option
server.manage-gids which should
>>>>>>>>>> mitigate
>>>>>>>>>> this limit and raise it to 90
something. Is this the option,
>>>>>>>>>> which
>>>>>>>>>> can be used for increasing sec=sys
limit. Sadly documentation
>>>>>>>>>> does not
>>>>>>>>>> have clear description about this
option, what exactly this
>>>>>>>>>> option
>>>>>>>>>> does and how it should be used.
>>>>>>>>> server.manage-gids is an option to resolve
the groups of a uid
>>>>>>>>> in the
>>>>>>>>> brick process. You probably need to also
use the
>>>>>>>>> nfs.server-aux-gids
>>>>>>>>> option so that the NFS-server resolves the
gids of the uid
>>>>>>>>> accessing the
>>>>>>>>> NFS-server.
>>>>>>>>>
>>>>>>>>> The nfs.server-aux-gids option is used to
overcome the
>>>>>>>>> AUTH_SYS/AUTH_UNIX limit of (I thought 32?)
groups.
>>>>>>>>>
>>>>>>>>> The server.manage-gids option is used to
overcome the GlusterFS
>>>>>>>>> protocol
>>>>>>>>> limit of ~93 groups.
>>>>>>>>>
>>>>>>>>> If your users do not belong to 90+ groups,
you would not need to
>>>>>>>>> set the
>>>>>>>>> server.manage-gids option, and
nfs.server-aux-gids might be
>>>>>>>>> sufficient.
>>>>>>>>>
>>>>>>>>> HTH,
>>>>>>>>> Niels
>>>>>>>>>
>>>>>>>>>> J.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 29 Jul 2015, at 16:16, Jiffin Tony
Thottan
>>>>>>>>>> <jthottan at redhat.com
<mailto:jthottan at redhat.com>
>>>>>>>>>> <mailto:jthottan at
redhat.com>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 29/07/15 18:04, J?ri Palis
wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> setfacl for dir on local
filesystem:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. set acl setfacl -m
g:x_meie_sec-test02:rx test
>>>>>>>>>>>> 2. get acl
>>>>>>>>>>>>
>>>>>>>>>>>> # getfacl test
>>>>>>>>>>>> user::rwx
>>>>>>>>>>>> group::r-x
>>>>>>>>>>>> group:x_meie_sec-test02:r-x
>>>>>>>>>>>> mask::r-x
>>>>>>>>>>>> other::r-x
>>>>>>>>>>>>
>>>>>>>>>>>> setfacl for dir on GlusterFS
volume which is NFS mounted to
>>>>>>>>>>>> client
>>>>>>>>>>>> system
>>>>>>>>>>>>
>>>>>>>>>>>> 1. same command is used for
setting ACE, no error is
>>>>>>>>>>>> returned by
>>>>>>>>>>>> that command
>>>>>>>>>>>> 2. get acl
>>>>>>>>>>>>
>>>>>>>>>>>> #getfacl test
>>>>>>>>>>>> user::rwx
>>>>>>>>>>>> group::r-x
>>>>>>>>>>>> other::---
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If I use ordinary file as a
target on GlusterFS like this
>>>>>>>>>>>>
>>>>>>>>>>>> setfacl -m
g:x_meie_sec-test02:rw dummy
>>>>>>>>>>>>
>>>>>>>>>>>> then ACE entry is set for file
dummy stored on GlusterFS
>>>>>>>>>>>>
>>>>>>>>>>>> # getfacl dummy
>>>>>>>>>>>> user::rw-
>>>>>>>>>>>> group::r--
>>>>>>>>>>>> group:x_meie_sec-test02:rw-
>>>>>>>>>>>> mask::rw-
>>>>>>>>>>>> other::?
>>>>>>>>>>>>
>>>>>>>>>>>> So, as you can see setting ACLs
for files works but does not
>>>>>>>>>>>> work
>>>>>>>>>>>> for directories.
>>>>>>>>>>>>
>>>>>>>>>>>> This all is happening on
CentOS7, running GlusterFS 3.7.2
>>>>>>>>>>> Hi Jyri,
>>>>>>>>>>>
>>>>>>>>>>> It seems there are couple of issues
,
>>>>>>>>>>>
>>>>>>>>>>> 1.) when u set a named group acl
for file/directory, it
>>>>>>>>>>> clears the
>>>>>>>>>>> permission of others too.
>>>>>>>>>>> 2.) named group acl is not working
properly for directories ,
>>>>>>>>>>>
>>>>>>>>>>> I will try the same on my setup and
share my findings.
>>>>>>>>>>> --
>>>>>>>>>>> Jiffin
>>>>>>>>
>>>>>>>> In my setup (glusterfs 3.7.2 and RHEL 7.1
client) it worked
>>>>>>>> properly
>>>>>>>>
>>>>>>>> I followed the same steps mentioned by you.
>>>>>>>> #cd /mnt
>>>>>>>> # mkdir dir
>>>>>>>> # touch file
>>>>>>>> # getfacl file
>>>>>>>> # file: file
>>>>>>>> # owner: root
>>>>>>>> # group: root
>>>>>>>> user::rw-
>>>>>>>> group::r--
>>>>>>>> other::r--
>>>>>>>>
>>>>>>>> # getfacl dir
>>>>>>>> # file: dir
>>>>>>>> # owner: root
>>>>>>>> # group: root
>>>>>>>> user::rwx
>>>>>>>> group::r-x
>>>>>>>> other::r-x
>>>>>>>>
>>>>>>>> # setfacl -m g:gluster:rw file
>>>>>>>> # getfacl file
>>>>>>>> # file: file
>>>>>>>> # owner: root
>>>>>>>> # group: root
>>>>>>>> user::rw-
>>>>>>>> group::r--
>>>>>>>> group:gluster:rw-
>>>>>>>> mask::rw-
>>>>>>>> other::r--
>>>>>>>>
>>>>>>>> setfacl -m g:gluster:r-x dir
>>>>>>>> getfacl dir
>>>>>>>> # file: dir
>>>>>>>> # owner: root
>>>>>>>> # group: root
>>>>>>>> user::rwx
>>>>>>>> group::r-x
>>>>>>>> group:gluster:r-x
>>>>>>>> mask::r-x
>>>>>>>> other::r-x
>>>>>>>>
>>>>>>>>
>>>>>>>> So can u share the following information from
the server.
>>>>>>>> 1.) gluster vol info
>>>>>>>> 2.) nfs.log (nfs-server log)
>>>>>>>> 3.) brick logs
>>>>>>>>
>>>>>>>> and also can u try the same on fuse
mount(gluster native mount).
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jiffin
>>>>>>>>
>>>>>>>>>>>> J.
>>>>>>>>>>>> On 29 Jul 2015, at 15:16,
Jiffin Thottan <jthottan at redhat.com
>>>>>>>>>>>> <mailto:jthottan at
redhat.com>
>>>>>>>>>>>> <mailto:jthottan at
redhat.com>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Original Message
-----
>>>>>>>>>>>>> From: "J?ri
Palis" <jyri.palis at gmail.com
>>>>>>>>>>>>> <mailto:jyri.palis at
gmail.com>
>>>>>>>>>>>>> <mailto:jyri.palis at
gmail.com>>
>>>>>>>>>>>>> To:gluster-users at
gluster.org
>>>>>>>>>>>>> <mailto:gluster-users at
gluster.org><mailto:gluster-users at gluster.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sent: Wednesday, July 29,
2015 4:19:20 PM
>>>>>>>>>>>>> Subject: [Gluster-users]
GlusterFS 3.7.2 and ACL
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> Setup:
>>>>>>>>>>>>> GFS 3.7.2, NFS is used for
host access
>>>>>>>>>>>>>
>>>>>>>>>>>>> Problem:
>>>>>>>>>>>>> POSIX ACL work correctly
when ACLs are applied to files but do
>>>>>>>>>>>>> not work when ACLs are
applied to directories on GFS volumes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I debug this issue
more deeply?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can you please explain the
issue with more details, i.e what
>>>>>>>>>>>>> exactly not working
properly , is it setting acl or any
>>>>>>>>>>>>> functionality issue, in
which client?
>>>>>>>>>>>>> __
>>>>>>>>>>>>> Jiffin
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Jyri
>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>>> Gluster-users at
gluster.org
>>>>>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>>>>
>>>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>>>
>>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>
_______________________________________________
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>>
>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>
_______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>
>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>
>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
<mailto:Gluster-users at gluster.org>
>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users