Yes, that?s the case here. ACLs do not ?stick? to directories on NFS mounts when
default ACL is not in place. Independent (user, group, etc) ACL entires can be
set and displayed correctly right after setting default ACL.
BTW. I discovered another side effect, when default ACL is set you can?t remove
this ACL from directory on NFS mount (setfacl -k dir) but it woks correctly when
native mount is used.
Regards,
J.
On 31 Jul 2015, at 15:47, Soumya Koduri <skoduri at redhat.com> wrote:
> The issue here is the way we fetch ACLs on a directory.
>
> If 'directory', we first fetch DEFAULT_ACL and then fetch
ACCESS_ACL. But there seems like a bug in the code. If there is no default ACL
set on a directory, we seem to be bailing out and skip fetching ACCESS_ACL.
>
>
> Here is the code snippet -->
>
> 324 /* acl3_default_getacl_cbk: fetch and decode the ACL set in the
> 325 * POSIX_ACL_DEFAULT_XATTR xattr.
> 326 *
> 327 * The POSIX_ACL_DEFAULT_XATTR xattr is only set on directories, not on
files.
> 328 *
> 329 * When done with POSIX_ACL_DEFAULT_XATTR, we also need to get and
decode the
> 330 * ACL that can be set in POSIX_ACL_DEFAULT_XATTR.
> 331 */
> 332 int
> 333 acl3_default_getacl_cbk (call_frame_t *frame, void *cookie, xlator_t
*this,
> 334 int32_t op_ret, int32_t op_errno, dict_t
*dict,
> 335 dict_t *xdata)
> 336 {
> ........
> .......
> ........
> 353 if ((op_ret < 0) && (op_errno != ENODATA &&
op_errno != ENOATTR)) {
> 354 stat = nfs3_cbk_errno_status (op_ret, op_errno);
> 355 goto err;
> 356 } else if (!dict) {
> 357 /* no ACL has been set */
> 358 stat = NFS3_OK;
> 359 goto err;
> 360 }
> ......
> ......
> ......
> 385 ret = nfs_getxattr (cs->nfsx, cs->vol, &nfu,
&cs->resolvedloc,
> 386 POSIX_ACL_ACCESS_XATTR, NULL,
acl3_getacl_cbk, cs);
> 387 if (ret < 0) {
> 388 stat = nfs3_errno_to_nfsstat3 (-ret);
> 389 goto err;
> 390 }
> 391
> 392 return 0;
> 393
> 394 err:
> 395 if (getaclreply)
> 396 getaclreply->status = stat;
> 397 acl3_getacl_reply (cs->req, getaclreply);
> 398 nfs3_call_state_wipe (cs);
> 399 return 0;
> 400 }
> <----
>
>
> To verify this, along with ACCESS ACL, please set an inherit/default ACL on
that directory and check getfacl output.
>
> Thanks,
> Soumya
>
> On 07/31/2015 05:51 PM, Soumya Koduri wrote:
>>
>>
>> On 07/31/2015 05:33 PM, J?ri Palis wrote:
>>> Playing around with my GlusterFS test setup I discovered following
>>> anomaly
>>>
>>> On volume with low access traffic, ACLs on directories (managed
over
>>> NFS) sort of work, I can add new ACL entry to directory and when I
>>> execute getfacl right after setting ACL, it displays correct
settings.
>>> However this data is never replicated to another GlusterFS node
>>> hosting this particular volume and ACL disappears few minutes after
>>> setting it.
>>>
>>> Same operation when performed on file works correctly, ACL entry
set
>>> to particular file on one GlusterFS NFS server is replicated to
>>> participating node almost immediately.
>>>
>>> So, it would be interesting to see if someone here can replicate
this
>>> anomaly.
>>>
>>> J.
>>
>> I could reproduce this similar issue - After remounting the volume, the
>> directory ACLs are not displayed. I shall look further into this and
>> update the findings.
>>
>> # getfacl /mnt/dir5
>> getfacl: Removing leading '/' from absolute path names
>> # file: mnt/dir5
>> # owner: root
>> # group: root
>> user::rwx
>> group::r-x
>> group:tmpgroup:r-x
>> mask::r-x
>> other::r-x
>> #
>> # umount /mnt
>> # mount -t nfs 10.70.xx.xx:/vol0 /mnt
>> # getfacl /mnt/dir5
>> getfacl: Removing leading '/' from absolute path names
>> # file: mnt/dir5
>> # owner: root
>> # group: root
>> user::rwx
>> group::r-x
>> other::r-x
>> #
>>
>> Though these ACLs are displayed when done getfacl using brick-path
>> directly.
>>
>> Thanks,
>> Soumya
>>>
>>> On 31 Jul 2015, at 09:35, Soumya Koduri <skoduri at
redhat.com> wrote:
>>>
>>>> I have tested it using the gluster-NFS server with GlusterFS
version
>>>> 3.7.* running on a RHEL7 machine and RHEL 6.7 as NFS client.
ACLs
>>>> with named groups got properly set on the directory.
>>>>
>>>> Could you please provide us the packet trace (better taken on
the
>>>> server side so that we can check Gluster operations too) while
doing
>>>> setfacl and getfacl ?
>>>>
>>>> Thanks,
>>>> Soumya
>>>>
>>>> On 07/30/2015 07:38 PM, J?ri Palis wrote:
>>>>> Hi,
>>>>>
>>>>> Mounted GlusterFS volume with native mount and ACL?s are
working as
>>>>> expected, mounted same volume with nfs protocol and the
result is
>>>>> exactly the same as I described below. ACL set to files
work and ACL
>>>>> set
>>>>> to directory do not work as expected. Ohh, I?m out of ideas
:(
>>>>>
>>>>> J.
>>>>> On 30 Jul 2015, at 16:38, J?ri Palis <jyri.palis at
gmail.com
>>>>> <mailto:jyri.palis at gmail.com>> wrote:
>>>>>
>>>>>>
>>>>>> [2015-07-30 13:16:01.002296] T
[rpcsvc.c:316:rpcsvc_program_actor]
>>>>>> 0-rpc-service: Actor found: ACL3 - SETACL for
10.1.1.32:742
>>>>>> [2015-07-30 13:16:01.002325] T [MSGID: 0]
[acl3.c:672:acl3svc_setacl]
>>>>>> 0-nfs-ACL: FH to Volume: acltest
>>>>>> [2015-07-30 13:16:01.004287] T
[rpcsvc.c:1319:rpcsvc_submit_generic]
>>>>>> 0-rpc-service: submitted reply for rpc-message (XID:
0x16185ddc,
>>>>>> Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport
>>>>>> (socket.nfs-server)
>>>>>> [2015-07-30 13:16:22.823894] T
[rpcsvc.c:316:rpcsvc_program_actor]
>>>>>> 0-rpc-service: Actor found: ACL3 - GETACL for
10.1.1.32:742
>>>>>> [2015-07-30 13:16:22.823900] T [MSGID: 0]
[acl3.c:532:acl3svc_getacl]
>>>>>> 0-nfs-ACL: FH to Volume: acltest
>>>>>> [2015-07-30 13:16:22.824218] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:1156:client3_3_getxattr_cbk]
0-acltest-client-1:
>>>>>> remote operation failed: No data available. Path:
>>>>>> <gfid:12f02b4f-a181-47d4-9b5b-69e889483570>
>>>>>> (12f02b4f-a181-47d4-9b5b-69e889483570). Key:
system.posix_acl_default
>>>>>> [2015-07-30 13:16:22.825675] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:1156:client3_3_getxattr_cbk]
0-acltest-client-0:
>>>>>> remote operation failed: No data available. Path:
>>>>>> <gfid:12f02b4f-a181-47d4-9b5b-69e889483570>
>>>>>> (12f02b4f-a181-47d4-9b5b-69e889483570). Key:
system.posix_acl_default
>>>>>> [2015-07-30 13:16:22.825713] T
[rpcsvc.c:1319:rpcsvc_submit_generic]
>>>>>> 0-rpc-service: submitted reply for rpc-message (XID:
0x63815edc,
>>>>>> Program: ACL3, ProgVers: 3, Proc: 1) to rpc-transport
>>>>>> (socket.nfs-server)
>>>>>> [2015-07-30 13:16:22.828243] T
[rpcsvc.c:316:rpcsvc_program_actor]
>>>>>> 0-rpc-service: Actor found: ACL3 - SETACL for
10.1.1.32:742
>>>>>> [2015-07-30 13:16:22.828266] T [MSGID: 0]
[acl3.c:672:acl3svc_setacl]
>>>>>> 0-nfs-ACL: FH to Volume: acltest
>>>>>> [2015-07-30 13:16:22.829931] T
[rpcsvc.c:1319:rpcsvc_submit_generic]
>>>>>> 0-rpc-service: submitted reply for rpc-message (XID:
0x75815edc,
>>>>>> Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport
>>>>>> (socket.nfs-server)
>>>>>>
>>>>>> Enabled trace for few moments and tried to make any
sense of it by
>>>>>> searching for lines containing ?acl? according to this
everything
>>>>>> kind of works except lines which state that ?remote
operation failed?
>>>>>> GlusterFS failed to replicate or commit acl changes?
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 07/30/2015 06:22 PM, J?ri Palis wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Thanks Niels, your hints about those two
options did the trick
>>>>>>>> although
>>>>>>>> I had to enable both of them and I had to add
nscd (sssd provides
>>>>>>>> user
>>>>>>>> identities) to this mix as well.
>>>>>>>>
>>>>>>>> Now back to the problem with ACL?s. Is your
test setup something
>>>>>>>> like
>>>>>>>> this: GlusterFS 3.7.2 replicated volume on
Centos/RHEL 7 and
>>>>>>>> client or
>>>>>>>> clients accessing GlusterFS volumes by NFS
protocol, correct?
>>>>>>>>
>>>>>>> As Jiffin had suggested, did you try the same
command on GlusterFS
>>>>>>> Native mount?
>>>>>>>
>>>>>>> Log levels can be increased to TRACE/DEBUG mode
using the command
>>>>>>> 'gluster vol set <volname>
diagnostics.client-log-level
>>>>>>> [TRACE,DEBUG]'
>>>>>>>
>>>>>>> Also please capture a packet trace on the
server-side using the
>>>>>>> command - 'tcpdump -i any -s 0 -w
/var/tmp/nfs-acl.pcap tcp and not
>>>>>>> port 22'
>>>>>>>
>>>>>>> Verify the packets sent by Gluster-NFS process to
the brick process
>>>>>>> to set the ACL.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Soumya
>>>>>>>
>>>>>>>> # gluster volume info acltest
>>>>>>>> Volume Name: acltest
>>>>>>>> Type: Replicate
>>>>>>>> Volume ID: 9e0de3f5-45ba-4612-a4f1-16bc5d1eb985
>>>>>>>> Status: Started
>>>>>>>> Number of Bricks: 1 x 2 = 2
>>>>>>>> Transport-type: tcp
>>>>>>>> Bricks:
>>>>>>>> Brick1:
vfs-node-01:/data/gfs/acltest/brick0/brick
>>>>>>>> Brick2:
vfs-node-02:/data/gfs/acltest/brick0/brick
>>>>>>>> Options Reconfigured:
>>>>>>>> server.manage-gids: on
>>>>>>>> nfs.server-aux-gids: on
>>>>>>>> performance.readdir-ahead: on
>>>>>>>> server.event-threads: 32
>>>>>>>> performance.cache-size: 2GB
>>>>>>>> storage.linux-aio: on
>>>>>>>> nfs.disable: off
>>>>>>>> performance.write-behind-window-size: 1GB
>>>>>>>> performance.nfs.io-cache: on
>>>>>>>> performance.nfs.write-behind-window-size: 250MB
>>>>>>>> performance.nfs.stat-prefetch: on
>>>>>>>> performance.nfs.read-ahead: on
>>>>>>>> performance.nfs.io-threads: on
>>>>>>>> cluster.readdir-optimize: on
>>>>>>>> network.remote-dio: on
>>>>>>>> auth.allow: 10.1.1.32,10.1.1.42
>>>>>>>> diagnostics.latency-measurement: on
>>>>>>>> diagnostics.count-fop-hits: on
>>>>>>>> nfs.rpc-auth-allow: 10.1.1.32,10.1.1.42
>>>>>>>> nfs.trusted-sync: on
>>>>>>>>
>>>>>>>> Maybe there is a way to increase verbosity of
nfs server which could
>>>>>>>> help me to trace this problem. I did not find
any good hints for
>>>>>>>> increasing verbosity of nfs server in
documentation.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> J.
>>>>>>>>
>>>>>>>> On 30 Jul 2015, at 10:09, Jiffin Tony Thottan
<jthottan at redhat.com
>>>>>>>> <mailto:jthottan at redhat.com>
>>>>>>>> <mailto:jthottan at redhat.com>>
wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 29/07/15 20:14, Niels de Vos wrote:
>>>>>>>>>> On Wed, Jul 29, 2015 at 05:22:31PM
+0300, J?ri Palis wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Another issue with NFS and sec=sys
mode. As we all know there
>>>>>>>>>>> is a
>>>>>>>>>>> limit of 15 security ids involved
when running NFS in sec=sys
>>>>>>>>>>> mode.
>>>>>>>>>>> This limit makes effective and
granular usage of ACL assigned
>>>>>>>>>>> through
>>>>>>>>>>> groups almost unusable. One way to
overcome this limit is to use
>>>>>>>>>>> kerberised NFS but GlusterFS does
not natively support this
>>>>>>>>>>> access
>>>>>>>>>>> mode . Another option, at least
according to one email thread,
>>>>>>>>>>> states
>>>>>>>>>>> that GlusterFS has an option
server.manage-gids which should
>>>>>>>>>>> mitigate
>>>>>>>>>>> this limit and raise it to 90
something. Is this the option,
>>>>>>>>>>> which
>>>>>>>>>>> can be used for increasing sec=sys
limit. Sadly documentation
>>>>>>>>>>> does not
>>>>>>>>>>> have clear description about this
option, what exactly this
>>>>>>>>>>> option
>>>>>>>>>>> does and how it should be used.
>>>>>>>>>> server.manage-gids is an option to
resolve the groups of a uid
>>>>>>>>>> in the
>>>>>>>>>> brick process. You probably need to
also use the
>>>>>>>>>> nfs.server-aux-gids
>>>>>>>>>> option so that the NFS-server resolves
the gids of the uid
>>>>>>>>>> accessing the
>>>>>>>>>> NFS-server.
>>>>>>>>>>
>>>>>>>>>> The nfs.server-aux-gids option is used
to overcome the
>>>>>>>>>> AUTH_SYS/AUTH_UNIX limit of (I thought
32?) groups.
>>>>>>>>>>
>>>>>>>>>> The server.manage-gids option is used
to overcome the GlusterFS
>>>>>>>>>> protocol
>>>>>>>>>> limit of ~93 groups.
>>>>>>>>>>
>>>>>>>>>> If your users do not belong to 90+
groups, you would not need to
>>>>>>>>>> set the
>>>>>>>>>> server.manage-gids option, and
nfs.server-aux-gids might be
>>>>>>>>>> sufficient.
>>>>>>>>>>
>>>>>>>>>> HTH,
>>>>>>>>>> Niels
>>>>>>>>>>
>>>>>>>>>>> J.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 29 Jul 2015, at 16:16, Jiffin
Tony Thottan
>>>>>>>>>>> <jthottan at redhat.com
<mailto:jthottan at redhat.com>
>>>>>>>>>>> <mailto:jthottan at
redhat.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 29/07/15 18:04, J?ri Palis
wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> setfacl for dir on local
filesystem:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. set acl setfacl -m
g:x_meie_sec-test02:rx test
>>>>>>>>>>>>> 2. get acl
>>>>>>>>>>>>>
>>>>>>>>>>>>> # getfacl test
>>>>>>>>>>>>> user::rwx
>>>>>>>>>>>>> group::r-x
>>>>>>>>>>>>> group:x_meie_sec-test02:r-x
>>>>>>>>>>>>> mask::r-x
>>>>>>>>>>>>> other::r-x
>>>>>>>>>>>>>
>>>>>>>>>>>>> setfacl for dir on
GlusterFS volume which is NFS mounted to
>>>>>>>>>>>>> client
>>>>>>>>>>>>> system
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. same command is used for
setting ACE, no error is
>>>>>>>>>>>>> returned by
>>>>>>>>>>>>> that command
>>>>>>>>>>>>> 2. get acl
>>>>>>>>>>>>>
>>>>>>>>>>>>> #getfacl test
>>>>>>>>>>>>> user::rwx
>>>>>>>>>>>>> group::r-x
>>>>>>>>>>>>> other::---
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If I use ordinary file as a
target on GlusterFS like this
>>>>>>>>>>>>>
>>>>>>>>>>>>> setfacl -m
g:x_meie_sec-test02:rw dummy
>>>>>>>>>>>>>
>>>>>>>>>>>>> then ACE entry is set for
file dummy stored on GlusterFS
>>>>>>>>>>>>>
>>>>>>>>>>>>> # getfacl dummy
>>>>>>>>>>>>> user::rw-
>>>>>>>>>>>>> group::r--
>>>>>>>>>>>>> group:x_meie_sec-test02:rw-
>>>>>>>>>>>>> mask::rw-
>>>>>>>>>>>>> other::?
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, as you can see setting
ACLs for files works but does not
>>>>>>>>>>>>> work
>>>>>>>>>>>>> for directories.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This all is happening on
CentOS7, running GlusterFS 3.7.2
>>>>>>>>>>>> Hi Jyri,
>>>>>>>>>>>>
>>>>>>>>>>>> It seems there are couple of
issues ,
>>>>>>>>>>>>
>>>>>>>>>>>> 1.) when u set a named group
acl for file/directory, it
>>>>>>>>>>>> clears the
>>>>>>>>>>>> permission of others too.
>>>>>>>>>>>> 2.) named group acl is not
working properly for directories ,
>>>>>>>>>>>>
>>>>>>>>>>>> I will try the same on my setup
and share my findings.
>>>>>>>>>>>> --
>>>>>>>>>>>> Jiffin
>>>>>>>>>
>>>>>>>>> In my setup (glusterfs 3.7.2 and RHEL 7.1
client) it worked
>>>>>>>>> properly
>>>>>>>>>
>>>>>>>>> I followed the same steps mentioned by you.
>>>>>>>>> #cd /mnt
>>>>>>>>> # mkdir dir
>>>>>>>>> # touch file
>>>>>>>>> # getfacl file
>>>>>>>>> # file: file
>>>>>>>>> # owner: root
>>>>>>>>> # group: root
>>>>>>>>> user::rw-
>>>>>>>>> group::r--
>>>>>>>>> other::r--
>>>>>>>>>
>>>>>>>>> # getfacl dir
>>>>>>>>> # file: dir
>>>>>>>>> # owner: root
>>>>>>>>> # group: root
>>>>>>>>> user::rwx
>>>>>>>>> group::r-x
>>>>>>>>> other::r-x
>>>>>>>>>
>>>>>>>>> # setfacl -m g:gluster:rw file
>>>>>>>>> # getfacl file
>>>>>>>>> # file: file
>>>>>>>>> # owner: root
>>>>>>>>> # group: root
>>>>>>>>> user::rw-
>>>>>>>>> group::r--
>>>>>>>>> group:gluster:rw-
>>>>>>>>> mask::rw-
>>>>>>>>> other::r--
>>>>>>>>>
>>>>>>>>> setfacl -m g:gluster:r-x dir
>>>>>>>>> getfacl dir
>>>>>>>>> # file: dir
>>>>>>>>> # owner: root
>>>>>>>>> # group: root
>>>>>>>>> user::rwx
>>>>>>>>> group::r-x
>>>>>>>>> group:gluster:r-x
>>>>>>>>> mask::r-x
>>>>>>>>> other::r-x
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So can u share the following information
from the server.
>>>>>>>>> 1.) gluster vol info
>>>>>>>>> 2.) nfs.log (nfs-server log)
>>>>>>>>> 3.) brick logs
>>>>>>>>>
>>>>>>>>> and also can u try the same on fuse
mount(gluster native mount).
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jiffin
>>>>>>>>>
>>>>>>>>>>>>> J.
>>>>>>>>>>>>> On 29 Jul 2015, at 15:16,
Jiffin Thottan <jthottan at redhat.com
>>>>>>>>>>>>> <mailto:jthottan at
redhat.com>
>>>>>>>>>>>>> <mailto:jthottan at
redhat.com>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> ----- Original Message
-----
>>>>>>>>>>>>>> From: "J?ri
Palis" <jyri.palis at gmail.com
>>>>>>>>>>>>>> <mailto:jyri.palis
at gmail.com>
>>>>>>>>>>>>>> <mailto:jyri.palis
at gmail.com>>
>>>>>>>>>>>>>> To:gluster-users at
gluster.org
>>>>>>>>>>>>>>
<mailto:gluster-users at gluster.org><mailto:gluster-users at
gluster.org>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sent: Wednesday, July
29, 2015 4:19:20 PM
>>>>>>>>>>>>>> Subject:
[Gluster-users] GlusterFS 3.7.2 and ACL
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Setup:
>>>>>>>>>>>>>> GFS 3.7.2, NFS is used
for host access
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Problem:
>>>>>>>>>>>>>> POSIX ACL work
correctly when ACLs are applied to files but do
>>>>>>>>>>>>>> not work when ACLs are
applied to directories on GFS volumes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How can I debug this
issue more deeply?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you please explain
the issue with more details, i.e what
>>>>>>>>>>>>>> exactly not working
properly , is it setting acl or any
>>>>>>>>>>>>>> functionality issue, in
which client?
>>>>>>>>>>>>>> __
>>>>>>>>>>>>>> Jiffin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Jyri
>>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>>> Gluster-users mailing
list
>>>>>>>>>>>>>> Gluster-users at
gluster.org
>>>>>>>>>>>>>>
<mailto:Gluster-users at gluster.org><mailto:Gluster-users at
gluster.org>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>>> Gluster-users at
gluster.org
>>>>>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>>>>
>>>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>>>
>>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>
_______________________________________________
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>>
>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>
>>>>>>>>>
_______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> <mailto:Gluster-users at
gluster.org><mailto:Gluster-users at gluster.org>
>>>>>>>>>
>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
<mailto:Gluster-users at gluster.org>
>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users