[Gluster-users] Replicated striped data lose

Mahdi Adnan
2016-03-15T07:37:21Z

Thanks Krutika,

I have deleted the volume and created a new one.
I found that it may be an issue with the NFS itself, i have created a
new striped volume and enabled sharding and mounted it via glusterfs and
it worked just fine, if i mount it with nfs it will fail and gives me
the same errors.

Respectfully*
**Mahdi A. Mahdi*

On 03/15/2016 06:24 AM, Krutika Dhananjay wrote:
> Hi,
>
> So could you share the xattrs associated with the file at
> <BRICK_PATH>/.glusterfs/c3/e8/c3e88cc1-7e0a-4d46-9685-2d12131a5e1c
>
> Here's what you need to execute:
>
> # getfattr -d -m . -e hex
> /mnt/b1/v/.glusterfs/c3/e8/c3e88cc1-7e0a-4d46-9685-2d12131a5e1c on the
> first node and
>
> # getfattr -d -m . -e hex
> /mnt/b2/v/.glusterfs/c3/e8/c3e88cc1-7e0a-4d46-9685-2d12131a5e1c on the
> second.
>
>
> Also, it is normally advised to use a replica 3 volume as opposed to
> replica 2 volume to guard against split-brains.
>
> -Krutika
>
> On Mon, Mar 14, 2016 at 3:17 PM, Mahdi Adnan
> <mahdi.adnan at earthlinktele.com <mailto:mahdi.adnan at
earthlinktele.com>>
> wrote:
>
>     sorry for serial posting but, i got new logs it might help..
>
>     the message appear during the migration;
>
>     /var/log/glusterfs/nfs.log
>
>
>     [2016-03-14 09:45:04.573765] I [MSGID: 109036]
>     [dht-common.c:8043:dht_log_new_layout_for_dir_selfheal]
>     0-testv-dht: Setting layout of /New Virtual Machine_1 with
>     [Subvol_name: testv-stripe-0, Err: -1 , Start: 0 , Stop:
>     4294967295 , Hash: 1 ],
>     [2016-03-14 09:45:04.957499] E
>     [shard.c:369:shard_modify_size_and_block_count]
>
(-->/usr/lib64/glusterfs/3.7.8/xlator/cluster/distribute.so(dht_file_setattr_cbk+0x14f)
>     [0x7f27a13c067f]
>
-->/usr/lib64/glusterfs/3.7.8/xlator/features/shard.so(shard_common_setattr_cbk+0xcc)
>     [0x7f27a116681c]
>
-->/usr/lib64/glusterfs/3.7.8/xlator/features/shard.so(shard_modify_size_and_block_count+0xdd)
>     [0x7f27a116584d] ) 0-testv-shard: Failed to get
>     trusted.glusterfs.shard.file-size for
>     c3e88cc1-7e0a-4d46-9685-2d12131a5e1c
>     [2016-03-14 09:45:04.957577] W [MSGID: 112199]
>     [nfs3-helpers.c:3418:nfs3_log_common_res] 0-nfs-nfsv3: /New
>     Virtual Machine_1/New Virtual Machine-flat.vmdk => (XID: 3fec5a26,
>     SETATTR: NFS: 22(Invalid argument for operation), POSIX:
>     22(Invalid argument)) [Invalid argument]
>     [2016-03-14 09:45:05.079657] E [MSGID: 112069]
>     [nfs3.c:3649:nfs3_rmdir_resume] 0-nfs-nfsv3: No such file or
>     directory: (192.168.221.52:826 <http://192.168.221.52:826>) testv
>     : 00000000-0000-0000-0000-000000000001
>
>
>
>     Respectfully*
>     **Mahdi A. Mahd
>
>     *
>     On 03/14/2016 11:14 AM, Mahdi Adnan wrote:
>>     So i have deployed a new server "Cisco UCS C220M4" and
created a
>>     new volume;
>>
>>     Volume Name: testv
>>     Type: Stripe
>>     Volume ID: 55cdac79-fe87-4f1f-90c0-15c9100fe00b
>>     Status: Started
>>     Number of Bricks: 1 x 2 = 2
>>     Transport-type: tcp
>>     Bricks:
>>     Brick1: 10.70.0.250:/mnt/b1/v
>>     Brick2: 10.70.0.250:/mnt/b2/v
>>     Options Reconfigured:
>>     nfs.disable: off
>>     features.shard-block-size: 64MB
>>     features.shard: enable
>>     cluster.server-quorum-type: server
>>     cluster.quorum-type: auto
>>     network.remote-dio: enable
>>     cluster.eager-lock: enable
>>     performance.stat-prefetch: off
>>     performance.io-cache: off
>>     performance.read-ahead: off
>>     performance.quick-read: off
>>     performance.readdir-ahead: off
>>
>>     same error ..
>>
>>     can anyone share with me the info of a working striped volume ?
>>
>>     On 03/14/2016 09:02 AM, Mahdi Adnan wrote:
>>>     I have a pool of two bricks in the same server;
>>>
>>>     Volume Name: k
>>>     Type: Stripe
>>>     Volume ID: 1e9281ce-2a8b-44e8-a0c6-e3ebf7416b2b
>>>     Status: Started
>>>     Number of Bricks: 1 x 2 = 2
>>>     Transport-type: tcp
>>>     Bricks:
>>>     Brick1: gfs001:/bricks/t1/k
>>>     Brick2: gfs001:/bricks/t2/k
>>>     Options Reconfigured:
>>>     features.shard-block-size: 64MB
>>>     features.shard: on
>>>     cluster.server-quorum-type: server
>>>     cluster.quorum-type: auto
>>>     network.remote-dio: enable
>>>     cluster.eager-lock: enable
>>>     performance.stat-prefetch: off
>>>     performance.io-cache: off
>>>     performance.read-ahead: off
>>>     performance.quick-read: off
>>>     performance.readdir-ahead: off
>>>
>>>     same issue ...
>>>     glusterfs 3.7.8 built on Mar 10 2016 20:20:45.
>>>
>>>
>>>     Respectfully*
>>>     **Mahdi A. Mahdi*
>>>
>>>     Systems Administrator
>>>     IT. Department
>>>     Earthlink Telecommunications
>>>     <https://www.facebook.com/earthlinktele>
>>>
>>>     Cell: 07903316180
>>>     Work: 3352
>>>     Skype: mahdi.adnan at outlook.com <mailto:mahdi.adnan at
outlook.com>
>>>     On 03/14/2016 08:11 AM, Niels de Vos wrote:
>>>>     On Mon, Mar 14, 2016 at 08:12:27AM +0530, Krutika Dhananjay
wrote:
>>>>>     It would be better to use sharding over stripe for your
vm use case. It
>>>>>     offers better distribution and utilisation of bricks
and better heal
>>>>>     performance.
>>>>>     And it is well tested.
>>>>     Basically the "striping" feature is deprecated,
"sharding" is its
>>>>     improved replacement. I expect to see "striping"
completely dropped in
>>>>     the next major release.
>>>>
>>>>     Niels
>>>>
>>>>
>>>>>     Couple of things to note before you do that:
>>>>>     1. Most of the bug fixes in sharding have gone into
3.7.8. So it is advised
>>>>>     that you use 3.7.8 or above.
>>>>>     2. When you enable sharding on a volume, already
existing files in the
>>>>>     volume do not get sharded. Only the files that are
newly created from the
>>>>>     time sharding is enabled will.
>>>>>          If you do want to shard the existing files, then
you would need to cp
>>>>>     them to a temp name within the volume, and then rename
them back to the
>>>>>     original file name.
>>>>>
>>>>>     HTH,
>>>>>     Krutika
>>>>>
>>>>>     On Sun, Mar 13, 2016 at 11:49 PM, Mahdi Adnan
<mahdi.adnan at earthlinktele.com
>>>>>     <mailto:mahdi.adnan at earthlinktele.com>
>>>>>>     wrote:
>>>>>>     I couldn't find anything related to cache in
the HBAs.
>>>>>>     what logs are useful in my case ? i see only bricks
logs which contains
>>>>>>     nothing during the failure.
>>>>>>
>>>>>>     ###
>>>>>>     [2016-03-13 18:05:19.728614] E [MSGID: 113022]
[posix.c:1232:posix_mknod]
>>>>>>     0-vmware-posix: mknod on
>>>>>>
/bricks/b003/vmware/.shard/17d75e20-16f1-405e-9fa5-99ee7b1bd7f1.511 failed
>>>>>>     [File exists]
>>>>>>     [2016-03-13 18:07:23.337086] E [MSGID: 113022]
[posix.c:1232:posix_mknod]
>>>>>>     0-vmware-posix: mknod on
>>>>>>
/bricks/b003/vmware/.shard/eef2d538-8eee-4e58-bc88-fbf7dc03b263.4095 failed
>>>>>>     [File exists]
>>>>>>     [2016-03-13 18:07:55.027600] W
[trash.c:1922:trash_rmdir] 0-vmware-trash:
>>>>>>     rmdir issued on /.trashcan/, which is not permitted
>>>>>>     [2016-03-13 18:07:55.027635] I [MSGID: 115056]
>>>>>>     [server-rpc-fops.c:459:server_rmdir_cbk]
0-vmware-server: 41987: RMDIR
>>>>>>     /.trashcan/internal_op
(00000000-0000-0000-0000-000000000005/internal_op)
>>>>>>     ==> (Operation not permitted) [Operation not
permitted]
>>>>>>     [2016-03-13 18:11:34.353441] I [login.c:81:gf_auth]
0-auth/login: allowed
>>>>>>     user names: c0c72c37-477a-49a5-a305-3372c1c2f2b4
>>>>>>     [2016-03-13 18:11:34.353463] I [MSGID: 115029]
>>>>>>     [server-handshake.c:612:server_setvolume]
0-vmware-server: accepted client
>>>>>>     from
gfs002-2727-2016/03/13-20:17:43:613597-vmware-client-4-0-0 (version:
>>>>>>     3.7.8)
>>>>>>     [2016-03-13 18:11:34.591139] I [login.c:81:gf_auth]
0-auth/login: allowed
>>>>>>     user names: c0c72c37-477a-49a5-a305-3372c1c2f2b4
>>>>>>     [2016-03-13 18:11:34.591173] I [MSGID: 115029]
>>>>>>     [server-handshake.c:612:server_setvolume]
0-vmware-server: accepted client
>>>>>>     from
gfs002-2719-2016/03/13-20:17:42:609388-vmware-client-4-0-0 (version:
>>>>>>     3.7.8)
>>>>>>     ###
>>>>>>
>>>>>>     ESXi just keeps telling me "Cannot clone T:
The virtual disk is either
>>>>>>     corrupted or not a supported format.
>>>>>>     error
>>>>>>     3/13/2016 9:06:20 PM
>>>>>>     Clone virtual machine
>>>>>>     T
>>>>>>     VCENTER.LOCAL\Administrator
>>>>>>     "
>>>>>>
>>>>>>     My setup is 2 servers with a floating ip controlled
by CTDB and my ESXi
>>>>>>     server mount the NFS via the floating ip.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>     On 03/13/2016 08:40 PM, pkoelle wrote:
>>>>>>
>>>>>>>     Am 13.03.2016 um 18:22 schrieb David Gossage:
>>>>>>>
>>>>>>>>     On Sun, Mar 13, 2016 at 11:07 AM, Mahdi
Adnan <
>>>>>>>>     mahdi.adnan at earthlinktele.com
>>>>>>>>     <mailto:mahdi.adnan at
earthlinktele.com>
>>>>>>>>
>>>>>>>>>     wrote:
>>>>>>>>>
>>>>>>>>     My HBAs are LSISAS1068E, and the filesystem
is XFS.
>>>>>>>>>     I tried EXT4 and it did not help.
>>>>>>>>>     I have created a stripted volume in one
server with two bricks, same
>>>>>>>>>     issue.
>>>>>>>>>     and i tried a replicated volume with
just "sharding enabled" same issue,
>>>>>>>>>     as soon as i disable the sharding it
works just fine, niether sharding
>>>>>>>>>     nor
>>>>>>>>>     striping works for me.
>>>>>>>>>     i did follow up with some of threads in
the mailing list and tried some
>>>>>>>>>     of
>>>>>>>>>     the fixes that worked with the others,
none worked for me. :(
>>>>>>>>>
>>>>>>>>>
>>>>>>>>     Is it possible the LSI has write-cache
enabled?
>>>>>>>>
>>>>>>>     Why is that relevant? Even the backing
filesystem has no idea if there is
>>>>>>>     a RAID or write cache or whatever. There are
blocks and sync(), end of
>>>>>>>     story.
>>>>>>>     If you lose power and screw up your recovery OR
do funky stuff with SAS
>>>>>>>     multipathing that might be an issue with a
controller cache. AFAIK thats
>>>>>>>     not what we are talking about.
>>>>>>>
>>>>>>>     I'm afraid but unless the OP has some logs
from the server, a
>>>>>>>     reproducible testcase or a backtrace from
client or server this isn't
>>>>>>>     getting us anywhere.
>>>>>>>
>>>>>>>     cheers
>>>>>>>     Paul
>>>>>>>
>>>>>>>
>>>>>>>>     On 03/13/2016 06:54 PM, David Gossage
wrote:
>>>>>>>>>     On Sun, Mar 13, 2016 at 8:16 AM, Mahdi
Adnan <
>>>>>>>>>     mahdi.adnan at earthlinktele.com
>>>>>>>>>     <mailto:mahdi.adnan at
earthlinktele.com>> wrote:
>>>>>>>>>
>>>>>>>>>     Okay so i have enabled shard in my test
volume and it did not help,
>>>>>>>>>>     stupidly enough, i have enabled it
in a production volume
>>>>>>>>>>     "Distributed-Replicate"
and it currpted  half of my VMs.
>>>>>>>>>>     I have updated Gluster to the
latest and nothing seems to be changed in
>>>>>>>>>>     my situation.
>>>>>>>>>>     below the info of my volume;
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>     I was pointing at the settings in that
email as an example for
>>>>>>>>>     corruption
>>>>>>>>>     fixing. I wouldn't recommend
enabling sharding if you haven't gotten the
>>>>>>>>>     base working yet on that cluster. What
HBA's are you using and what is
>>>>>>>>>     layout of filesystem for bricks?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>     Number of Bricks: 3 x 2 = 6
>>>>>>>>>>     Transport-type: tcp
>>>>>>>>>>     Bricks:
>>>>>>>>>>     Brick1: gfs001:/bricks/b001/vmware
>>>>>>>>>>     Brick2: gfs002:/bricks/b004/vmware
>>>>>>>>>>     Brick3: gfs001:/bricks/b002/vmware
>>>>>>>>>>     Brick4: gfs002:/bricks/b005/vmware
>>>>>>>>>>     Brick5: gfs001:/bricks/b003/vmware
>>>>>>>>>>     Brick6: gfs002:/bricks/b006/vmware
>>>>>>>>>>     Options Reconfigured:
>>>>>>>>>>     performance.strict-write-ordering:
on
>>>>>>>>>>     cluster.server-quorum-type: server
>>>>>>>>>>     cluster.quorum-type: auto
>>>>>>>>>>     network.remote-dio: enable
>>>>>>>>>>     performance.stat-prefetch: disable
>>>>>>>>>>     performance.io-cache: off
>>>>>>>>>>     performance.read-ahead: off
>>>>>>>>>>     performance.quick-read: off
>>>>>>>>>>     cluster.eager-lock: enable
>>>>>>>>>>     features.shard-block-size: 16MB
>>>>>>>>>>     features.shard: on
>>>>>>>>>>     performance.readdir-ahead: off
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     On 03/12/2016 08:11 PM, David
Gossage wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     On Sat, Mar 12, 2016 at 10:21 AM,
Mahdi Adnan <
>>>>>>>>>>     <mahdi.adnan at
earthlinktele.com>
>>>>>>>>>>     <mailto:mahdi.adnan at
earthlinktele.com>mahdi.adnan at earthlinktele.com
>>>>>>>>>>     <mailto:mahdi.adnan at
earthlinktele.com>> wrote:
>>>>>>>>>>
>>>>>>>>>>     Both servers have HBA no RAIDs and
i can setup a replicated or
>>>>>>>>>>>     dispensers without any issues.
>>>>>>>>>>>     Logs are clean and when i tried
to migrate a vm and got the error,
>>>>>>>>>>>     nothing showed up in the logs.
>>>>>>>>>>>     i tried mounting the volume
into my laptop and it mounted fine but,
>>>>>>>>>>>     if i
>>>>>>>>>>>     use dd to create a data file it
just hang and i cant cancel it, and i
>>>>>>>>>>>     cant
>>>>>>>>>>>     unmount it or anything, i just
have to reboot.
>>>>>>>>>>>     The same servers have another
volume on other bricks in a distributed
>>>>>>>>>>>     replicas, works fine.
>>>>>>>>>>>     I have even tried the same
setup in a virtual environment (created two
>>>>>>>>>>>     vms and install gluster and
created a replicated striped) and again
>>>>>>>>>>>     same
>>>>>>>>>>>     thing, data corruption.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>     I'd look through mail archives
for a topic "Shard in Production" I
>>>>>>>>>>     think
>>>>>>>>>>     it's called.  The shard portion
may not be relevant but it does discuss
>>>>>>>>>>     certain settings that had to be
applied with regards to avoiding
>>>>>>>>>>     corruption
>>>>>>>>>>     with VM's.  You may want to try
and disable the
>>>>>>>>>>     performance.readdir-ahead
>>>>>>>>>>     also.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>     On 03/12/2016 07:02 PM, David
Gossage wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>     On Sat, Mar 12, 2016 at 9:51
AM, Mahdi Adnan <
>>>>>>>>>>>     <mahdi.adnan at
earthlinktele.com>
>>>>>>>>>>>     <mailto:mahdi.adnan at
earthlinktele.com>mahdi.adnan at earthlinktele.com
>>>>>>>>>>>     <mailto:mahdi.adnan at
earthlinktele.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>     Thanks David,
>>>>>>>>>>>>     My settings are all
defaults, i have just created the pool and
>>>>>>>>>>>>     started
>>>>>>>>>>>>     it.
>>>>>>>>>>>>     I have set the settings as
your recommendation and it seems to be the
>>>>>>>>>>>>     same issue;
>>>>>>>>>>>>
>>>>>>>>>>>>     Type: Striped-Replicate
>>>>>>>>>>>>     Volume ID:
44adfd8c-2ed1-4aa5-b256-d12b64f7fc14
>>>>>>>>>>>>     Status: Started
>>>>>>>>>>>>     Number of Bricks: 1 x 2 x 2
= 4
>>>>>>>>>>>>     Transport-type: tcp
>>>>>>>>>>>>     Bricks:
>>>>>>>>>>>>     Brick1: gfs001:/bricks/t1/s
>>>>>>>>>>>>     Brick2: gfs002:/bricks/t1/s
>>>>>>>>>>>>     Brick3: gfs001:/bricks/t2/s
>>>>>>>>>>>>     Brick4: gfs002:/bricks/t2/s
>>>>>>>>>>>>     Options Reconfigured:
>>>>>>>>>>>>     performance.stat-prefetch:
off
>>>>>>>>>>>>     network.remote-dio: on
>>>>>>>>>>>>     cluster.eager-lock: enable
>>>>>>>>>>>>     performance.io-cache: off
>>>>>>>>>>>>     performance.read-ahead: off
>>>>>>>>>>>>     performance.quick-read: off
>>>>>>>>>>>>     performance.readdir-ahead:
on
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>     Is their a raid controller
perhaps doing any caching?
>>>>>>>>>>>
>>>>>>>>>>>     In the gluster logs any errors
being reported during migration
>>>>>>>>>>>     process?
>>>>>>>>>>>     Since they aren't in use
yet have you tested making just mirrored
>>>>>>>>>>>     bricks
>>>>>>>>>>>     using different pairings of
servers two at a time to see if problem
>>>>>>>>>>>     follows
>>>>>>>>>>>     certain machine or network
ports?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>     On 03/12/2016 03:25 PM,
David Gossage wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>     On Sat, Mar 12, 2016 at
1:55 AM, Mahdi Adnan <
>>>>>>>>>>>>     <mahdi.adnan at
earthlinktele.com>
>>>>>>>>>>>>     <mailto:mahdi.adnan at
earthlinktele.com>mahdi.adnan at earthlinktele.com
>>>>>>>>>>>>     <mailto:mahdi.adnan at
earthlinktele.com>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>     Dears,
>>>>>>>>>>>>>     I have created a
replicated striped volume with two bricks and two
>>>>>>>>>>>>>     servers but I can't
use it because when I mount it in ESXi and try
>>>>>>>>>>>>>     to
>>>>>>>>>>>>>     migrate a VM to it, the
data get corrupted.
>>>>>>>>>>>>>     Is any one have any
idea why is this happening ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>     Dell 2950 x2
>>>>>>>>>>>>>     Seagate 15k 600GB
>>>>>>>>>>>>>     CentOS 7.2
>>>>>>>>>>>>>     Gluster 3.7.8
>>>>>>>>>>>>>
>>>>>>>>>>>>>     Appreciate your help.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>     Most reports of this I have
seen end up being settings related.  Post
>>>>>>>>>>>>     gluster volume info. Below
is what I have seen as most common
>>>>>>>>>>>>     recommended
>>>>>>>>>>>>     settings.
>>>>>>>>>>>>     I'd hazard a guess you
may have some the read ahead cache or prefetch
>>>>>>>>>>>>     on.
>>>>>>>>>>>>
>>>>>>>>>>>>     quick-read=off
>>>>>>>>>>>>     read-ahead=off
>>>>>>>>>>>>     io-cache=off
>>>>>>>>>>>>     stat-prefetch=off
>>>>>>>>>>>>     eager-lock=enable
>>>>>>>>>>>>     remote-dio=on
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>     Mahdi Adnan
>>>>>>>>>>>>>     System Admin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>>     Gluster-users mailing
list
>>>>>>>>>>>>>     <Gluster-users at
gluster.org>
>>>>>>>>>>>>>
<mailto:Gluster-users at gluster.org>Gluster-users at gluster.org
>>>>>>>>>>>>>
<mailto:Gluster-users at gluster.org>
>>>>>>>>>>>>>
<http://www.gluster.org/mailman/listinfo/gluster-users>
<http://www.gluster.org/mailman/listinfo/gluster-users>
>>>>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>
_______________________________________________
>>>>>>>>     Gluster-users mailing list
>>>>>>>>     Gluster-users at gluster.org
<mailto:Gluster-users at gluster.org>
>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>
>>>>>>>>
>>>>>>>     _______________________________________________
>>>>>>>     Gluster-users mailing list
>>>>>>>     Gluster-users at gluster.org
<mailto:Gluster-users at gluster.org>
>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>     _______________________________________________
>>>>>>     Gluster-users mailing list
>>>>>>     Gluster-users at gluster.org
<mailto:Gluster-users at gluster.org>
>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>     _______________________________________________
>>>>>     Gluster-users mailing list
>>>>>     Gluster-users at gluster.org <mailto:Gluster-users
at gluster.org>
>>>>>     http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>>     _______________________________________________
>>>     Gluster-users mailing list
>>>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>>>     http://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>     _______________________________________________
>>     Gluster-users mailing list
>>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>>     http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>     http://www.gluster.org/mailman/listinfo/gluster-users
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160315/e9ad3263/attachment.html>