thr3ads.net - Gluster users - [Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding [Jan 2018]

If this information is useful, please help other people find it:
Share via:

Ing. Luca Lazzeroni - Trend Servizi Srl

2018-Jan-17 13:51 UTC

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

Hi,

after our IRC chat I've rebuilt a virtual machine with FUSE based 
virtual disk. Everything worked flawlessly.

Now I'm sending you the output of the requested getfattr command on the 
disk image:

# file: TestFUSE-vda.qcow2
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x40ffafbbe987445692bb31295fa40105
trusted.gfid2path.dc9dde61f0b77eab=0x31326533323631662d373839332d346262302d383738632d3966623765306232336263652f54657374465553452d7664612e71636f7732
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000c15300000000000000000000000000000060be900000000000000000

Hope this helps.



Il 17/01/2018 11:37, Ing. Luca Lazzeroni - Trend Servizi Srl ha
scritto:>
> I actually use FUSE and it works. If i try to use "libgfapi"
direct
> interface to gluster in qemu-kvm, the problem appears.
>
>
>
> Il 17/01/2018 11:35, Krutika Dhananjay ha scritto:
>> Really? Then which protocol exactly do you see this issue with? 
>> libgfapi? NFS?
>>
>> -Krutika
>>
>> On Wed, Jan 17, 2018 at 3:59 PM, Ing. Luca Lazzeroni - Trend Servizi 
>> Srl <luca at gvnet.it <mailto:luca at gvnet.it>> wrote:
>>
>>     Of course. Here's the full log. Please, note that in FUSE mode
>>     everything works apparently without problems. I've installed 4
vm
>>     and updated them without problems.
>>
>>
>>
>>     Il 17/01/2018 11:00, Krutika Dhananjay ha scritto:
>>>
>>>
>>>     On Tue, Jan 16, 2018 at 10:47 PM, Ing. Luca Lazzeroni - Trend
>>>     Servizi Srl <luca at gvnet.it <mailto:luca at
gvnet.it>> wrote:
>>>
>>>         I've made the test with raw image format (preallocated
too)
>>>         and the corruption problem is still there (but without
>>>         errors in bricks' log file).
>>>
>>>         What does the "link" error in bricks log files
means ?
>>>
>>>         I've seen the source code looking for the lines where
it
>>>         happens and it seems a warning (it doesn't imply a
failure).
>>>
>>>
>>>     Indeed, it only represents a transient state when the shards
are
>>>     created for the first time and does not indicate a failure.
>>>     Could you also get the logs of the gluster fuse mount process?
>>>     It should be under /var/log/glusterfs of your client machine
>>>     with the filename as a hyphenated mount point path.
>>>
>>>     For example, if your volume was mounted at /mnt/glusterfs, then
>>>     your log file would be named mnt-glusterfs.log.
>>>
>>>     -Krutika
>>>
>>>
>>>
>>>         Il 16/01/2018 17:39, Ing. Luca Lazzeroni - Trend Servizi
Srl
>>>         ha scritto:
>>>>
>>>>         An update:
>>>>
>>>>         I've tried, for my tests, to create the vm volume
as
>>>>
>>>>         qemu-img create -f qcow2 -o preallocation=full
>>>>         gluster://gluster1/Test/Test-vda.img 20G
>>>>
>>>>         et voila !
>>>>
>>>>         No errors at all, neither in bricks' log file (the
"link
>>>>         failed" message disappeared), neither in VM (no
corruption
>>>>         and installed succesfully).
>>>>
>>>>         I'll do another test with a fully preallocated raw
image.
>>>>
>>>>
>>>>
>>>>         Il 16/01/2018 16:31, Ing. Luca Lazzeroni - Trend
Servizi
>>>>         Srl ha scritto:
>>>>>
>>>>>         I've just done all the steps to reproduce the
problem.
>>>>>
>>>>>         Tha VM volume has been created via "qemu-img
create -f
>>>>>         qcow2 Test-vda2.qcow2 20G" on the gluster
volume mounted
>>>>>         via FUSE. I've tried also to create the volume
with
>>>>>         preallocated metadata, which moves the problem a
bit far
>>>>>         away (in time). The volume is a replice 3 arbiter 1
volume
>>>>>         hosted on XFS bricks.
>>>>>
>>>>>         Here are the informations:
>>>>>
>>>>>         [root at ovh-ov1 bricks]# gluster volume info gv2a2
>>>>>
>>>>>         Volume Name: gv2a2
>>>>>         Type: Replicate
>>>>>         Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f
>>>>>         Status: Started
>>>>>         Snapshot Count: 0
>>>>>         Number of Bricks: 1 x (2 + 1) = 3
>>>>>         Transport-type: tcp
>>>>>         Bricks:
>>>>>         Brick1: gluster1:/bricks/brick2/gv2a2
>>>>>         Brick2: gluster3:/bricks/brick3/gv2a2
>>>>>         Brick3: gluster2:/bricks/arbiter_brick_gv2a2/gv2a2
(arbiter)
>>>>>         Options Reconfigured:
>>>>>         storage.owner-gid: 107
>>>>>         storage.owner-uid: 107
>>>>>         user.cifs: off
>>>>>         features.shard: on
>>>>>         cluster.shd-wait-qlength: 10000
>>>>>         cluster.shd-max-threads: 8
>>>>>         cluster.locking-scheme: granular
>>>>>         cluster.data-self-heal-algorithm: full
>>>>>         cluster.server-quorum-type: server
>>>>>         cluster.quorum-type: auto
>>>>>         cluster.eager-lock: enable
>>>>>         network.remote-dio: enable
>>>>>         performance.low-prio-threads: 32
>>>>>         performance.io-cache: off
>>>>>         performance.read-ahead: off
>>>>>         performance.quick-read: off
>>>>>         transport.address-family: inet
>>>>>         nfs.disable: off
>>>>>         performance.client-io-threads: off
>>>>>
>>>>>         /var/log/glusterfs/glusterd.log:
>>>>>
>>>>>         [2018-01-15 14:17:50.196228] I [MSGID: 106488]
>>>>>        
[glusterd-handler.c:1548:__glusterd_handle_cli_get_volume]
>>>>>         0-management: Received get vol req
>>>>>         [2018-01-15 14:25:09.555214] I [MSGID: 106488]
>>>>>        
[glusterd-handler.c:1548:__glusterd_handle_cli_get_volume]
>>>>>         0-management: Received get vol req
>>>>>
>>>>>         (empty because today it's 2018-01-16)
>>>>>
>>>>>         /var/log/glusterfs/glustershd.log:
>>>>>
>>>>>         [2018-01-14 02:23:02.731245] I
>>>>>         [glusterfsd-mgmt.c:1821:mgmt_getspec_cbk]
0-glusterfs: No
>>>>>         change in volfile,continuing
>>>>>
>>>>>         (empty too)
>>>>>
>>>>>         /var/log/glusterfs/bricks/brick-brick2-gv2a2.log
(the
>>>>>         interested volume):
>>>>>
>>>>>         [2018-01-16 15:14:37.809965] I [MSGID: 115029]
>>>>>         [server-handshake.c:793:server_setvolume]
0-gv2a2-server:
>>>>>         accepted client from
>>>>>        
ovh-ov1-10302-2018/01/16-15:14:37:790306-gv2a2-client-0-0-0
>>>>>         (version: 3.12.4)
>>>>>         [2018-01-16 15:16:41.471751] E [MSGID: 113020]
>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting
gfid on
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
>>>>>         failed
>>>>>         [2018-01-16 15:16:41.471745] W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
>>>>>         [File exists]
>>>>>         [2018-01-16 15:16:42.593392] W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
>>>>>         [File exists]
>>>>>         [2018-01-16 15:16:42.593426] E [MSGID: 113020]
>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting
gfid on
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
>>>>>         failed
>>>>>         [2018-01-16 15:17:04.129593] W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
>>>>>         [File exists]
>>>>>         The message "W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
>>>>>         [File exists]" repeated 5 times between
[2018-01-16
>>>>>         15:17:04.129593] and [2018-01-16 15:17:04.129593]
>>>>>         [2018-01-16 15:17:04.129661] E [MSGID: 113020]
>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting
gfid on
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
>>>>>         failed
>>>>>         [2018-01-16 15:17:08.279162] W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
>>>>>         [File exists]
>>>>>         [2018-01-16 15:17:08.279162] W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
>>>>>         [File exists]
>>>>>         The message "W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
>>>>>         [File exists]" repeated 2 times between
[2018-01-16
>>>>>         15:17:08.279162] and [2018-01-16 15:17:08.279162]
>>>>>
>>>>>         [2018-01-16 15:17:08.279177] E [MSGID: 113020]
>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting
gfid on
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>>>         failed
>>>>>         The message "W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
>>>>>         [File exists]" repeated 6 times between
[2018-01-16
>>>>>         15:16:41.471745] and [2018-01-16 15:16:41.471807]
>>>>>         The message "W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
>>>>>         [File exists]" repeated 2 times between
[2018-01-16
>>>>>         15:16:42.593392] and [2018-01-16 15:16:42.593430]
>>>>>         [2018-01-16 15:17:32.229689] W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
>>>>>         [File exists]
>>>>>         [2018-01-16 15:17:32.229720] E [MSGID: 113020]
>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting
gfid on
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
>>>>>         failed
>>>>>         [2018-01-16 15:18:07.154330] W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
>>>>>         [File exists]
>>>>>         [2018-01-16 15:18:07.154375] E [MSGID: 113020]
>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting
gfid on
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
>>>>>         failed
>>>>>         The message "W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
>>>>>         [File exists]" repeated 7 times between
[2018-01-16
>>>>>         15:17:32.229689] and [2018-01-16 15:17:32.229806]
>>>>>         The message "W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
>>>>>         [File exists]" repeated 3 times between
[2018-01-16
>>>>>         15:18:07.154330] and [2018-01-16 15:18:07.154357]
>>>>>         [2018-01-16 15:19:23.618794] W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
>>>>>         [File exists]
>>>>>         [2018-01-16 15:19:23.618827] E [MSGID: 113020]
>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix: setting
gfid on
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
>>>>>         failed
>>>>>         The message "W [MSGID: 113096]
>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix: link
>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
>>>>>         ->
>>>>>        
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
>>>>>         [File exists]" repeated 3 times between
[2018-01-16
>>>>>         15:19:23.618794] and [2018-01-16 15:19:23.618794]
>>>>>
>>>>>         Thank you,
>>>>>
>>>>>
>>>>>         Il 16/01/2018 11:40, Krutika Dhananjay ha scritto:
>>>>>>         Also to help isolate the component, could you
answer these:
>>>>>>
>>>>>>         1. on a different volume with shard not
enabled, do you
>>>>>>         see this issue?
>>>>>>         2. on a plain 3-way replicated volume (no
arbiter), do
>>>>>>         you see this issue?
>>>>>>
>>>>>>
>>>>>>
>>>>>>         On Tue, Jan 16, 2018 at 4:03 PM, Krutika
Dhananjay
>>>>>>         <kdhananj at redhat.com <mailto:kdhananj
at redhat.com>> wrote:
>>>>>>
>>>>>>             Please share the volume-info output and the
logs
>>>>>>             under /var/log/glusterfs/ from all your
nodes. for
>>>>>>             investigating the issue.
>>>>>>
>>>>>>             -Krutika
>>>>>>
>>>>>>             On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca
Lazzeroni
>>>>>>             - Trend Servizi Srl <luca at gvnet.it
>>>>>>             <mailto:luca at gvnet.it>> wrote:
>>>>>>
>>>>>>                 Hi to everyone.
>>>>>>
>>>>>>                 I've got a strange problem with a
gluster setup:
>>>>>>                 3 nodes with Centos 7.4, Gluster 3.12.4
from
>>>>>>                 Centos/Gluster repositories, QEMU-KVM
version
>>>>>>                 2.9.0 (compiled from RHEL sources).
>>>>>>
>>>>>>                 I'm running volumes in replica 3
arbiter 1 mode
>>>>>>                 (but I've got a volume in
"pure" replica 3 mode
>>>>>>                 too). I've applied the
"virt" group settings to
>>>>>>                 my volumes since they host VM images.
>>>>>>
>>>>>>                 If I try to install something (eg:
Ubuntu Server
>>>>>>                 16.04.3) on a VM (and so I generate a
bit of I/O
>>>>>>                 inside it) and configure KVM to access
gluster
>>>>>>                 volume directly (via libvirt), install
fails
>>>>>>                 after a while because the disk content
is
>>>>>>                 corrupted. If I inspect the block
inside the disk
>>>>>>                 (by accessing the image directly from
outside) I
>>>>>>                 can found many files filled with
"^@".
>>>>>>
>>>>>>
>>>>>>         Also, what exactly do you mean by accessing the
image
>>>>>>         directly from outside? Was it from the brick
directories
>>>>>>         directly? Was it from the mount point of the
volume?
>>>>>>         Could you elaborate? Which files exactly did
you check?
>>>>>>
>>>>>>         -Krutika
>>>>>>
>>>>>>
>>>>>>                 If, instead, I configure KVM to access
VM images
>>>>>>                 via a FUSE mount, everything seems to
work correctly.
>>>>>>
>>>>>>                 Note that the problem with install is
verified
>>>>>>                 100% time with QCOW2 image, while it
appears only
>>>>>>                 after with RAW disk images.
>>>>>>
>>>>>>                 Is there anyone who experienced the
same problem ?
>>>>>>
>>>>>>                 Thank you,
>>>>>>
>>>>>>
>>>>>>                 -- 
>>>>>>                 Ing. Luca Lazzeroni
>>>>>>                 Responsabile Ricerca e Sviluppo
>>>>>>                 Trend Servizi Srl
>>>>>>                 Tel: 0376/631761
>>>>>>                 Web: https://www.trendservizi.it
>>>>>>
>>>>>>                
_______________________________________________
>>>>>>                 Gluster-users mailing list
>>>>>>                 Gluster-users at gluster.org
>>>>>>                 <mailto:Gluster-users at
gluster.org>
>>>>>>                
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>                
<http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>         -- 
>>>>>         Ing. Luca Lazzeroni
>>>>>         Responsabile Ricerca e Sviluppo
>>>>>         Trend Servizi Srl
>>>>>         Tel: 0376/631761
>>>>>         Web:https://www.trendservizi.it
>>>>>
>>>>>
>>>>>         _______________________________________________
>>>>>         Gluster-users mailing list
>>>>>         Gluster-users at gluster.org
<mailto:Gluster-users at gluster.org>
>>>>>        
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>        
<http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>
>>>>         -- 
>>>>         Ing. Luca Lazzeroni
>>>>         Responsabile Ricerca e Sviluppo
>>>>         Trend Servizi Srl
>>>>         Tel: 0376/631761
>>>>         Web:https://www.trendservizi.it
>>>>
>>>>
>>>>         _______________________________________________
>>>>         Gluster-users mailing list
>>>>         Gluster-users at gluster.org <mailto:Gluster-users
at gluster.org>
>>>>         http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>        
<http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>
>>>         -- 
>>>         Ing. Luca Lazzeroni
>>>         Responsabile Ricerca e Sviluppo
>>>         Trend Servizi Srl
>>>         Tel: 0376/631761
>>>         Web:https://www.trendservizi.it
>>>
>>>
>>>         _______________________________________________
>>>         Gluster-users mailing list
>>>         Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>>>         http://lists.gluster.org/mailman/listinfo/gluster-users
>>>        
<http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>
>>>
>>
>>     -- 
>>     Ing. Luca Lazzeroni
>>     Responsabile Ricerca e Sviluppo
>>     Trend Servizi Srl
>>     Tel: 0376/631761
>>     Web:https://www.trendservizi.it
>>
>>
>
> -- 
> Ing. Luca Lazzeroni
> Responsabile Ricerca e Sviluppo
> Trend Servizi Srl
> Tel: 0376/631761
> Web:https://www.trendservizi.it
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-- 
Ing. Luca Lazzeroni
Responsabile Ricerca e Sviluppo
Trend Servizi Srl
Tel: 0376/631761
Web: https://www.trendservizi.it

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180117/a617a0cd/attachment.html>

Ing. Luca Lazzeroni - Trend Servizi Srl

2018-Jan-18 08:09 UTC

head link

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

Another update.

I've setup a replica 3 volume without sharding and tried to install a VM 
on a qcow2 volume on that device; however the result is the same and the 
vm image has been corrupted, exactly at the same point.

Here's the volume info of the create volume:

Volume Name: gvtest
Type: Replicate
Volume ID: e2ddf694-ba46-4bc7-bc9c-e30803374e9d
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gluster1:/bricks/brick1/gvtest
Brick2: gluster2:/bricks/brick1/gvtest
Brick3: gluster3:/bricks/brick1/gvtest
Options Reconfigured:
user.cifs: off
features.shard: off
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: enable
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off


Il 17/01/2018 14:51, Ing. Luca Lazzeroni - Trend Servizi Srl ha
scritto:>
> Hi,
>
> after our IRC chat I've rebuilt a virtual machine with FUSE based 
> virtual disk. Everything worked flawlessly.
>
> Now I'm sending you the output of the requested getfattr command on 
> the disk image:
>
> # file: TestFUSE-vda.qcow2
> trusted.afr.dirty=0x000000000000000000000000
> trusted.gfid=0x40ffafbbe987445692bb31295fa40105
>
trusted.gfid2path.dc9dde61f0b77eab=0x31326533323631662d373839332d346262302d383738632d3966623765306232336263652f54657374465553452d7664612e71636f7732
> trusted.glusterfs.shard.block-size=0x0000000004000000
>
trusted.glusterfs.shard.file-size=0x00000000c15300000000000000000000000000000060be900000000000000000
>
> Hope this helps.
>
>
>
> Il 17/01/2018 11:37, Ing. Luca Lazzeroni - Trend Servizi Srl ha scritto:
>>
>> I actually use FUSE and it works. If i try to use "libgfapi"
direct
>> interface to gluster in qemu-kvm, the problem appears.
>>
>>
>>
>> Il 17/01/2018 11:35, Krutika Dhananjay ha scritto:
>>> Really? Then which protocol exactly do you see this issue with? 
>>> libgfapi? NFS?
>>>
>>> -Krutika
>>>
>>> On Wed, Jan 17, 2018 at 3:59 PM, Ing. Luca Lazzeroni - Trend
Servizi
>>> Srl <luca at gvnet.it <mailto:luca at gvnet.it>> wrote:
>>>
>>>     Of course. Here's the full log. Please, note that in FUSE
mode
>>>     everything works apparently without problems. I've
installed 4
>>>     vm and updated them without problems.
>>>
>>>
>>>
>>>     Il 17/01/2018 11:00, Krutika Dhananjay ha scritto:
>>>>
>>>>
>>>>     On Tue, Jan 16, 2018 at 10:47 PM, Ing. Luca Lazzeroni -
Trend
>>>>     Servizi Srl <luca at gvnet.it <mailto:luca at
gvnet.it>> wrote:
>>>>
>>>>         I've made the test with raw image format
(preallocated too)
>>>>         and the corruption problem is still there (but without
>>>>         errors in bricks' log file).
>>>>
>>>>         What does the "link" error in bricks log
files means ?
>>>>
>>>>         I've seen the source code looking for the lines
where it
>>>>         happens and it seems a warning (it doesn't imply a
failure).
>>>>
>>>>
>>>>     Indeed, it only represents a transient state when the
shards
>>>>     are created for the first time and does not indicate a
failure.
>>>>     Could you also get the logs of the gluster fuse mount
process?
>>>>     It should be under /var/log/glusterfs of your client
machine
>>>>     with the filename as a hyphenated mount point path.
>>>>
>>>>     For example, if your volume was mounted at /mnt/glusterfs,
then
>>>>     your log file would be named mnt-glusterfs.log.
>>>>
>>>>     -Krutika
>>>>
>>>>
>>>>
>>>>         Il 16/01/2018 17:39, Ing. Luca Lazzeroni - Trend
Servizi
>>>>         Srl ha scritto:
>>>>>
>>>>>         An update:
>>>>>
>>>>>         I've tried, for my tests, to create the vm
volume as
>>>>>
>>>>>         qemu-img create -f qcow2 -o preallocation=full
>>>>>         gluster://gluster1/Test/Test-vda.img 20G
>>>>>
>>>>>         et voila !
>>>>>
>>>>>         No errors at all, neither in bricks' log file
(the "link
>>>>>         failed" message disappeared), neither in VM
(no corruption
>>>>>         and installed succesfully).
>>>>>
>>>>>         I'll do another test with a fully preallocated
raw image.
>>>>>
>>>>>
>>>>>
>>>>>         Il 16/01/2018 16:31, Ing. Luca Lazzeroni - Trend
Servizi
>>>>>         Srl ha scritto:
>>>>>>
>>>>>>         I've just done all the steps to reproduce
the problem.
>>>>>>
>>>>>>         Tha VM volume has been created via
"qemu-img create -f
>>>>>>         qcow2 Test-vda2.qcow2 20G" on the gluster
volume mounted
>>>>>>         via FUSE. I've tried also to create the
volume with
>>>>>>         preallocated metadata, which moves the problem
a bit far
>>>>>>         away (in time). The volume is a replice 3
arbiter 1
>>>>>>         volume hosted on XFS bricks.
>>>>>>
>>>>>>         Here are the informations:
>>>>>>
>>>>>>         [root at ovh-ov1 bricks]# gluster volume info
gv2a2
>>>>>>
>>>>>>         Volume Name: gv2a2
>>>>>>         Type: Replicate
>>>>>>         Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f
>>>>>>         Status: Started
>>>>>>         Snapshot Count: 0
>>>>>>         Number of Bricks: 1 x (2 + 1) = 3
>>>>>>         Transport-type: tcp
>>>>>>         Bricks:
>>>>>>         Brick1: gluster1:/bricks/brick2/gv2a2
>>>>>>         Brick2: gluster3:/bricks/brick3/gv2a2
>>>>>>         Brick3:
gluster2:/bricks/arbiter_brick_gv2a2/gv2a2 (arbiter)
>>>>>>         Options Reconfigured:
>>>>>>         storage.owner-gid: 107
>>>>>>         storage.owner-uid: 107
>>>>>>         user.cifs: off
>>>>>>         features.shard: on
>>>>>>         cluster.shd-wait-qlength: 10000
>>>>>>         cluster.shd-max-threads: 8
>>>>>>         cluster.locking-scheme: granular
>>>>>>         cluster.data-self-heal-algorithm: full
>>>>>>         cluster.server-quorum-type: server
>>>>>>         cluster.quorum-type: auto
>>>>>>         cluster.eager-lock: enable
>>>>>>         network.remote-dio: enable
>>>>>>         performance.low-prio-threads: 32
>>>>>>         performance.io-cache: off
>>>>>>         performance.read-ahead: off
>>>>>>         performance.quick-read: off
>>>>>>         transport.address-family: inet
>>>>>>         nfs.disable: off
>>>>>>         performance.client-io-threads: off
>>>>>>
>>>>>>         /var/log/glusterfs/glusterd.log:
>>>>>>
>>>>>>         [2018-01-15 14:17:50.196228] I [MSGID: 106488]
>>>>>>        
[glusterd-handler.c:1548:__glusterd_handle_cli_get_volume]
>>>>>>         0-management: Received get vol req
>>>>>>         [2018-01-15 14:25:09.555214] I [MSGID: 106488]
>>>>>>        
[glusterd-handler.c:1548:__glusterd_handle_cli_get_volume]
>>>>>>         0-management: Received get vol req
>>>>>>
>>>>>>         (empty because today it's 2018-01-16)
>>>>>>
>>>>>>         /var/log/glusterfs/glustershd.log:
>>>>>>
>>>>>>         [2018-01-14 02:23:02.731245] I
>>>>>>         [glusterfsd-mgmt.c:1821:mgmt_getspec_cbk]
0-glusterfs: No
>>>>>>         change in volfile,continuing
>>>>>>
>>>>>>         (empty too)
>>>>>>
>>>>>>        
/var/log/glusterfs/bricks/brick-brick2-gv2a2.log (the
>>>>>>         interested volume):
>>>>>>
>>>>>>         [2018-01-16 15:14:37.809965] I [MSGID: 115029]
>>>>>>         [server-handshake.c:793:server_setvolume]
0-gv2a2-server:
>>>>>>         accepted client from
>>>>>>        
ovh-ov1-10302-2018/01/16-15:14:37:790306-gv2a2-client-0-0-0
>>>>>>         (version: 3.12.4)
>>>>>>         [2018-01-16 15:16:41.471751] E [MSGID: 113020]
>>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix:
setting gfid on
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
>>>>>>         failed
>>>>>>         [2018-01-16 15:16:41.471745] W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
>>>>>>         [File exists]
>>>>>>         [2018-01-16 15:16:42.593392] W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
>>>>>>         [File exists]
>>>>>>         [2018-01-16 15:16:42.593426] E [MSGID: 113020]
>>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix:
setting gfid on
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
>>>>>>         failed
>>>>>>         [2018-01-16 15:17:04.129593] W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
>>>>>>         [File exists]
>>>>>>         The message "W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed
>>>>>>         [File exists]" repeated 5 times between
[2018-01-16
>>>>>>         15:17:04.129593] and [2018-01-16
15:17:04.129593]
>>>>>>         [2018-01-16 15:17:04.129661] E [MSGID: 113020]
>>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix:
setting gfid on
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8
>>>>>>         failed
>>>>>>         [2018-01-16 15:17:08.279162] W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
>>>>>>         [File exists]
>>>>>>         [2018-01-16 15:17:08.279162] W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
>>>>>>         [File exists]
>>>>>>         The message "W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed
>>>>>>         [File exists]" repeated 2 times between
[2018-01-16
>>>>>>         15:17:08.279162] and [2018-01-16
15:17:08.279162]
>>>>>>
>>>>>>         [2018-01-16 15:17:08.279177] E [MSGID: 113020]
>>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix:
setting gfid on
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9
>>>>>>         failed
>>>>>>         The message "W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed
>>>>>>         [File exists]" repeated 6 times between
[2018-01-16
>>>>>>         15:16:41.471745] and [2018-01-16
15:16:41.471807]
>>>>>>         The message "W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed
>>>>>>         [File exists]" repeated 2 times between
[2018-01-16
>>>>>>         15:16:42.593392] and [2018-01-16
15:16:42.593430]
>>>>>>         [2018-01-16 15:17:32.229689] W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
>>>>>>         [File exists]
>>>>>>         [2018-01-16 15:17:32.229720] E [MSGID: 113020]
>>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix:
setting gfid on
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
>>>>>>         failed
>>>>>>         [2018-01-16 15:18:07.154330] W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
>>>>>>         [File exists]
>>>>>>         [2018-01-16 15:18:07.154375] E [MSGID: 113020]
>>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix:
setting gfid on
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
>>>>>>         failed
>>>>>>         The message "W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed
>>>>>>         [File exists]" repeated 7 times between
[2018-01-16
>>>>>>         15:17:32.229689] and [2018-01-16
15:17:32.229806]
>>>>>>         The message "W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed
>>>>>>         [File exists]" repeated 3 times between
[2018-01-16
>>>>>>         15:18:07.154330] and [2018-01-16
15:18:07.154357]
>>>>>>         [2018-01-16 15:19:23.618794] W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
>>>>>>         [File exists]
>>>>>>         [2018-01-16 15:19:23.618827] E [MSGID: 113020]
>>>>>>         [posix.c:1485:posix_mknod] 0-gv2a2-posix:
setting gfid on
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
>>>>>>         failed
>>>>>>         The message "W [MSGID: 113096]
>>>>>>         [posix-handle.c:770:posix_handle_hard]
0-gv2a2-posix:
>>>>>>         link
>>>>>>        
/bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21
>>>>>>         ->
>>>>>>        
/bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed
>>>>>>         [File exists]" repeated 3 times between
[2018-01-16
>>>>>>         15:19:23.618794] and [2018-01-16
15:19:23.618794]
>>>>>>
>>>>>>         Thank you,
>>>>>>
>>>>>>
>>>>>>         Il 16/01/2018 11:40, Krutika Dhananjay ha
scritto:
>>>>>>>         Also to help isolate the component, could
you answer these:
>>>>>>>
>>>>>>>         1. on a different volume with shard not
enabled, do you
>>>>>>>         see this issue?
>>>>>>>         2. on a plain 3-way replicated volume (no
arbiter), do
>>>>>>>         you see this issue?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>         On Tue, Jan 16, 2018 at 4:03 PM, Krutika
Dhananjay
>>>>>>>         <kdhananj at redhat.com
<mailto:kdhananj at redhat.com>> wrote:
>>>>>>>
>>>>>>>             Please share the volume-info output and
the logs
>>>>>>>             under /var/log/glusterfs/ from all your
nodes. for
>>>>>>>             investigating the issue.
>>>>>>>
>>>>>>>             -Krutika
>>>>>>>
>>>>>>>             On Tue, Jan 16, 2018 at 1:30 PM, Ing.
Luca Lazzeroni
>>>>>>>             - Trend Servizi Srl <luca at
gvnet.it
>>>>>>>             <mailto:luca at gvnet.it>>
wrote:
>>>>>>>
>>>>>>>                 Hi to everyone.
>>>>>>>
>>>>>>>                 I've got a strange problem with
a gluster setup:
>>>>>>>                 3 nodes with Centos 7.4, Gluster
3.12.4 from
>>>>>>>                 Centos/Gluster repositories,
QEMU-KVM version
>>>>>>>                 2.9.0 (compiled from RHEL sources).
>>>>>>>
>>>>>>>                 I'm running volumes in replica
3 arbiter 1 mode
>>>>>>>                 (but I've got a volume in
"pure" replica 3 mode
>>>>>>>                 too). I've applied the
"virt" group settings to
>>>>>>>                 my volumes since they host VM
images.
>>>>>>>
>>>>>>>                 If I try to install something (eg:
Ubuntu Server
>>>>>>>                 16.04.3) on a VM (and so I generate
a bit of I/O
>>>>>>>                 inside it) and configure KVM to
access gluster
>>>>>>>                 volume directly (via libvirt),
install fails
>>>>>>>                 after a while because the disk
content is
>>>>>>>                 corrupted. If I inspect the block
inside the
>>>>>>>                 disk (by accessing the image
directly from
>>>>>>>                 outside) I can found many files
filled with "^@".
>>>>>>>
>>>>>>>
>>>>>>>         Also, what exactly do you mean by accessing
the image
>>>>>>>         directly from outside? Was it from the
brick directories
>>>>>>>         directly? Was it from the mount point of
the volume?
>>>>>>>         Could you elaborate? Which files exactly
did you check?
>>>>>>>
>>>>>>>         -Krutika
>>>>>>>
>>>>>>>
>>>>>>>                 If, instead, I configure KVM to
access VM images
>>>>>>>                 via a FUSE mount, everything seems
to work
>>>>>>>                 correctly.
>>>>>>>
>>>>>>>                 Note that the problem with install
is verified
>>>>>>>                 100% time with QCOW2 image, while
it appears
>>>>>>>                 only after with RAW disk images.
>>>>>>>
>>>>>>>                 Is there anyone who experienced the
same problem ?
>>>>>>>
>>>>>>>                 Thank you,
>>>>>>>
>>>>>>>
>>>>>>>                 -- 
>>>>>>>                 Ing. Luca Lazzeroni
>>>>>>>                 Responsabile Ricerca e Sviluppo
>>>>>>>                 Trend Servizi Srl
>>>>>>>                 Tel: 0376/631761
>>>>>>>                 Web: https://www.trendservizi.it
>>>>>>>
>>>>>>>                
_______________________________________________
>>>>>>>                 Gluster-users mailing list
>>>>>>>                 Gluster-users at gluster.org
>>>>>>>                 <mailto:Gluster-users at
gluster.org>
>>>>>>>                
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>                
<http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>         -- 
>>>>>>         Ing. Luca Lazzeroni
>>>>>>         Responsabile Ricerca e Sviluppo
>>>>>>         Trend Servizi Srl
>>>>>>         Tel: 0376/631761
>>>>>>         Web:https://www.trendservizi.it
>>>>>>
>>>>>>
>>>>>>         _______________________________________________
>>>>>>         Gluster-users mailing list
>>>>>>         Gluster-users at gluster.org
<mailto:Gluster-users at gluster.org>
>>>>>>        
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>        
<http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>>
>>>>>         -- 
>>>>>         Ing. Luca Lazzeroni
>>>>>         Responsabile Ricerca e Sviluppo
>>>>>         Trend Servizi Srl
>>>>>         Tel: 0376/631761
>>>>>         Web:https://www.trendservizi.it
>>>>>
>>>>>
>>>>>         _______________________________________________
>>>>>         Gluster-users mailing list
>>>>>         Gluster-users at gluster.org
<mailto:Gluster-users at gluster.org>
>>>>>        
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>        
<http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>
>>>>         -- 
>>>>         Ing. Luca Lazzeroni
>>>>         Responsabile Ricerca e Sviluppo
>>>>         Trend Servizi Srl
>>>>         Tel: 0376/631761
>>>>         Web:https://www.trendservizi.it
>>>>
>>>>
>>>>         _______________________________________________
>>>>         Gluster-users mailing list
>>>>         Gluster-users at gluster.org <mailto:Gluster-users
at gluster.org>
>>>>         http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>        
<http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>
>>>>
>>>
>>>     -- 
>>>     Ing. Luca Lazzeroni
>>>     Responsabile Ricerca e Sviluppo
>>>     Trend Servizi Srl
>>>     Tel: 0376/631761
>>>     Web:https://www.trendservizi.it
>>>
>>>
>>
>> -- 
>> Ing. Luca Lazzeroni
>> Responsabile Ricerca e Sviluppo
>> Trend Servizi Srl
>> Tel: 0376/631761
>> Web:https://www.trendservizi.it
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
> -- 
> Ing. Luca Lazzeroni
> Responsabile Ricerca e Sviluppo
> Trend Servizi Srl
> Tel: 0376/631761
> Web:https://www.trendservizi.it
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-- 
Ing. Luca Lazzeroni
Responsabile Ricerca e Sviluppo
Trend Servizi Srl
Tel: 0376/631761
Web: https://www.trendservizi.it

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180118/772bbacb/attachment.html>

Seemingly Similar Threads

Search for more seemingly similar threads

Gluster users - Jan 2018 - [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

[Gluster-users] [Possibile SPAM] Re: Problem with Gluster 3.12.4, VM and sharding

Seemingly Similar Threads